It is thought that the first town was built on the ruins of Alba Longa, a town dating back to the pre-Roman era founded according to tradition by Ascanio, the son of Enea.

A great deal of archaeological evidence has been found near Castel Gandolfo, including the remains of the villa of Domitian and buildings from the late-Republican and Imperial eras. The borgo owes its current name to the powerful Longobard family of the Gandolfi, which in the XII century controlled the area.

The village then became the possession of the Savelli and was raised to the status of a Duchy under Bernardino Savelli, until the estate was seized in payment of debts in 1596 and from 1604 was included in the inalienable possessions of the Holy See. The first Pope to stay in the town was Urban VIII in 1628 and it was he who was responsible for the first Papal bull ex Arce Gandulphi.

In 1798 a cruel battle known as the battle of Frattocchie or Castel Gandolfo against the French of Gioacchino Murat devastated the town and the troops pillaged the Papal Palace In 1870 when the bersaglieri entered Rome, the power of the Papal State came to an end once and for all and Pope Pius IX and his successors no longer stayed in the papal residence, until, thanks to the Guarentigie Laws, in 1871 the area of the Papal Palace obtained extra-territorial status.

In stipulating the Patti Lateranensi in 1929 Benito Mussolini granted possession of the Papal Palace and the adjacent properties to the new State of the Vatican City. An interesting piece of information: the first post box in the world was located in Castel Gandolfo in 1820 in piazza della Libertà and can still be seen today.

Today the town is an attractive medieval borgo, located within the Regional Park of the Roman Castles and is famous for being the summer residence of the Pope.