Two stone perimeter walls which have been incorporated into the outer walls of the Sydney Observatory on Observatory Hill are all that remain of Fort Phillip, an ill-conceived military fort that was never completed and never had the potential to be used as anything but a barracks or, as happened, a signal station. Constructed in 1804, Fort Phillip was the brainchild of Captain John Hunter who was a naval man with little knowledge of land-based fortifications. Its function was to defend the colony from rebellious convicts and possible French attack.

The six-sided fort was designed by a French born Ensign of the NSW Corps, Francis Barrallier who, many years later, was responsible for the erection of Nelson's Column (1840) in Trafalgar Square, London. Had Fort Phillip's guns been fired, they would in all probability have hit houses and the hospital in The Rocks, which were located in their line of fire. The fort's armoury of guns were never mounted properly. Upon taking over the Governorship, William Bligh found them in a state of disrepair and on carriages that had been eaten away by termites.

From 1825, signals were sent to ships in the harbour and to the South Head Signal Station using flags at the abandoned fort which was partially demolished in 1840. A new signal station designed by the colonial architect Mortimer Lewis was built in 1848, it outer perimeter walls incorporated the two walls that remained from the fort. Sydney Observatory actual began as a simple time-ball tower. Every day at exactly 1.00 pm, the time ball on top of the tower would drop to signal the correct time. At the same time a cannon on Fort Denison was fired. The two walls of the fort inside which the Signal Station and Observatory were built remain today.
UBD Map 1 Ref E 10