Locust Grove has seen three major phases of development. Samuel Morse acquired a Georgian house, built in 1830 by John and Isabella Montgomery, as part of the estate he purchased in 1847. Central to his plan for improving the estate, Morse consulted his friend, the well known architect Alexander Jackson Davis, and together they created a villa in the popular Tuscan style. To the original home, Morse and Davis added two wings to the north and south, creating an octagon, as well as the porte-cochère and billiards room to the East, and the four-story tower structure facing west toward the river. This tower established a powerful focal point for the landscape that Morse was already planning.

In 1901, the Young family added the large dining room wing at the north end, bringing the house to its present form. The ground floor contains the original kitchen and the laundry, both used throughout the history of the house.

The unique combination of landscaped lawns, vistas, and architecture makes Locust Grove one of the most handsome Hudson River estates. In 1963, it became the first in the valley to be designated a National Historic Landmark.

Other Alexander Jackson Davis houses in the Hudson Valley include Lyndhurst in Tarrytown and Montgomery Place in Annandale-on-Hudson, both "Gothic Revival" in style.

Nearby Springside, Matthew Vassar's estate, was landscaped by Andrew Jackson Downing, whose ideas Morse tried to incorporate into Locust Grove.