E.J. Lilly began construction on Fair Oaks in 1858 and his family lived there over 100 years. The house incorporates a Georgian plan with Greek Revival and Italianate elements. During General Sherman's occupation of Fayetteville in 1865 some of the Union troops camped on the grounds. A silver tray bearing the bullet holes of Sherman's soldiers, who used the tray for target practice, still remains in the Lilly family.