It apparently was built around 1860 by Jarvis Friou (1806-1896), a shoe manufacturer and later the owner of the Milton Hotel. However, some reports indicate that Dr. James Augustus Hurdle, D.D.S. (1849-1925) was a subsequent owner.
If Dr. Hurdle lived in the house that no longer stands, query when he purchased the house next door (that still stands). And what happened to the house that no longer stands? When was it demolished, burned, etc.? This was before the early 1970s when Ruth Little and Tony Wrenn began taking photographs for the Caswell County architectural book (published 1979).
The house that no longer stands has been referred to as:
"Salzman Jewelry/Town Hall"
Note that Otto Howard Salzman married Fannie Malinda Hines (1842-1979), daughter of prominent Milton tailor Benjamin Hines (1808-1887). Census records (1880) indicate that Otto and his three daughters lived in the Benjamin Hines household (see below). Thus, it is possible the building that no longer stands was just used as the Salzman jewelry business and not as a residence.
The 1870 US Census is not instructive as Otto and Fannie were married in 1872. Otto Salzman arrived in the US from Hamburg, Germany, 27 June 1865. At some point after Fannie died Otto apparently moved to Danville, Virginia, where he owned and operated a retail jewelry business. He died 1816 and rests at Green Hill Cemetery (Danville, Virginia). Otto apparently was Swiss, which is appropriate for a jeweler.
Of interest is the mother of Fannie Malinda Hines Salzman: Sarah Price Holder (1810-1853). Yes, she is the daughter of the James Holder (1774-1836) who built the Holder House now owned by Danny and Kimberly Cash!