The couple had three houses: Ockham Park - a 17th century English country house in Ockham, County Surrey, a Scottish estate on Loch Torridon, and a house in London. From 1845 the family's main home was a 19th century country house, Horsley Towers.
They had three children: Byron, born in 1836, Anna Isabella, called Annabella, born in 1837, and Ralph Gordon, born in 1839.