Colby is closely linked to the coal mining industry in the area and on the estate. The Colby Lodge was built between 1802 and 1804. The  3.2 ha (8 acres) gardens and its wider estate have been developed since the 1870s until the 1980s.