Benjamin Withers was christened on 2 Mar 1783 in Great Shefford, Berkshire, England. He was the son of Benjamin Withers and Maria Honey, who married in Great Shefford on 7 November 1773. The younger Benjamin married Maria Miles on 23 Nov 1806 in Great Shefford. Maria was christened 6 Mar 1785 in Preshute (near Marlborough), Wiltshire, England. She was the daughter of Thomas Miles and Rebecca New, who married on 27 October 1766 in Preshute.
Benjamin Withers was a farm labourer. He and Maria remained at Great Shefford until after the birth of their first son and then moved to Marlborough in Wiltshire. They had all of their subsequent children at Marlborough. In the late 1830s, their oldest son William relocated to Andover in Hampshire. By the mid-1840s several of their other sons were in Andover and by 1850 most of the family had moved there, including Benjamin and Maria. Some found homes in Andover, while others settled in the nearby hamlet of Charlton. The sons all followed their father into work as farm labourers.