James Backhouse (4) was born in York on 22nd October 1825, son of the botanist, nurseryman and Quaker missionary James Backhouse (3) (born 1794, died 1869) and his wife Deborah (née Lowe) (born 1793, died 1827). His mother died when he was about two years old. When he was six, his father went to Australia to do missionary work for 7 years and then exploring Mauritius and southern Africa for 3 years, leaving him in the care of relatives until his return in 1841.
From 1843 onwards, he spent time exploring the uplands and mountains of Britain and Europe with his father, collecting alpine plants. After the death of his uncle, Thomas, in 1845, James (4) worked with his father in the family nursery business in York. James (4) was also an enthusiastic archaeologist and geologist. Together with his son, James (5), he excavated the Moking Hurth Cave (also known as Backhouse Cave, or the Teesdale Cave, or Malkins Cave) at Forest-in-Teesdale, County Durham, between 1878 and 1888.