Ajayi grew up in Osogun, West Africa, among the Yoruba people. In his early teens, he was captured by raiders and forced aboard a Portuguese slave ship with other Africans bound for the New World. A British warship patrolling the West African coast intercepted the vessel in an effort to stop the slave trade. A violent engagement followed, and only 87 of the 189 enslaved passengers survived. Ajayi was among those rescued.
He was taken to Sierra Leone, where British officials placed him in a school run by the Church Mission Society. There, he learned English, began studying the Scriptures, became a Christian, and was baptized in 1825, taking the name Samuel.
Two years later, Crowther enrolled as the first student of the newly established Fourah Bay College in Sierra Leone. He distinguished himself academically, devoting particular attention to Greek and Latin. His strong interest in language also led him to study West African languages, including Temne, the principal language of Sierra Leone. As a result, missionaries of the Church Mission Society increasingly relied on Crowther for assistance in the study of African languages.