James Castle was born September 25, 1899, in the small mountain town of Garden Valley, Idaho to Frances J. Castle and Mary Nora Scanlon. The fifth of seven children, Castle was born two months premature. His mother was the local midwife and his father was the postmaster and ran the community post office and general store from the living room of the family home.

Castle was profoundly deaf from birth and did not attend school until he was ten years old, when he was enrolled at the Gooding School for the Deaf and Blind in southeastern Idaho. Castle attended and lived at the school for five years, from 1910-1915.

While at the school, Castle was taught the oral method of communication—sign language was not taught in th...