He was born at Bagni di Lucca, Italy, the only son of the American sculptor Thomas Crawford and Louisa Cutler Ward, the brother of writer Mary Crawford Fraser (aka Mrs. Hugh Fraser), and the nephew of Julia Ward Howe, the American poet. He studied successively at St Paul's School, Concord, New Hampshire; Cambridge University; University of Heidelberg; and the University of Rome.
In 1879 he went to India, where he studied Sanskrit and edited in Allahabad The Indian Herald. Returning to America in February 1881, he continued to study Sanskrit at Harvard University for a year and for two years contributed to various periodicals, mainly The Critic. Early in 1882 he established his lifelong close friendship with Isabella Stewart Gardner.
During this period he lived most of the time in Boston at his Aunt Julia Ward Howe's house and in the company of his Uncle, Sam Ward. His family was concerned about his employment prospects. They suggested that he become a professional singer; he had a baritone voice and had entertained friends with recitals of songs by Franz Schubert. In January 1882, his family asked George Henschel, who was then conductor of the Boston Symphony Orchestra, to assess whether Crawford was capable of a professional singing career. Henschel told Crawford and his family that he would "never be able to sing in perfect tune.