He was born in Portsmouth, New Hampshire on December 31, 1817 and named James Field; the family later added the "s". His father was a sea captain and died before Fields was three. He and his brother were raised by their mother and her siblings, their aunt Mary and uncle George. At the age of 14, Fields took a job at the Old Corner Bookstore in Boston as an apprentice to publishers Carter and Hendee. His first published poetry was included in the Portsmouth Journal in 1837 but he drew more attention when, on September 13, 1838, he delivered his "Anniversary Poem" to the Boston Mercantile Library Association.