Born in Lemont, Illinois, Paine was the son of Reverend Samuel Delahaye Paine. Rev. Paine was born in London and served in the British Army during the Crimean War before emigrating to America in 1856. Rev. Paine served as a lieutenant in the 2nd Maine Battery during the American Civil War and later was chaplain-in-chief of the Grand Army of the Republic.