You have heard the story, haven’t you, about the man who was tarred and feathered and carried out of town on a rail? A man in the crowd asked him how he liked it. His reply was that if it was not for the honor of the thing, he would much rather walk.

Abraham Lincoln (1809–1865) American lawyer, politician, US President (1861–65)

Comment (1861 ca.) (Attributed)
    (Source)

When asked how he liked being president. Quoted in Emanuel Hertz, Lincoln Talks: A Biography in Anecdote (1939).

The earliest references I can find to this come from 1902 and 1903 (possibly the same source, as neither is biographical and both use the same phrasing:

I feel something like the Irishman who, after being ridden on a rail, said: "Bejabbers, if it wasn't for the honor of the thing I'd rather walk."

Those two sources refer to a Colonel Alexander of Topeka, Kansas, hearing the quip while visiting the White House in 1861. I cannot find such a person mentioned anywhere, but Lieutenant Colonel Alexander Bliss of Topeka, Kansas, assistant commander of the Quartermaster Corps, was known to Lincoln: the signed manuscript of the Gettysburg Address that Lincoln sent him in February 1864 is considered the definitive copy (its wording was used in the engravings at the Lincoln Memorial, and the manuscript is on display in the Lincoln Bedroom of the White House). None of that definitively links the anecdote to Bliss, and I can find nothing that does.

The same general quote is given in a 1923 source, though claiming an 1862 date when it occurred. The anecdote is sometimes, and with no more certainty, attributed to Mark Twain.

The punch line itself is more frequently found in what seems to have been a very popular joke in the late 19th Century about an Irishman riding in a sedan chair, then having to run through the mud when the bottom falls out: 1872, 1890, 1893, 1895, 1900.