Poverty, sir, is no disgrace to a man, but it is confoundedly inconvenient.

Sydney Smith (1771-1845) English clergyman, essayist, wit

Sydney Smith: His Wit and Wisdom (1900) [ed. J. Potter Briscoe]
    (Source)

In the Edinburgh Review (1855-07) coverage of Lady Holland's A Memoir of the Reverend Sydney Smith (1855), the reviewer notes that Smith himself attributed this phrase to "a fellow-passenger in a stage coach."