A sad spectacle! If they [other stars] be inhabited, what a scope for misery and folly; if they be na inhabited, what a waste of space!

Thomas Carlyle
Thomas Carlyle (1795–1881) Scottish essayist and historian
Attributed
    (Source)

Quoted by John Burroughs in his essay, "Shall We Accept the Universe?" (collected in Accepting the Universe (1920)). Burroughs may have originated the quote as we know it; I cannot find an earlier record of it online, let alone anything from Carlyle's corpus.

The attribution was popularized in (somewhat) more recent years by Carl Sagan. Sagan quoted it (not mentioning Burroughs) at the "Life on Earth and the Mind of Man" symposium at Boston University (1972-11-20).

A shorter variant of the quote is also associated with Sagan through Contact. Though his 1995 novel doesn't use it, it shows up in multiple places in the 1997 movie adaptation (screenplay by James Hart and Michael Goldenberg):

YOUNG ELLIE: Dad, do you think there's people on other planets?
TED ARROWAY: I don't know, Sparks. But I guess I'd say if it is just us... seems like an awful waste of space.
[Contact (1997)]

ELLIE ARROWAY: You know, there are four hundred billion stars out there, just in our galaxy alone. If only one out of a million of those had planets, and just of out of a million of those had life, and just one out of a million of those had intelligent life; there would be, literally ... millions of civilizations out there.
PALMER JOSS: (looking at the night sky) Well, if there wasn't, it'd be an awful waste of space.
ELLIE ALLOWAY: Amen.
[Contact (1997)]

ELLIE ARROWAY: (to a group of children) I'll tell you one thing about the universe, though. The universe is a pretty big place. It's bigger than anything anyone has ever dreamed of before. So if it's just us ... seems like an awful waste of space. Right?
[Contact (1997); last lines]