The punishment which the wise suffer who refuse to take part in the government is to live under the government of worse men.

Plato (c.428-347 BC) Greek philosopher
Republic, Book 1, 347c

In Ralph Waldo Emerson, "Eloquence," Society and Solitude (1870).

Alt. trans.:

• "One of the penalties for refusing to participate in politics, is that you end up being governed by your inferiors."
• The Constitution Party (1952-68) used on their letterhead the variant, "The penalty good men pay for indifference to public affairs is to be ruled by evil men."
• "The price of apathy is to be ruled by evil men."
• "Those who are too smart to engage in politics are punished by being governed by those who are dumber."

More discussion here.

In context (Plato in Twelve Volumes, Vols. 5 & 6 [tr. Shorey (1969)]):

[346e] "Then, Thrasymachus, is not this immediately apparent, that no art or office provides what is beneficial for itself -- but as we said long ago it provides and enjoins what is beneficial to its subject, considering the advantage of that, the weaker, and not the advantage the stronger? That was why, friend Thrasymachus, I was just now saying that no one of his own will chooses to hold rule and office and take other people's troubles in hand to straighten them out, but everybody expects pay for that, [347a] because he who is to exercise the art rightly never does what is best for himself or enjoins it when he gives commands according to the art, but what is best for the subject. That is the reason, it seems, why pay must be provided for those who are to consent to rule, either in form of money or honor or a penalty if they refuse." "What do you mean by that, Socrates?" said Glaucon. "The two wages I recognize, but the penalty you speak of and described as a form of wage I don't understand." "Then," said I, "you don't understand the wages of the best men [347b] for the sake of which the finest spirits hold office and rule when they consent to do so. Don't you know that to be covetous of honor and covetous of money is said to be and is a reproach?" "I do," he said. "Well, then," said I, "that is why the good are not willing to rule either for the sake of money or of honor. They do not wish to collect pay openly for their service of rule and be styled hirelings nor to take it by stealth from their office and be called thieves, nor yet for the sake of honor, [347c] for they are not covetous of honor. So there must be imposed some compulsion and penalty to constrain them to rule if they are to consent to hold office. That is perhaps why to seek office oneself and not await compulsion is thought disgraceful. But the chief penalty is to be governed by someone worse if a man will not himself hold office and rule. It is from fear of this, as it appears to me, that the better sort hold office when they do, and then they go to it not in the expectation of enjoyment nor as to a good thing, but as to a necessary evil and because they are unable to turn it over to better men than themselves [347d] or to their like. For we may venture to say that, if there should be a city of good men only, immunity from office-holding would be as eagerly contended for as office is now, and there it would be made plain that in very truth the true ruler does not naturally seek his own advantage but that of the ruled; so that every man of understanding would rather choose to be benefited by another than to be bothered with benefiting him. "