The time to assert rights is when they are denied; the men to assert them are those to whom they are denied. The community which dares not protect its humblest and most hated member in the free utterance of his opinions, no matter how false or hateful, is only a gang of slaves.

Wendell Phillips (1811-1884) American abolitionist, orator, social activist

“Mobs and Education,” Speech, Twenty-Eighth Congregational Society, Boston (16 Dec 1860)
    (Source)

As reported in the Liberator (21 Dec 1860).

Note: There is a synthetic quotation frequently attributed to Phillips that is a actually combination of this one, and these three others:

No matter whose lips that would speak, they must be free and ungagged. The community which dares not protect its humblest and most hated member in the free utterance of his opinions, no matter how false or hateful, is only a gang of slaves. If there is anything in the universe that can’t stand discussion, let it crack.

While Phillips often reused rhetorical elements (as most orators do), this particular combination appears to be combination not actually found in his speeches or writing.