What I particularly admire in him is the firm stand he has taken, not only against the oppressors of his countrymen, but also against those opportunists who are always ready to compromise with the Devil. He perceives very clearly that the world is in greater peril from those who tolerate or encourage evil than from those who actually commit it.
 
[Was ich aber an ihm besonders bewundere ist seine charaktervolle Haltung nicht nur gegen die Unterdrücker seines Volkes, sondern auch gegen alle diejenigen Opportunisten, die immer bereit sind, mit dem Teufel zu paktieren. Er hat klar erkannt, dass die Welt mehr bedroht ist durch die, welche das Uebel dulden oder ihm Vorschub leisten, als durch die Uebeltäter selbst.]

Albert Einstein (1879-1955) German-American physicist

In Josep Maria Corredor, Conversations avec Pablo Casals [Conversations with Casals], Preface (1955) [tr. Mangeot (1956)]
    (Source)

The last part of the last sentence here is most frequently quoted. The text is from a letter Einstein wrote to Corredor about Pablo Casals, the Spanish cellist (30 Mar 1953), of which part was included in the Preface. The book of interviews with Casals was originally published in French, and used this translation to that language:

Ce que j’admire cependant particulièrement en lui, c’est sa ferme attitude non seulement à l’endroit des oppresseurs de son peuple, mais également à l’endroit des opportunistes toujours prêts à pactiser avec le diable. Il a su comprendre avec beaucoup de clairvoyance que le monde court un plus grand danger de la part de ceux qui tolèrent le mal ou l’encouragent que de la part de ceux-là mêmes qui le commettent.

Variants / paraphrases:

• "The world is a dangerous place, not because of those who do evil, but because of those who look on and do nothing."
• "The world is too dangerous to live in, not because of people’s evil deeds but because of those who sit and let it happen."
• "The world is a dangerous place not because there are so many evil people in it, but because there are so many good ones willing to sit back and let evil happen."
• "The world is not dangerous because of those who do harm. It’s dangerous because of those who watch and do nothing."
• "The world will not be destroyed by those who do evil, but by those who watch them without doing anything."

More discussion and background of this quotation:

• The World Is in Greater Peril from Those Who Tolerate or Encourage Evil Than from Those Who Actually Commit It – Quote Investigator.
• ZITATFORSCHUNG: "Die Welt wird nicht bedroht von den Menschen, die böse sind, sondern von denen, die das Böse zulassen." Albert Einstein (angeblich).
• Juttas Zitateblog: Die Welt wird nicht bedroht von den Menschen, die böse sind, sondern von denen, die das Böse zulassen.