It is always our inabilities that irritate us.

[Ce sont toujours nos impuissances qui nous irritent.]

Joseph Joubert (1754-1824) French moralist, philosopher, essayist, poet
Pensées [Thoughts], ch. 5 “Des Passions et des Affections de l’Âme [On the Soul], ¶ 29 (1850 ed.) [tr. Calvert (1866)]
    (Source)

(Source (French)). Alternate translations:

Our worries always come from our weaknesses.
[tr. Attwell (1896), ¶ 65]

It is always our incapacities that irritate us.
[tr. Lyttelton (1899), ch. 4, ¶ 19]

It is always our inabilities that vex us.
[tr. Collins (1928), ch. 5]