Hilchos Rechilus, chapter 7, section 2

Rechilus is forbidden even about an am ha'aretz (a person without Torah learning), because he is included in "your fellow" (amitecha).

Still, speaking rechilus about a Torah scholar is more serious, for several reasons:

a. The rechilus is most likely false, because a Torah scholar most likely does not do harm to people or insult them for no reason.

b. There is a commandment to honor Torah scholars and to stay close to them. How much more so must one not start fights against him — that is the opposite of staying close to him.

c. The person the rechilus is about will feel more hatred if he hears that a Torah scholar spoke about him. Hatred toward the scholar will enter his heart.