Hilchos Lashon Hara chapter 9, section 4

It is forbidden to live in a neighborhood of baalei lashon hara (people who regularly speak lashon hara). All the more so it is forbidden to sit with them and listen to what they say. Even if he does not plan to accept it, since he is turning his ear to listen, it is forbidden to sit with them.

If he knows that one of his students is a baal lashon hara, he must send him away from himself.

If something beyond his control makes him end up in a group of baalei lashon hara, and he hears them speaking lashon hara: if his correction will make them stop, he is certainly obligated by Torah law to correct them. Even if he thinks his correction will not help, but it also will not make things worse, he is obligated to correct them.

This is why it is forbidden to sit with baalei lashon hara: even by staying silent he is punished, when he hears their words and does not correct them.