Hilchos Lashon Hara chapter 6, section 6

When does this apply? Only if they were not saying forbidden things when he sat down with them, and he also cannot get away from them now.

But if they were already saying forbidden things when he wanted to sit with them, or if he could get away and is just too lazy, he is called a sinner just like them. This is because he went against what Chazal commanded: to stay far away from hearing improper talk.

And it is even worse if he wants to hear what they say. Then his sin is too heavy to carry. Because of this, he is written above, in the Book of Remembrance, as a wicked person and a baal lashon hara (someone who speaks lashon hara).