Hilchos Lashon Hara chapter 6, section 8

All this applies when the story is about an average person, who sometimes is careful about this sin and sometimes fails in it.

It is even more so if the story is about a God-fearing person. For him the mitzvah "You shall judge your fellow charitably" applies even more. If the listener does not judge him favorably, but judges him the bad way and agrees with what the speaker said against him, he certainly breaks the prohibition of accepting lashon hara.