Hilchos Lashon Hara chapter 7, section 2

There is no permission at all to believe lashon hara, even if the speaker said it in front of the person he spoke about. This is because we did not hear any confirmation from that person himself. It is even more so if the speaker did not really say it in front of him, but only claims "I would say this to his face." That is no reason to believe him.

Even if the person spoken about was there and stayed silent, this does not prove the story is true. Even if he is someone who usually does not stay silent, and this time he did, maybe he controlled his nature and decided not to get into a fight. Or maybe he saw that no matter how much he denied it, they would not believe him.

So it is forbidden to take his silence as proof that the story is true.