Hilchos Rechilus, chapter 6, section 2

It is forbidden to accept rechilus even if the speaker said it to the person's face: "You said such and such about so-and-so." Even so, his silence is no proof that he really said it.

Even if he is someone who usually does not stay quiet, his silence proves nothing. Perhaps this time he decided to overcome his nature and stay quiet. Or perhaps he thinks that denying it won't help anyway, and people will believe the story about him no matter what.