Hilchos Lashon Hara chapter 7, section 3

Just as it is forbidden to accept lashon hara heard from one person, the same is true if it was heard from two people or more.

Even if they told him something that would not make them wicked for saying it if it is true, it is still forbidden to believe it and decide that it is true. This is because testimony only counts in a beis din (Jewish court), not outside of it. And if they lie outside a beis din, they are not called false witnesses, but simply people who spread a bad name.

All this is about deciding that it is true. But being cautious is allowed, even if he heard it from only one person.