Very specific psychiatric symptoms can occur even when there’s nothing actually there. There are quite a lot of special-purpose structures in the brain, and if they’re triggered by something like a tumor, you can hear things that aren’t there, see things, smell things, become impulsive, experience intense emotions: all from direct physiological causes. So while it’s comforting to believe that those little people represent some form of reality that is only revealed to these mushroom-eaters, there is plenty of precedent that says that it’s not necessarily so.