My first horror/fantasy novel has received some criticism from peers, many of them claiming that I introduce the antagonist too early. The introduction of the antagonist is nearly instant, but extremely ambiguous. It appears in 2 major forms, as a young boy, and a demon-like creature that vaguely resembles a very tall human. The criticizing peers have claimed that I had to exploit the fear of the unknown, and hide the antagonist for awhile. However, the antagonist is only shown relatively briefly in both of the main character's introductions, with no information on what it is, only that it (very messily) eats humans, and soon after, that it likes to torture its victims with horrifying illusions before killing them. There is no context on what it actually is, how intelligent it is, how powerful it really is, its true intentions, etc. I avoided revealing any significant information about it at all. Would this suffice to compensate for the missing "fear of the unknown"? And more importantly, what are the effects of either of them; when should you use one or the other?
Trust your Beta Readers:
Without knowing HOW exactly you're doing it, I can't give you a solid opinion. If I were you, I'd rely on my beta readers, who apparently don't think it works. No amount of "It should work" competes with "readers think otherwise."
You may be coming on too strong too fast. Horror should build and build, with reveals as you go. Horror is more about the journey than the destination. While seeing the last few minutes of a horror movie might be intense and action-filled, it's not by itself scary. This is my more general advice on horror. You might be going more for gore, which depends more on action, so this might not entirely apply. I don't personally find gore-style horror very frightening; it ends up gory, but not scary.
The best horror builds on itself, always ratcheting up pressure. If there's torture, you find a body with signs of torture before you see actual torture. If there's killing, you see someone about to be killed before you actually see a death. Always leave yourself room for things to get more gruesome.
Subtle hints can certainly work, if done right. But it needs to be very subtle at first. Take for example Stephen King's It.
It's one of the characters being informed the terrible IT is still a threat, >!and they would rather kill themselves than face the horror again. It establishes the sheer TERROR of the story without actually telling you exactly >!WHAT the terror is.
A person being chased by an unseen terror establishes fear, but you spoil the effect by trying to ramp up the tension too quickly. A dead person is less effective at the beginning than a dead rabbit. A torture victim stumbling out into a street and being hit by a car is subtle. But a head on a pike twisted in terror beats an actual torture scene early on. A terrified person begging to the darkness for mercy is subtle. but a voice from the darkness saying, "Mommy is wrong. It's more fun to play with my food." can be fairly subtle compared to a full reveal.
But if your baddie can use illusions, then it can avoid being seen. There are endless ways illusion can be employed to hide the real danger or make your villain obscure.
No matter how good you are at imagining something that you think is scary (for you), the imagination of your readers can conjure something infinitely scarier (for them).
Leave it to them to try and figure out what it is that should be scary. Your task is to shine the light on the backdrop. Provide the reader evidence that something is going on by showing the crime scene after the fact, interview the witnesses after the fact (but they are unable to provide conclusive details), make characters build theories to help the reader. By no means give any answer until the very last moment, or you deflate the expectation to a bare 'ah, ok, it is just a child-demon, I thought it was something worse'.