The key to horror is that fear comes from knowing that something is wrong but not knowing what it is or what you can do about it. You don't need to gross out or shock your reader. But you do what them to be nervous that you'll do just that at any moment.
The Saw franchise is one of the most overtly grotesque horror franchises out there. But the horror doesn't come from watching the victims get chewed up by the traps, it comes from knowing that they're about to die painfully but have nothing they can do about it. In stark contrast, the movie Signs hardly has a single drop of blood spilled throughout the entire film but still has scenes that build up oppressive senses of dread and panic.
Here are some techniques I've noticed come up consistently in excellent horror stories:
Tying the horror to the story's meaning
This is the most important rule. I'm putting it first for a reason.
A good story is more than just a series of events that happen. It's a series of decisions made by characters to overcome conflict in a way that sheds light on thoughtful themes. And so, following these core principles of good writing, it should come as no surprise that good horror is more than a parade of unfortunate situations. Make it clear to us why the protagonist must go through with the horrible situations they find themselves in, even as everything in their mind and body is telling them to get away and save themselves. Make it clear to us how truly severe the consequences will be if the villains are not stopped. And make everything that happens tie back to this central conflict.
You don't have to get to creative to do this. In the movie Alien, the poor main characters are trapped on the spaceship with nowhere to escape. Their options are to fight back or die.
But you can absolutely get creative if you want to. In the game The Evil Within, the main character is a detective who gets sucked into the mind of a serial killer because of sci-fi brain technology. Like Alien, the reason he goes on his journey is largely because he has no choice. But in the sequel, he willingly goes into another nightmare brain world powered by the same technology, knowing exactly how wrong it can go before he does so! The reason he makes this decision is because he discovers his daughter, who he assumed has been dead for years, is actually trapped in this nightmarescape, and this is his chance to save her. The entire game revolves around this. The horror isn't palpable just because the main character goes through hell. It's made much worse by the knowledge that his poor little girl, wherever she is, is certainly going through far worse.
And in Resident Evil 2, one of the protagonists is a policeman who gets caught in a zombie apocalypse. In his bad luck, this happens on his first day officially on the job, and he's too determined to save his city to back away just when he's getting started. What's more, as he learns more about the zombies, he discovers that they were knowingly created by an evil corporation that has plans to take over the world. We realize that he's not fighting just for his own survival, or even the city's. The consequences if he fails are too terrible to contemplate. And so of course, we find ourselves imagining just that, and the horror is elevated beyond "zombies are scary."
A lack of safety
Throughout your story, rarely give your protagonist a chance to be completely safe and let their guard down. Consistently put them in places where they're in danger, but not imminently. You want it to be very possible for them to suddenly be in immediate trouble at any moment, even though most of the time things are tensely quiet.
And you can set the tone for the entire story by letting the protagonist finally get somewhere they believe they're safe, then having something happen to them when they least expect it.
The movie Alien: Covenant has some problems, but it nails this aspect of horror. It's about the crew of a spaceship looking for a planet to build a colony on. They find a planet that seems perfect, but when they land, crew members start getting strangely ill. When one of the sick crew members suddenly has a violent alien burst out of their body and kill several other, still-healthy crew members, the threat is visceral: Any of the sick crew members can turn into a monster at any time. For most of the rest of the movie, the remaining crew members aren't actively turning, but the threat that it could happen at any time keeps the rest of the story tense.
And at the end of the movie, the few survivors escape back onto their ship and fly away from the planet. To celebrate, a husband and wife take a shower together, relieved to finally be safe enough to just enjoy each other's company. But then a spiked tail curls its way up behind the stall door... We thought we were safe and past the climax already, but there's a monster in our spaceship, the one safe place we could rely on the entire film.
Slow buildups with plenty of foreshadowing
Each major act or chapter of most horror stories revolves around one major, in-your-face scare. This scare happens at the end of the chapter, but everything else builds up to inexorably. The pattern looks like this:
- The chapter starts after the hero has barely escaped the last threat. They finally have a chance to catch their breath and consider their next steps.
- They head off into the next part of their adventure with some sort of goal. But as they find the trail to their goal, something is off.
- We find out how this next part of the story is scary. Maybe something is following us. Maybe a character we had trusted up to now is acting strangely. Maybe we realize we're trapped somewhere we thought was safe. But we do not find out what it is we're scared of or why we should be scared of it. We only see the effects it has.
- As the threat gets closer and closer to us, we get a drip-feed of clues about what the thing we should be scared of is. We see how a previous victim was killed, we hear its bone-chilling howl, we get a taunting call from another character telling us to watch our backs.
- At the same time the character reaches their goal, the antagonist that's been haunting us comes out in full view. This is when you get a few pages of in-your-face, grotesque monstrosity. We see the psycho chasing the hero with an axe, we hold our breath with the hero as they struggle with the lock as the room fills up with sewage, we are disgusted as the hypnotized victims are fed into a meat grinder.
- But our hero escapes, whether the goal is accomplished or not. And now, the build begins again...
You might be looking at this and thinking it's formulaic. If you do this every time, then your readers will know to expect something coming at the end of each section of your story. This is exactly the point. Again, horror comes from knowing something is wrong, but not knowing what it is or how to deal with it. So make damn sure your readers know something is wrong. And follow through on your promises that something will happen every single time so that your readers can't rely on the hope that maybe you'll back off this time. They know it will be terrifying when they reach the chapter's climax. Lean into this foreshadowing and structure so that your poor readers can only squirm in suspense as you build up over and over again.
Besides, this formula comes from a very sensible place. It's just the common pattern of rising action leading to a climax and denouement that shows up in all good storytelling. The only difference is that it's applied specifically to the conventions of horror.
The game Silent Hill 3 leans into this formula unapologetically and heavily. The game is about a young woman, Heather, who finds out a deranged priestess is using her to complete a cultic ritual to summon a demon. She has to travel to the titular Silent Hill, a cursed town that causes people's worst fears to appear as monsters, and confront the priestess before the ritual is completed.
The game is split into 5 major dungeons: A shopping mall, a subway, a hospital, an amusement park, and a chapel. Each of these chapters follows the formula I described above. Each area starts out relatively normal, but is gradually overwhelmed by Silent Hill's curse. More and more monsters appear, the environments become increasingly inhospitable, and characters gradually reveal more about the cultic ritual the priestess is trying to complete. Then, when Heather finally accomplishes each of her goals at the end of each chapter, she finds herself cornered in a fight to the death against the dungeon boss, which are consistently grotesque and overwrought even by the game's already disgusting standards.
The entire game follows this structure, as well! At the beginning of the game, all we know is that the priestess wants to find Heather. The further we get into the story, the more we find out about the connection between Heather and the priestess, the twisted religion, and how the ritual works with Heather at the heart of it. At the climax, the ritual is completed, and Heather has to kill the summoned demon immediately before it can damn the world.
Gaslighting the reader
This is a bit of a strange one, to be honest. It's not about building up your story, but rather about keeping your reader on edge.
To get inside your reader's head, you can feed them details that are contradictory, nonsensical, or unhelpful. For example, you can describe a wall as having a picture with a happy woman by herself on it. Then a while later, without any way for the picture to have plausibly been modified, nonchalantly mention the baby in her arms crying blood. Or repeatedly mention a bloody handkerchief that the hero notices in strange places, but it never ends up relating to anything or being explained. You can allow your hero to notice and respond to this strangeness or keep it a secret between you and the reader. And maybe there's a rational or even plot relevant explanation for the surreal things that happen, but maybe not. Either way, you'll keep the reader off-balance, and that will heighten the fear that they feel.
The game Layers of Fear is about an artist who is deep in mental illness. The game's story is about what his emotional experience is like, so its setup lets it quickly detach itself from any semblance of rational cause-and-effect. Instead, to capture how out of his mind the artist is, it yanks the player's chain around all the time. You'll look at something, turn around, and by the time you've looked back again, it will have disappeared. Rooms are connected in impossible ways; down a hallway, reaching a dead end, and returning to the entrance will often take you to a different room altogether. By the end of the game, you're climbing up a towering library that gets taller and more precarious every time you scramble onto another bookshelf, and there isn't enough space to swing your arms in other rooms without hitting an object that becomes possessed or is part of a psychedelic blood sacrifice, all without any explanation other than you're insane.
It's not horror, but the story I've found with the best examples of gaslighting is The Stanely Parable. The game has a narrator who tries to narrate your decisions through the game for you, and you're free to ignore his narration whenever you want to. The harder you disobey his suggestions, the more frustrated he becomes and the more surreally he attempts to reassert control over the story. He'll force you to play games that are deliberately awful in an attempt to teach you a lesson, try to kill the main character only to be replaced by a different narrator begging you to restart the game moments before the protagonist dies, and give Stanely entire lectures about the importance of making the right decisions.
You don't need to gaslight the reader if you don't think it would fit your story. The Alien franchise is both horror and hard sci-fi, so it avoids anything that can't be explained entirely within the movies' universe. But if you're writing looser or more experiential horror, this is a tool in your pocket.
Inhuman, but not necessarily gory, imagery
You do want to make your reader to feel uncomfortable with the imagery you're using. But as other answers have noted, this doesn't necessarily mean splatter porn. Vicera and gore are definitely disquieting. But so are claustrophobic spaces, filth, ordinary objects that don't work quite the way we expect them to, spaces that connect up in nonsensical ways, endless industrial areas, and grime and decay. You have an enormous palette of uncomfortable imagery to use. With skill, you can make completely family-friendly settings and scenarios still leave your readers sweating.
Again, Silent Hill 3 is an excellent example of this. When the Silent Hill curse takes over an area, it is twisted into something inhospitable by using some specific imagery. To be honest, there is a lot of blood and gore; decaying bodies and puddles of blood are scattered around everywhere. But it uses other imagery at least as often. It likes to create mazes using chain-link fences, an off-putting choice for indoor spaces. Every surface is either bare concrete or metal, and it's all stained, rusted, or covered in grime. There's barely a single carpet, wood floor, or wallpapered room in the game. Lighting is unnaturally dark, like there's a low ceiling without many lights, even when you're outside. And hallways are lined with rows upon rows of doors, most of which are impossible to open - but you can't know which ones work until you try all of them.
It's a lesser known game, but Infra is a horror game that uses some very unique imagery. You play as a government engineer for a second-world country sent out to inspect its infrastructure. As you explore power plants, sewage systems, dams, and the like, you discover two things: The government is crawling with corruption, and as a result, the infrastructure is utterly dilapidated. This means that as you make your inspections, you consistently find yourself in very dangerous situations, and there's no one to rescue you if - when - things go wrong. One memorable scene has you trapped on an underground bridge as flood gates are opening, forcing you to scramble to unlock the escape doors before you're washed away and killed. All of this game's imagery is completely grounded. There are no spirits, monsters, hallucinations, insane visions, or even gore. But it's nevertheless very tense.
Don't answer all of the story's secondary questions
Remember, in horror, your goal is to leave your readers feeling that something bad can happen at any time. This is a decidedly different goal than in other genres. In a mystery, it's to give your readers the satisfaction of solving a puzzle. In sci-fi, it's to fully explore the implications of technological changes that might come to pass. These genres require you to fully answer most or all of the questions your story asks.
Horror does not have the same expectations! By deliberately leaving some central questions unanswered, you can make it unclear to the reader whether the hero is truly safe even when the story is finished, allowing the sense of horror to persist beyond the point your reader closes your book.
The movie Silent Hill, based off the video games, is about the main character looking for her daughter after she is lost in the eponymous town. At the end of the story, the mom and her child make it home. But even when they're in the same room as the protagonist's husband, he can't see them. They escaped Silent Hill, but did they escape the town's curse?
Inception isn't horror, but the top that doesn't quite stop spinning before the credits roll is famous.
