How do you make a "bad" ending satisfying for the readers? I want to write a story where the protagonist dies without achieving his goals, but I am not sure if it's possible to write such an ending while making it satisfying for the readers. The only way I think I can make it work is by writing a terrible main character, anti-hero, like the Joker, but I am not sure if there's a way to make it work while writing a generic main character who is not a anti-hero or is at least morally grey like Geralt from The Witcher.
I can't help but feel that you're looking at this backwards - for a "bad" ending to work effectively it needs to be a natural product of the story. Starting with the idea of "I want a bad ending" and working backwards is always going to be difficult to do, it's very easy to end up writing characters' actions or plot events in a counterintuitive way because they're constrained by that element of the ending and the whole thing stands a very good chance of feeling contrived.
It's not quite the same scenario but as a recent example consider the recent Obi-Wan Kenobi series, Obi-Wan defeats Vader as is to be expected, Star Wars after all generally adheres to the traditions of good triumphing over evil. Then, with Vader defeated and absolutely everything in-universe, and by the general notions of common sense screaming that he should kill him. He just.. walks away. Were the writers trying to tell us something about the evils of killing? Were they trying to say that everyone is redeemable? Were they even just trying to tell us that even after everything Obi-Wan still couldn't bring himself to kill a person he had considered a brother? Unfortunately the answer to all of these is no - instead they were simply constrained by the fact that Obi-Wan killing Vader then would have invalidated the films that take place afterwards (from an in-universe chronological perspective). From a writing point of view that moment was unsatisfying, frankly bad writing. They tried to write a story that said one thing but they had a hard plot constraint that didn't fit that and as a result it forced a hard left turn at the last second which was jarring.
A worthwhile thought exercise for your writing brain would be to pick that or something similar to your tastes and identify why it doesn't work and how you would go about fixing it - this is actually a really important thing to do as a writer, IMO anyone who tries to learn by only examining successful examples is only getting half an education. Learning what doesn't work and why is just as important.
If you contrast this with examples where the "bad" ending was a natural consequence of the story being told and the difference is palpable. Alan Moore's Watchmen for example, the book ends with the "villain" of the story (who is a well-intentioned extremist) not only winning, but to all intents and purposes convincing the heroes of their argument and, ultimately, being proven right. This works because a large part of the point of the book is a subversion of the normal comic book archetypes and tropes and the ending stays true to that.
Perhaps the most extreme example I can think of in terms of pushing this boundary would be Kafka's The Trial, where the book's protagonist is summarily executed at the end without even finding any answers as to what's going on or even what his crime was, and neither does the reader - while in a conventional story this would be seen as unsatisfying in The Trial the impenetrable nature of the situation and the wholesale futility of it all is the whole theme of the book and had the ending been victory for the protagonist, or even anything that demystified what was going on it would have completely undermined the rest of the story.
So if you want to create a story with the sort of ending you describe you need to think about what sort of story would have that as a thematically appropriate ending. The archetype of the main character is at most a minor factor, if the story your trying to tell and it's themes fit that ending then it will work - it doesn't really matter if they are a hero, morally grey, anti-hero, or even outright villain protagonist.
The goals of the protagonist may be different from the goals of the story.
Suppose that all that the protagonist really wants is to find a wife, marry, have kids, be happy. Instead, in pursuit of that goal, she saves her city, the country, and dies saving the world. She may not have achieved her goals, but the reader may be pretty satisfied with her saving the world.
The protagonist doesn't need to get a happy ending to make a good story ending. Sacrificing their own happiness for a greater good can elevate them as heroes.
Even if the protagonist does achieve their goals, that doesn't necessarily mean they get a happy ending. A happy ending might be precluded form the start by terminal disease, or some other inevitable end. It would be a bitter-sweet ending where they achieve their goal before being swallowed by fate.
Your readers' satisfaction may be indirect: Your story made them live through strong emotions — in the end sadness, sure, but still strong.
Stories with a sad ending are commonly called tragedies. Some of the greatest plays from antiquity are tragedies, as are some of Shakespeare's greatest plays. Romeo and Juliet, the story of the lovers who cannot live their sincere love, has become proverbial.
Perhaps we can briefly explore the question just why anybody who is not a masochist would watch a story with a sad ending, instead of one with a happy one. One answer surely is catharsis: By living through strong ersatz emotions, the theory goes, we get them out of our system, or at least reduce their grip on us.
It is also a known psychological fact that realizations we have during emotional upheaval are much better retained: We learn things that are connected to emotions much better (PTSD is the extreme, re-wiring our nerves). A tragedy or a tragic book that explores moral and ethical questions in terms of a tragic story — how we wish with all our heart that these innocent teenagers can live the life they deserve! How heart-wrenching that they can't! Whose fault is this? — may have a larger effect on the audience than your average romantic comedy.
Last not least I personally think that sad stories also let you appreciate your hopefully not-quite-so-sad life more.
You are describing a tragedy
One of the most famous plays ever written by Shakespeare is the tragedy "Hamlet". In "Hamlet", out of a grand total of 18 characters here are their fates. 11 end up dead, and only one ends up clearly achieving their stated goal. (Fortinbras gains control of the castle, it is unclear in some readings if Hamlet avenged his father) The reason people get invested is they know things will end badly, but it is not clear how. Telling readers explicitly that the story will end badly in a dramatic fashion will have readers waiting to put the pieces of their end together. You can do this either by directly telling the readers about the event with a flash forward or have other characters discover a tombstone with markings of a massive battle around it.
An ending isn't satisfying because it ends the narrative on a high note, but because it resolves all the story arcs in a meaningful way.
A bad ending with the protagonist dying can be such a resolution. For example when:
- It becomes obvious that the goal the protagonist was fighting for was never achievable for them.
- When the central conflict gets resolved by the death of the protagonist.
- When the protagonist has a major character development just before their death.
However, when there are still dangling plot threats which aren't resolved but cut short by the death of the protagonist, then that's an unsatisfying ending. The example from the question, the protagonist dies without achieving their goal, appears to be such a dangling plot threat. This is because the central story arc - how the goal is achieved - remains unresolved. But that is only the case when the story was actually about achieving that goal in the first place! That's not necessarily the case. Good stories are often not about the protagonist achieving their goal, but about their character development during the process. For example, a story "Bob climbs a mountain" might be not so much about telling the story of how the mountain gets climbed, but rather about what Bob learns about himself while mountain climbing and how that changes him as a person. The mountain and the process of climbing it just serve as a backdrop for that character development.
It is very well possible to write a story where the character has that satisfying epiphany the plot was working towards before reaching their stated goal. Perhaps they achieve it not despite but because they fail. They can then die in peace, because the story already has its satisfying conclusion.
If you watch the movie "Gladiator" with Russell Crowe; you will see a "happy" (satisfying) unhappy ending.
Crowe is a Roman general, victorious in battle, the favorite and an old friend of his Emperor. It is the end of war, and Crowe's only desire is to return home to his wife and young son.
But his Emperor dies, the Emperor's evil and jealous son takes over. He tortures and kills Crowe's wife and son, Crowe is reduced to a slave, sold as a Gladiator to fight to the death, against impossible odds, in the Colosseum against Roman soldiers. To die for the entertainment of the new Emperor.
The rest of the movie, Crowe fights with distinction, but in the end does die in the arena, but takes out the Emperor's son with him. As he dies, bleeding out on the Colosseum floor, we see a dream state in which he finally reunites with his wife and son in their villa and farm, the ending he deserved and never got.
We the audience are wistful and feel great pity for Crowe, but we feel like he died a worthy soldier. It is an unhappy ending that satisfies.
Is that a happy ending for Crowe?
I don't think so. He was the hero of the war, both the architect of victory and a master strategist and warrior himself. And this ending is his just rewards?
Yet it seems like a good ending. In the end, Crowe, in keeping with his character from the very start as a master tactician and warrior, gets his revenge against the man that stole away and destroyed everything and everyone he loved.
But that was far, far from the goal he began the story with.
The lesson of Gladiator is that, to have an unhappy ending, your character's original goal must be subverted, and replaced. The protagonist's goals must change, until it seems like they succeed, they give their life to a greater cause.
Crowe dies as he lived, protecting his country. At first from Invaders, but just when he thinks he can relax, in the end he must protect Rome from a cancer within, the young jealous coward that inherited the throne.
Of course that is not the only formula for a satisfying unhappy ending, but you need the same idea.
The goals of the protagonist are irrevocably made impossible by the villains, and progressively she (by her character and skills, like Crowe) moves to new goals. The goalposts keep shifting, until in the end, the only way she can "succeed" is to sacrifice herself, and she makes the decision to do that.
And the ending resonates, her death is indeed for the greater good, like Crowe's death. That will be seen as a satisfying ending, and reflects real life:
Few of us achieve our own dreams and expectations we had for our lives as children; those fantasies were never realistic. And they became modified and toned down as we experienced life. But I think most of us still arrive at something realistic for us, that works in the real world.
I think the Gladiator story resonates with us because it is a similar arc of continuously revised expectations from childhood to adulthood, constantly modified from impossibly "idyllic" to brutally "realistic".
Look for that theme in your story, or revise it to fit.
You want to make sure that somewhere along the story, what the audience considers "bad" diverges from what the protagonist considers "bad".
That does not mean your protagonist has to be a villain or anti-hero. Look at the Batman movie "The Dark Knight", where he essentially spies on everyone's cellphones. Imagine that after defeating Joker, Batman had decided to keep this system running in order to fight future crime. At this point, the audience would've most likely considered that a "bad" outcome. If, after some other conflict or struggle, Batman fails to achieve this goal, say because some well-meaning computer guy spots the system and disables it, then he would've failed to achieve his goal (of preventing all crime), but the audience would be "it's better this way", even without turning Batman into a villain (he can continue being a vigilante, the role already has some moral ambiguity).
What matters is that the reader doesn't need any particular story goals to be accomplished in order to feel satisfied. He needs to have story arcs come to a satisfying conclusion. A failure, or a partial victory, can be satisfying if they make sense, conclude the story and character arcs, and make the reader feel that something was accomplished, or changed. The Hero's Journey doesn't need to lead to the point the hero wanted. It can lead to the point the hero needed to be.