Can it just pop out of nowhere, or does it have to be something that was slightly mentioned before that only expert readers would catch?
When I read a book or watch a movie, I never seem to get the foreshadowing. No matter how strongly the reviews complain about forseeable plot twists or endings, I never know what will happen next. Maybe I'm too stupid, or maybe those reviewers just accidentally expected what the book delivered, but however that may be, for a reader like myself you don't need to do any foreshadowing at all, because it is a wasted effort.
What I do need is that whatever happens in a book makes sense. It can be unexpected or expected, sudden or slowly building up, it really doesn't matter. But it must be motivated by the setting and the characters and the story you tell and fit the logic of your fictional universe, and it must not be random.
A good twist fits into the story organically, at least in hindsight. So it doesn't have to be foreshadowed, necessarily, but there should be elements of the former plot that take a new shape after the twist, for sure.
A good twist doesn't just change the story, it actually improves it. So Sixth Sense, as the obvious example, was an okay ghost story before the twist, and then turned into a really good ghost story after the twist. And while it was a bit clumsy to have the film maker go back and show us all the scenes from the newly understood perspective, it was also satisfying to see how they fit into the new framework.
A twist that doesn't fit organically into the story may feel like a deus ex machina or a random WTF moment. I don't think either of those is a satisfying storytelling element.
So deliberate foreshadowing? Great if you can do it subtly enough for it to work, but not strictly necessary as long as the twist can be seen, in hindsight, to have affected the story all the way through.
A plot twist is the opposite of foreshadowing. A close reader of a book and whatever you call a person who watches a movie with similar attention and astuteness should catch foreshadowing, especially since foreshadowing is done with a limited set of common devices. A plot twist, however, is calculated to surprise us.
A plot twist is a sudden and surprising quantum leap forward in the audience's understanding of the narrative's truth. It leaves the audience surprised, shocked, or nonplussed. That truth must be there from the beginning. You cannot surprise the audience with a change of truth but only with a realization of the truth. So to create the truth that surprises, the truth must lie there innocuously among red herrings
Meanwhile, foreshadowing tends to be writ large:
- He who coughs, dies.
- A gun in the first act must go off by the last act (Chekhov's rule).
- Thunder underscores foreshadowing.
- Memorable lines of dialog foreshadow prophetically. Consider Gerald O'Hara's speech in the first half of Gone with the Wind (1939, Victor Fleming, George Cukor):
GERALD: “Do you mean to tell me, Katie Scarlett O’Hara, that Tara, that land, doesn’t mean anything to you? Why, land is the only thing in the world worth workin’ for, worth fightin’ for, worth dyin’ for, because it’s the only thing that lasts.”
At that moment, Scarlett is still essentially a girl, and she could care less about land. But land lies at the center of the grownup Scarlett's heart throughout the second half of the movie, and that's what Gerald's brief oration on land foreshadows.
So the plot twist must be set up but it isn't foreshadowed. The setting up is done subtly so that the surprise isn't given away too soon. Occasional clashes might occur between the truth the audience naturally assumes and the truth that will later surprise them, but these clashes are glossed over, toned down, or passed over quickly so as not to spoil the impending surprise. In The Others (2001; dir. Alejandro Amenábar) Nicole Kidman's husband shows up for a short while and says enough strange things to heighten the impending sense of doom without giving away the truth about not just himself but everyone.
In The Sixth Sense (1999; M. Night Shyamalan) we see Bruce Willis get shot by a deranged former patient, but without explicitly being told, we assume he recovers from his wound since we soon see him back practicing child psychology.
Oftentimes the surprise centers upon identity because we live in an age saturated with and permeated by electronic media such that the doublethink of the corporate state bleeds into the faux sovereignty of the individual. We think we are self-propelled individuals when in reality we live in a panopticon and wear off-the-rack personalities. A new, post-dystopic genre of literature and film called Paranoid Fiction has emerged to imitate this corporate controlled life. The most sensationalistic example of paranoid fiction is of course The Matrix (1999; The Wachowski Brothers), which portrays a world entirely generated by computer, and all the characters in it exist in a symbolic/imaginary dream. The real is something quite horrific. The terms symbolic, imaginary, and real come from the model of the psyche proposed by French psychoanalyst Jacques Lacan, whose ideas work quite well to analyze Paranoid Fiction films and books.
The Bourne Identity (2002, dir. Doug Liman) deals with Jason Bourne's (Matt Damon) loss of identity as a result of a Faustian bargain with government forces in which Bourne has lost his soul and become a killing machine for the government. Love, at least in the interstices between sequels, allows Bourne to momentarily reclaim his humanity.
Yet ironically the biggest plot twist in paranoid fiction concerning identity is left as an exercise for the viewer: the surprise is not brought home to the audience sitting in the theater. Blade Runner (1982, Ridley Scott) brushes up against the question of Deckard's (Harrison Ford) humanity several times, but never answers it:
Rachel: You know that Voigt-Kampff test of yours? Did you ever take that test yourself?
Yet there is a reason why this plot twist is not consummated in the movie: if we leave the theater thinking about whether Deckard is human or replicant, we fail to apply the question to ourselves.
Perhaps I've digressed, but then again I have now strongly illustrated the subtlety with which plot twists are set up, and that's what you must do to surprise your audience, and surprise is what it's all about.