If let's say one wants to portray a villain in a novel that suffers from insanity/psychosis as he has literally lost his grip on reality and is kind of living in his own world, what are the things a writer should avoid when he writes such a character?
I would avoid the following:
- Stereotypes: it is very easy to describe cliches (screaming, sadistic violence, rage outbursts, contradictory or erratic behaviour, etc.) but they lead to a flat and not interesting character.
- Irrationality: don't let your character do random stuff just because it "sounds crazy". A mentally insane person always follows its own rationality, even though it differs from the more common one.
The main thing I see, in such a novel, is avoiding having this insanity be the equivalent of a deus ex machina that ultimately defeats him. I think the hero must defeat him in a way that would work even if he were not insane.
It would be an unsatisfying ending if after a long campaign in which Dr. Nutjob came close to defeating the hero several times, but finally gets distracted by something shiny so the hero can put a bullet in his head.
If your villain is insane but a daunting adversary throughout the story, he must be insane and a daunting adversary to the very end: His insanity can make him terrifying and unpredictable and thoroughly hated, but it cannot be the source of his defeat.
Do your research before you begin, look for disorders and conditions that match the symptoms you want your character to portray and then add some of the other symptoms commonly associated with that disorder. This will help you to build a profile of your character as well as knowing what triggers their symptoms to manifest. Presumably you already have their motives so use this profile to determine possible actions that would make sense from their point of view. For example, if they suffer from schizophrenia they may have auditory hallucinations. Use that. Delusions of grandeur? They may struggle with people in positions of authority such as the police ordering them to put the gun down. You mention psychosis which is a very broad term and will give you a lot of options but may make it difficult to create your profile.
Since you want your villain to be in their own world you need a diagnosis that makes sense (insanity is no longer a recognized condition). Psychosis fits well but again, schizophrenia (a subset of psychosis) is a good match here. It can be brought on by extreme mental stress so it can be the result of trauma which gives you an opportunity to build their back story. Make the audience sympathise, after all they are victims, they are suffering and need help.
Once you can explain their motivations, your character must still be able to execute their actions. Are they constantly tormented by voices? Then how do they sleep? They're going to be exhausted. How do they eat? Perhaps they have to steal to eat in which case they will likely be gaunt and weak. This is someone to pity, not hate.
Perhaps you are thinking more along the lines of someone who is high functioning, they can't tell the difference between their reality and ours but they have learned to walk the line between them. This is going to make them less sympathetic to the audience (think The Joker from Batman), but what happens when they lose that tenuous grip? How do they regain their control? Confrontation from your hero could trigger their more extreme symptoms and leave them vulnerable so they may need a way to avoid this outcome. Perhaps trapping your hero helps them to remain in control.
OK, this was long and a bit rambling but here's a summary:
- Avoid picking random symptoms, do your research and find the conditions that best match your vision
- Actions should match motives
- Develop a backstory for the audience to better understand your character, you can just provides hints if you don't want a full story
- Ensure your character is able to execute their plans, if they struggle to get through the day they probably won't be able to function without help
- Know your triggers, don't put your character in a situation that is likely to trigger their symptoms or if you do, ensure the outcome is realistic
Hope this helps
Make the insanity questionable. Think of Macbeth, he falls into madness as the play goes on but he might not be diagnosable. "Insanity" can be brought upon by something other than a mental illness.
The villain can be tormented by his past and hell-bent on some other goal which he's just using to try and deal with the old trauma.
The villain can become a nihilist and expressing how nothing matters good or bad everything is relative and it doesn't matter if he kills a woman or loves her. (Ex. Hamlet: "Nothing is good or bad but thinking makes it so.")
The villain could be a fanatic of some sort of ideology or ideals, even if the ideology is well intentioned, the villain followed it to its natural conclusion and brought great suffering. (Ex. Government/Socialism/Communism)
Try to make the villain's insanity somewhat logical so the audience can understand them and see the villain in themselves.
Start by defining the insanity. Mental illness is a very very broad field: you need to narrow down what you mean by "insanity/psychosis"; give your character a defined condition, and write him with the correct characteristics and actions for that condition. You don't have to name the condition, or refer to it at all outside of your personal notes; just understand that insanity is not a case of just behaving randomly; there is always an underlying reason for any symptoms.
Pick any insane or troubled character you can think of from literature -- Don Quixote, Hamlet, Moriarty, Scrooge, Eeyore, or many many others -- for all of the good ones, it is possible to read the stories and come to a definitive diagnosis of the character's mental illness.
You need to write your character like that. If you don't, people simply won't connect with the character and will find them unbelievable.