In a story I'm writing, the villain has his motives for his attitudes.
However, he is very mentally disturbed, which makes the logic of his motives be kind of illogic to other people (including the reader).
So how to make the reader find understandable the villain's motives even if it's illogic?
Is the illogic absolutely necessary?
Is it imperative to some aspect of your story that your villain's internal voice be self-contradicting and irrational?
Many mental illnesses can be represented in fiction just by establishing an incorrect yet fanatically held belief in the mind of the afflicted. It is not necessary to damage the afflicted's chain of logic. You just have to start that logical chain from an incorrect assumption.
A few examples might clarify what I am describing...
A meglomaniac only needs to believe that their insights, perceptions and intelligence so dwarf everyone else's, that no consideration of those other people is necessary. In every decision, only the afflicted's opinion matters because that opinion is simply better than all alternatives. Imagine a Nobel Laureate in a room full of kindergartners. Why would she listen to anything the children said or question her decisions in light of their insights? She wouldn't... and she would be right.
A paranoid only needs to believe that a major power is working against them. They can be considered crazy only if no such power exists. If they are being hunted by some sinister, secret agency, then everything they say and do is not only sane, it's prudent and appropriate.
In each of these cases, the diagnosis of insanity is conditional on the validity of a centrally held belief.
Find the belief that your villain holds exclusively, the belief that makes him the hero of his own story, despite the disbelief of others. Only then will your villain become a living part of your fiction, rather then just a shallow tool for manipulating its plot.
I'd make the villain reflect on or discuss his own illogical actions. It makes it clear to reader the villain is acting illogical, and that he is not able or willing to restrain himself any more in order to act logical.
Feelings like frustration and anger in combination with stress can bypass rational thoughts. A strong psychological trauma can drive someone to act upon on emotions instead of rational thoughts. For instance, the Joker and Batman in the Batman series often act illogically, although both highly capable of logical thinking, driven by their psychological trauma's.
Emotion vs ratio is something you find in many genre's of fiction and non-fiction. I think most readers would understand that if character is driven by emotion, especially destructive emotions like anger, he is no longer able to act in a rational manner.
If this is done from a POV that allows the reader to experience the character's thoughts directly, immerse us into his world view enough that the we the reader can understand why, to this character, this makes sense. His actions or logic don't have to make sense to the rest of the world or to someone without the character's particular world view. But it does need to make sense to that character and showing the difference between how the character views the world and how the world really is, is a great way to show this.
The TV show Dexter did a great job of making a psychopathic character sympathetic even as he is murdering people and it did it largely by showing us how and why he viewed the world he did. To the rest of the world his logic is flawed, yet because of seeing stuff through his eyes, understandable. If I can remember a literary example (since different media means different means of story telling) I'll edit to include.
People have been talking about the villain's POV in other responses so I thought I'd look at the other angle: where we have no direct insight to the villain, and just see them from the outside.
At some level, there must be a logic behind a character's action. Even if a character behaves differently on different occasions, well that unpredictability is evidently part of their logic, and all the more frightening to do so.
People like to fill in the gaps in their understanding of others, and the same is true of readers' attitudes to characters. If you can set up clues that relate to an underlying motivation for the character, then the readers can put the pieces together and you can build a character more powerful and more frightening than you could consciously.
A traditional approach which has much merit, not because the events of the backstory will appear directly or indirectly in the story, but because this will make the villain more real in your mind, and there will be more pieces that suggest themselves to you, to bring into the story.
Another traditional idea which has a lot of merit is that of introducing a chink of light into the darkness: this is something that will give the reader's thinking more to chew over. Curiously, this doesn't break the logic, but makes it more real.
One of the most frightening villains for me is Bill Sikes from Oliver Twist. We have no insight into the internal world of this appalling character, and yet even he arguably has a moment of neutrality, as he picks up Oliver and carries him off after the failed burglary.
Another very convincing villain for me (who I won't name to avoid spoiling) is the killer in "Sleeping Murder" by Agatha Christie. Almost by definition in a detective story we don't have access to the interior world of the killer, but here is someone who to me is totally convincing in their black-and-grey shadow, and much of the power of that novel (one of Christie's best) comes from this appalling character.
I want to end with yet another trope. In art, a character is suggested with some squiggles of paint, and it's no lie to say that the act of creation is as much in the reader's mind as in the writer's. When summoning a villain, this can be the more powerful if somehow the writer can tap into the fears and hatreds that lurk in all of us. So the villain is the more horrible for being part of ourselves.
Enough. Have fun.
