I know what goes in to making an unlikable character compelling and engaging but I'm not sure I know how to make them unlikable in the first place. In particular how does one show that a main character who is not the POV character can't, not just doesn't but can't, relate to his fellow man? Are there particular behaviours that are more effectively alienating than others?
They should possess those uncomfortable traits attributed to Lord Byron. They should be mad, bad, and dangerous to know.
For example, the captain of the spaceship Theseus in Peter Watts' science fiction novel Blindside meets these criteria. He belongs to a human subspecies that preys on normal humans. Jukka Sarasti is a vampire resurrected by palaeogenetics for junk DNA and the genes of sociopaths and autistics, and given command of a dangerous mission, because he was the best man for the job.
It's hard to like someone who is so potentially dangerous to everyone around him, and who could easily be the death of those who work with him and work under him.
Mad, bad, and dangerous to know decodes into unpredictable, doesn't play by the rules and can't be relied on, but dangerous to know is still dangerous to know (no further decoding is needed with this trait). What's not to dislike?
With a character that is hyper focused on their mission and disengaged with others, you can still accomplish having a compelling character (if you want) or totally disengage with your audience the emotional investment they may have in that character by making them too distant.
If you want to still be compelling, think Katniss Everdeen, or really any of Jennifer Lawrence portrayals. She can be very socially awkward on camera, but her characters are still compelling because she is so driven towards her goal.
Whether you want the reader to be engaged with this character or not, this would be the best advice I can think of:
Intention is Everything Readers can deal with a range of character flaws when the character intends good. We all want to see the good in us, and in others, so even when we cannot relate personally to a character, if they have good intentions, we tend to be engaged.
Minor Flaws You don't have to make your character for me to dislike it. Selfish desires that are extremely relatable make a character really hard to like. When a character's goal is self-gain in any way, even reasonable self-gain, you're more likely to pick at their flaws. That would allow you to highlight their social ineptitude and push your audience away from this character.
Don't Hate Unless the character is the antagonist (either primary or secondary) avoid causing your audience to hate the character, you may turn them off to the book entirely if you make them so undesirable that the reader can't stand to read about them.
Characters to consider
- Protagonists: Sherlock Holmes, Katniss Everdeen,
- Supporting Characters: Hermione Granger, Eeyore, Tinker Bell
- Antagonists: Shylock (Shakespeare's Merchants of Venice), Grima Wormtongue
This list would be my best summary of characters that are socially alienated/alienating but still play a part in the plot.
Having a character unliked by the other person on a book is easy. Having him unliked by all of the reader is much, much harder (even impossible?). People tend to like and dislike different things. For me, I hate lying character. However, I do like character who are unliked/hated by everyone else in the story.
To make a character unliked by the other characters, just make him hurt everyone by having no clue of how to act in society.
For the readers, I guess the best you can do is to make them indifferent about this character. Readers won't unlike him, they will just not care. And the first step to make this kind of character is to not tell the character story ("I don't know you, so I don't care about you"). You can also try to make him bad, but I'm not sure it fits with your definition of a "compelling character".
Callousness, high disregard for others when it gets in their way, overindulgence in vices, destructive habits and tendencies... there are many human, compelling qualities which also happen to be mostly unlikable.
You may pity a certain alcoholic, you may even understand why they ended up where they are, but you could never respect or like them.
I highly recomennd a viewing of X-File's "Musings of a Cigarette Smoking Man." Which is the closest thing to a back story for villain of the X-Files that the viewer really gets... I say closest because it's based on his own semi-autobiographical fiction that was further subjected to unspecified editorial changes. In so far as what we think we know, the character has worked his entire life as a major power player and achieved nothing more than a small apartment that he shares with no one and a quashed dream of a better life as a writer. I think he's pretty accurate in terms of mission oriented and socially disengaged. There's one scene towards the end where we see him who has the power to control world events, put Saddam Hussain on hold, and choose the winner of the next Superbowl and rig important Olympic events... and he's so invested in his work, he gives his closest co-workers ties for Christmas... Identical Ties at that...