Yes, it can be a bad idea to make a villain too villainous - though it depends on how you do it.
I got a fantasy novel thirty years ago in a box of novels that someone sold off. I bought the box because some of the novels were good.
That one particular fantasy novel started with (at least) two chapters of a point of view character torturing, skinning, and dismembering other people.
The descriptions were (extremely) vivid, and the character was quite obviously enjoying the process. What's more, it was implied that this (evil) character was of a common character type in the universe of the story and that the behaviour such characters was normal and accepted.
I did something then that I've never done before or afterwards: I stopped reading the novel, shredded it (by hand,) then burned the pieces before flushing the ashes down the toilet.
I am not an advocate of censorship. I just didn't want to read any more of that book, and didn't want that vile thing anywhere close to me. For all I know or care, there are millions of copies of it still around to be read and enjoyed by people who like it - or not.
I just know that I will not read a story like that.
On the other hand, I just re-read The Road of Danger by David Drake.
In it, a minor character on the side of the opposition is "elevated" from "competent fellow working for the other side" to "evil villain to be destroyed immediately" by his personal pastime of torturing and killing children.
There's a couple of chapters of build up around the competent fellow on the other side who is providing information to a group who wants to take over the local government (with other, larger, long term plans,) along with build up of a mysterious character who has been killing vagrant children and dumping the remains in the local harbor.
The good guys eventually find that the two are the same character. The villainous things he does are known to the folks using his talents - they protect him from the local police.
Two of the good guys (women, actually) invade the place where he does his spying and his killing. They capture all of his information and take over much of his electronics. They also kill him and all of his helpers.
There is a scene in which the villain has obviously been enjoying the process of killing another child, with the remains still lying there when the good guys arrive.
This is a villain. Straight up evil and disgusting, with nothing to make a reader like reading about him. The other characters find him disgusting. At a guess, the author found him disgusting.
As a reader, you cheer the death of this character.
Your villains can be vile and villainous - but don't try to make it so that your reader has to sympathize with truly horrid things they do.
- Villain who is trying to do what he thinks is right, but can't find any way forward that doesn't involve murder or torture - can be made to work.
- Villain who just does nasty shit to be nasty and enjoy it - don't try to make your readers like this character, or make it a POV character. I really don't want to read page after page of pornographic descriptions of killing, maiming, and torturing. Maybe somebody does. I don't, and I expect there are many who would agree.
It is, of course, always complicated.
Take those two characters up there who kill the child murderer.
If you look at it from outside, those two are villains:
- They stole a vehicle to gain access to the bad guy's lair.
- They drugged a vagrant child to use him as bait to get access to the bad guy's lair.
- They killed no less than a half a dozen people besides the bad guy in order to stop him and destroy his organization.
- They stole another vehicle to escape.
Those are vigilante actions, and all blatantly illegal.
On the other hand, the legal authorities were all taking bribes to ignore what was going on.
Who is right? Who is wrong?
That's the kind of thing that makes a good story good. It makes you think and consider.
Something else to consider is this:
A reader reading a well written story becomes the POV character.
The reader sees, feels, hears, smells, and thinks all the things the POV character sees, feels, hears, smells, and thinks.
I know that the character of the POV figures in good novels colors my thoughts and actions.
When I'm re-reading the Nicholas Seafort stories, I am more forceful and decisive.
When I re-read the Gateway stories, I am more passive and reflective.
A good author makes the reader into the POV character to some extent in real life.
Do you (as author) want to create pyschopathic monsters?
Do you (as a reader) want to become a psychopathic monster?
Keep that in mind as you are writing. Your words influence reality. What kind of reality do you want to create? Will your readers want in inhabit that reality as the characters you've created?