You ask, “how do you determine that they're too unlikeable?”
Just in general? Here are some questions to ask:
- How will my readers feel about this character?
- Is that how I want my readers to feel about this character?
- Does this character fit the tone of my story?
- Is any of the content going to be a problem for my audience?
- Would a more-sympathetic villain be more interesting?
- Do the other characters react appropriately to what the villain has done?
- Would another kind of villain help develop the themes of the story?
- How many words or how much time would I need to flesh out this character more, and what else could I do with it instead?
One example that jumps to mind is a Fantasy novelist who decided she wanted to explore why women like one of her protagonists stay with a certain kind of boyfriend, and give her readers a warning about them. And that only works if the reader understands what she saw in him.