Broadly speaking, likeable characters are relatable characters. I'll take as an example Prince Zuko, from Avatar.
At the beginning of the show, Zuko is a bad guy. He is mean to his subordinates and is conquering innocent people. His objective is clearly to capture (and possibly kill) Aang, the main character. Yet, he is very likeable. Why? Because, unlike, Hitler, we can understand why he does what he is doing. If we were put into his shoes, we might be doing the same.
A brief summary of Zuko: he comes from a troubled family. The only person who ever loved him was his mother (now missing). His father dislikes him and prefers his sister. His father humiliated him by burning his face and exiling him. The only way for him to get back home is to capture the Avatar (a pretty much impossible task).
Note that Zuko wants something we all would want in his situation. He wants to go back home and to be loved by his father. An important aspect here is that all of Zuko's unlikeable traits come from external sources this gives us hope that he may redeem himself at the end. This does not mean all flawed characters must be purely victims, but make sure he is at least partially a victim.
Another very important thing to keep in mind while looking at Zuko's success as likeable evil guy is that he actually shows signs of vulnerability and changing through the show. If he never hinted at becoming a better person, audiences would lose their interest. If he never showed his weaknesses, he would come up as bland and two-dimensional.
As a footnote, I will also add that Zuko undeniably has quite a good deal of "cool factor" on his side. He's an exiled badass prince with a burned face who shoots fire. Hell yeah.
Edit: @Piomicron was kind enough to remember me of Uncle Iroh. Iroh does indeed love Zuko, but Zuko is too selfish to realize that. Iroh will also be relevant as the character who motivates Zuko to change and redeem himself.
- Character should have weaknesses
- Character should be compelling/relatable
- Characters shows chances of redemption
- Character is a victim.
- Character has strong "cool factor"