Recently, I've begun on my journey to create the skeleton of a book I plan on writing; The 888. Anime has been a large inspiration in the way I consider composing my characters along with their backstories. This gave me the idea that I would like to create a group similar to anime's "akatsuki" AKA a group of scary bad guys that the readers can find themselves relating to. Is there a base of rules or fundamentals I should follow while creating my story's villains?
Make each character three dimensional and interesting - essentially real. The interesting villain is the one who is often right, just has methods that are objectionable.
It depends on the kind of book you want to write, but I find the antagonist who knows his own limits and wants to push them, who understands that what he is doing is unpopular but nevertheless believes that what he is doing is right, perhaps for the greater good is more chilling. Not only can we imagine them existing, we might think we know of someone like that. They seem real.
I get to know my characters, sometimes before I write, but often as I write them. I learn things about them that I know to be true, things that never make it on the page.
Ask yourself what your antagonists want, who they are. I start with a name and take the core of their personality from it. Are they working with others? Do they collaborate out of necessity but don’t trust the others?
Over-the-top lair-dwelling villains are fun, but we know they could never be real.
A businessman with the desire to buy a town so he can tear down all of the old buildings and build nice apartments for city dwellers who want to escape the city does not seem evil, but we can hate him. We understand him, know that he might not care about the people he displaces - wrong place, wrong time. He thinks of the money he will make and of the new people who will move to his community, bringing wealth. The townfolk might see him as a villain, but hope he can be stopped before he destroys their world.
Even villains are the hero of their own story. Villian wants to release a plague to decimate humanity because he knows it is the best thing that can happen. Overpopulation will kill them all and destroy the planet - a cull now is necessary. He does not do it for fun - but because it must be done and he is the only one with the courage to do it. It is folly to do otherwise.
You need to know your characters well and love them. If you hate your antagonist, he will be flat and dull. Love them all, know them all and give them all character arcs.
Give them full lives and they will be interesting. They need aspects other than evil, traits that can make them people who are dangerous because their world view is contrary to societal norms and smart enough to make it a nail biter as they have the talent and resources needed to do the dastardly deed.
A realistic villain is a good, memorable villain in my opinion. With that said a villain is never a villain 100% of the time. If you take some time to flesh out your villain you might find that the lord invading his previous ally without warning is doing so because he was reminded of his mortality and wants to ensure he can leave a strong kingdom to his children to rule. In this way your villain can commit terrible acts with minimal damage to their conscience by justifying what they're doing.
Think of Cersei from Game of Thrones, She is able to endorse all kinds of murder for the sake of ensuring her son, Joffrey, gets everything he is entitled to. Her lack of any remorse in plotting her atrocities makes her a chilling and memorable character, all the while though she is not a villain to Joffrey or Jamie and has relate-able love for them.
The best rule I ever read for making a villain is to treat your villain as the most important character in your story and make him/her the most interesting character, even more so than the hero. As alluded to in @Rasdashan's answer, one of the most discussed villains right now is Thanos from the Avengers movies, who has a similiar goal in solving the world's problems by a universe level cull of the population, thus solving the Overpopulation problem. What makes him interesting is that he actually does care about the people he's willing to kill... which is why it's a coin flip... so that he's not doing it out of malice or spite or love or affection, each person is given a 50:50 chance to live or die... if a sinner lives and saint dies, so be it. He's not going to use any system of evaluation other than random chance to determine this. This alone qualifies him as "Not as big of a jerk as you could have been award." What really pushes him into sympathetic territory is that he has children that he deeply loves and are all adopted. Even if they don't return his love for them, he never stops loving them... and in one of his final acts in the film, the fact that all but one of his six children are now dead, it allows him to commiserate with a hero who has just lost someone she loves... and he also tearfully admits that to him, the personal nature of his loss... is still worth it to him... which is the gut punch to the audience... It also sets up a major theme of the film... As Captain America points out when the simple solution to the threat of Thanos is the death of one character on his team, "we don't trade lives." Meanwhile, while never echoed by Thanos, he does exactly that because it is a simple solution.
When writing a villain, the best ones will never see themselves as the villain... they think they are the hero... that whatever they do, they're actions are justified for the greater good of the mission. At least in their own mind.
A cull would be a logical solution to overpopulation, but not an ethical one... to most people... to most people... but what is logical is not ethical and what is ethical is not logical. The best villains will make the hero counterpart question if they did the right thing in opposing him... and he might not like what the answer says about himself.
To look at another franchise, X-Men, the main villain, Magneto, is often one of the most beloved villains in all of superhero works... because he and his opposite number, Charlse Xavier, both have the same goal (Mutant safety and acceptance) but go about it in different ways that are unrecognizable (Often summed up in saying Magneto is Malcolm X to Xavier's Martin Luthor King Jr.) and they actually are good friends and Magneto will admit that Charles has some good ideas but he's seen a history that disputes the work-ability of them.
Magneto grew up a Polish Jew during the Nazi Occupation. His life was threatened once because of the circumstances of his birth and he does not want to be in that situation ever again. Xavier was never a victim in the same way Magneto was and believes that most people are better than the Nazis. Magneto has no tolerance for bigotry and will oppose it, viololently if necessary... Xavier believes that this only creates a circle and sees it more productive to show the world the good mutants can do. They can agree on goals, but not solutions to achieve these goals. The final conversation of the first film sums up the relationship perefectly. With Magneto in jail, Xavier goes to visit his friend and they have the following discussion:
Magneto: Doesn't it ever wake you in the middle of the night? The feeling that someday they will pass that foolish law, or one just like it, and come for you and your children?
Professor Xavier: It does, indeed.
Magneto: What do you do when you wake up to that?
Professor Xavier: I feel a great swell of pity for the poor soul who comes to that school looking for trouble.
It's here where Magneto realizes where he's wrong. Charles does not give up on the goodness of people, and feels that if he can show the world that mutants who can control their powers and understand how they work and use them for the defense of others, humans or mutants, then he can show mutants to be worthy of judgement on the content of their character, not their content of their genes... and if still they insist on starting a war, they know the enemy will be capable of defending themselves.