You aren't writing for Hallmark or the Lifetime Channel, are you?
I don't think anything is off the table if you do it right, but consider your audience. People aren't going to respond well to anything that glorifies evil, graphically portrays something deeply disrespectful in a way that doesn't emphasize humanity, and doesn't reach a conclusion that is either thoughtful or satisfying.
The storyline of The Messiah Stone ends with the amoral main character, after committing numerous murders, coming to a bad end. Most people who would want to read about graphic depictions of torture and rape are not people I would personally want as readers. I have a main character in a book who is a cannibal, a war criminal, and a child killer - and that is just the first three chapters. But you either get an explanation of why her behavior is culturally relevant, or she atones for it eventually when she realizes her mistakes.
People have complex demons driving them. Pretending they aren't there is not going to make them go away, and we need to be able to stare them in the face. But if we tell everyone what great guys the demons are, we risk being them.
Now I know people will disagree, but Dexter wouldn't exist if it were up to me. Even there, he's portrayed in a forgiving light - he kills only other serial killers. If you do glorify evil, make it clear there are trade-offs. Mad men suffer for their madness, and killers are themselves killed.
Doing your research is a good first step. Anything less leaves you open to legitimate criticisms. I'm not guaranteeing that you won't open yourself to criticism and condemnation, but I can't think that a writer who is afraid to write and risk is going to do anything terribly exciting or thought provoking.