I have an idea for a series, but I want the protagonist to die at the end of the first book, and make it seem like all hope is lost, and then in the second book others (one of his other relatives) rise up and finish what he started. I was wondering if this would be okay? And how could I make it work in such a way that all hope seems lost yet there's still space for a sequel?
Game of Thrones did exactly that, so it can't be that bad right?
There is a lot of risk you are putting on your work with this approach. Pretty much you are putting all your eggs in the basket of your writing being so good that people want to read the second book, even though the first one failed to to have a satisfying payoff.
Here are some bits of advice. First make it 100% clear that this is the first book in the series. People should probably even just going in know that they are not going to get the fill story from this one book. This will set expectations and make readers much less frustrated. It will for better, for worse, keep the kind of people that don't like series away from your wring.
Give the reader closure on something. Back to Game of Thrones, or Wheel of Time, some of these books get very frustrating as a 1000 pages later you feel like you have just consumed another part of the story. I feel like series like Dresden Files are much more respectful to their readers, giving conclusions to at least some plot points in each book. Maybe in your work the protagonist dies but the "relative" succeeds at something. Maybe thy escape, maybe they kill the licentiate that killed the protagonist. Pass the torch in the first book, and give the reader some sort of win, no matter how small to keep reading forward.
Past that go nuts. Modern readers seem to love hard core writers who are not afraid to kill their precious characters.
This is acceptable, and sort of done before. I don't know how familiar you are with games but the first 2 stories from the main StarCraft saga both end in a crushing defeat for the good guys (and the third one they had "won" the battle but were still losing the war). (Note: The first 3 stories were released together) Only until the 7th out of 9 stories the good guys finally seem to be winning(barely). With their victory eventually coming from multiple sides joining together.
Your description lacks a few details to give a proper answer so maybe add some more details on your situation? But generally speaking it is possible by just adding/revealing variables that were first unknown to the reader. (You can even hint at them as a form of foreshadowing). For example the dying actions of your main character could inspire neutrals/hostiles to have a change of heart and slowly switch sides boosting the chances of the remaining good guys.
Okay for whom? Everything that an author wants to write is acceptable for the author, as long it does not break the whatever laws in their jurisdiction. What is acceptable for publishers or readers is a different story. So edit your question and provide more information as to who you are asking about. The way your question is currently worded suggests that you are asking whether such a book could sell well. It may or may not, depending on a huge multitude of other factors, but the plot idea alone that you described cannot float or sink your book. Just look at the piles of trash that line the shelves of bookstores, if you do not believe me.