I am talking about writing a sequel to a novel that's in the public domain. For example, Wizard of Oz. As far as I know, this is completely legal; however, the issue is how to deal with the fandom that exists. There are things fans want to see in a sequel and things that would be blasphemous to them, so how should you navigate this? Any guideline or procedure? I am thinking Dune has a huge fandom, although I am not sure if Dune is in the public domain. I don't think it's would be a good move to start your writing career with a sequel of Dune, wouldn't it? You can use any franchise as an example (even if they're not in the public domain) in your answer.
IANAL but I definitely recommend verifying you're on legally safe ground, even if it means getting authorisation from an author or their estate, as with the example I'll discuss throughout the rest of this answer, which focuses on the writing side. Excluding franchises that were designed to have multiple authors such as The Hardy Boys, by far my favourite example of an author writing a sequel to another's work is Stephen Baxter's 1995 The Time Ships, an authorised centennial-marking sequel to The Time Machine by H. G. Wells.
If you've never read the original 35,000-word novella, do; just be warned it's hardly modern in its style. But read the sequel next. Baxter finds a way to believably step into those Victorian shoes without making the writing inaccessible to a 1990s audience. He takes what worked in the original (a premise he fleshes out in a much longer, more complex story that didn't feel padded even when I read it as a child), and eschews what wouldn't again (telling almost the entire story as a quotation; Wells technically used an irrelevant other character as the first person narrator). Baxter also explores many questions Wells didn't: how would history change once multiple time machines enter the picture, how do they work, and how was the first one built? This allows him to give us alternate twentieth-century events, rather than an 802,701 that's less relatable. I also liked the supporting Morlock, Nebogipfel, plus a number of other things I'd better not mention because the plot doesn't deserve me spoiling it.
I think the lesson to take from Baxter is you have to not only love the original and know why you do; you also have to know why it's loved in general, and what would make a fitting answer to that. But every work is different, so I can't say specifically what you'd need to do. If you do read Baxter, however, try to think about why his decisions were prudent (and won multiple awards!), and more importantly how he'd come to them.
Writing a sequel to a beloved book not your own is dangerous ground. It has been done, but rarely are the sequels successful.
Take Gone With the Wind - quite successful novel about the Civil War and the trials and tribulations of its characters. When Scarlett was released, I wondered if this writer thought they could actually pick up where GWTW left off. My curiosity was not sufficient to open the book.
As JG says, you would need to love the original and know it heart and soul. This is not a writing exercise to be undertaken just because you want to - that can seem like hubris.
Are you a good enough, experienced author whose work could seamlessly dovetail with another’s? Can you adapt your voice to seem more like the author you emulate? Can you truly do the character justice?
Let’s assume, for the sake of argument, that you have decided to write the sequel to Les Miserables, telling how Marius and Cosette live their lives. Could you write with the gravitas, poetry and beauty of Hugo? Probably not. Anything less would be an insult to the original work.
If you want to write fanfiction - go ahead.
I recommend that you choose a theme, plot and character of your own, developing them over time.
Writing something of your own that someday might garner a following is better than taking a character created by another and trying to write as well as the masters.
First thing you would need to decide is what you actually want to do with that sequel. Do you intend to write something that is basically "more of the same"? "More of the same", but more modern? Or do you want to deconstruct the source material in some way? All are viable options.
In either case, there are a few things you want to be very careful with: existing characters, and themes.
Characters: you don't want characters from the source material to suddenly start acting and thinking very differently from the way they did in the original, without good reason. You want them to be the same familiar characters, not different characters that just happen to carry the same name. As an example, Star Wars the Last Jedi received a lot of criticism because Luke is so different from the person we knew him to be in the originals. (The film tried to offer justification for his changed attitude, it's up to you to decide whether this justification was sufficient, and whether the change was a good idea in the first place.)
Themes: if your work is very different in spirit from the source material, fans who loved the original for what it is might respond negatively to the significantly different sequel. This is another reason The Last Jedi was attacked - it stepped away from the straightforward idealistic black-and-white hero's journey the original trilogy followed. (Again, not getting into whether it did so well or not.)
Whatever you write, if the original work has a significant fandom, there would be those who would feel your new material is "sacrilege", and nothing can compare to the original. There's nothing you can do about those voices, other than accept them as the price of writing a sequel to another's work.
You've received some fantastic answers so far. I would add to those answers that it's your responsibility as the author of such a sequel to learn and talk with the fans of that franchise. Fortunately, today in the age of the Internet, that research is not so hard to execute. You would need to visit and participate in Facebook groups, blogs, forums and more about the fandom and how they react to the franchise.
That type of research is not so different than the kind of research you would conduct to make sure you are writing about a topic reliably. The difference is that instead of studying about a concept, you explore the people surrounding an existing idea.