I'm writing some short stories that are all connected and are set in the same universe, because I planned to put them together into a novel later. My question is: If I publish those short stories, can I later modify and extend them into a novel, even if they have been published before? Is that considered self-plagiarism or something?
You can't plagiarize yourself.
It's actually pretty common for writers to turn a short story (or several) into a novel. Your only issues are about copyright.
If you self-publish the stories, you of course retain the copyright. If you use a traditional publisher (including magazines, websites, etc), you will have a contract (if you do not have a written contract, do not give permission for publication). The contract will specify what the copyright situation is and if there are any other limits.
It's common for the author to retain copyright but also be subject to some rules about it. For example, the short story publisher may have the right to publish it first (which seems easy, but sometimes projects get delayed). They may also insist on being credited if you republish the story.
Read your contracts carefully to make sure there are no surprises later. And remember, you can always negotiate the terms.
If you rewrite your short stories to fit a novel format, then they aren't the same stories, though you should still keep the original agreements, if they are anything more than wisps of inspiration.
If you are purposefully writing your short stories as if each were a chapter in your novel, then they aren't really short stories, but rather serialization. That's fine, it just needs to be made clear in your contracts.
Consider Nightfall by Issac Asimov & Robert Silverberg.
The novel was published in 1990, but it was actually an expansion of a short story that Asimov originally wrote in 1941.
Unless you have a contract with a publisher that legally prevents it, you can do whatever you like with your own work.
It's done all the time, Charles Stross's first two Laundry Files "novels" were in fact collections of earlier short fiction, albeit somewhat edited. The reverse is also done, this is called serialisation where large works are broken into small sections for publication in magazines.
You do need to make sure your contractual obligations are compatible with republication, in whichever direction you want to go, this is probably best done before you start your publication cycle.