The difference is thematic.
Fantasy deals with destiny, family lineage and dynasty, purity of purpose, and typically things of the past. Fantasy looks back to a "simpler" age with polarized b&w morality (reductive), to examine character journeys about becoming legendary archetypes and paragons – combine family lineage with paragon characters and we get Chosen One stories, sometimes "accidental chosen ones" who did not know the true nature of their lineage until they pass a purity test and the story eventually distills them into paragons. Characters get punished for defying their destiny – an unconscious signal of a setting that looks like the past is that destiny has already been written and cannot be altered.
The themes in Science Fiction are invention and evolution – the opposite of a pre-determined destiny. Characters evolve their way into and out of trouble. Typically this evolution is already underway, perhaps science is evolving faster than society (progressive), or one group of characters has an evolutionary advantage over another. Conflict is solved not through a purity test but by unconventional thinking and the confluence of ideas. The unconscious signal of setting stories in the future is that fate has not yet been decided, and can be altered. Grey morality means that Science Fiction often deals with the unexpected consequences of invention or evolution. There is no "destiny" in the Fantasy sense because the future is unwritten and anyone can evolve.
Themes are stronger than tropes
Despite the gonzo trappings of deliberately anachronistic tropes: telepathy druids, raygun princesses, future primitives, a theme about bloodline purity and pre-destiny is (for the most part) mutually exclusive to a theme about invention and evolution. Star Wars is Fantasy, period. There is not a single invention or scientist in the entire SW universe, and zero problems are solved through innovation or intelligence. In contrast, a story set in the historical past might feature an anachronistic inventor who builds a robot out of coconuts – the fact that the reader knows these inventions already, doesn't change the idea that the character is "winning" through invention and evolution.
In a nod to EvilSnack's answer, the 3rd genre in the trifecta is Horror which has its own themes about loss of personal agency and de-evolution of humanity, confrontation with the paranormal which goes unexplained, and no-win scenarios which the characters cannot completely overcome.