Universal Story Theory
Stories are about a protagonist wanting something. An antagonist or force is placed in opposition to the protagonist achieving his goal. Scenes establish the movement towards that goal or away from that goal. Think back to your “story arcs”. Rising tension until the conflict is achieved by continuously raising the stakes of your story, by introducing conflict and solving conflict until all threads come together to make the final moment where the protagonist either achieves or fails to achieve their goal.
A scene is valuable if it measurably advances the plot, supports something which does that, or most importantly earns the moments that are to come. Each of the scenes you’ve picked out are emotional beats designed to make the understanding of characters. This is important because understanding a character brings you closer to feeling emotion for them, which makes you invest in them and care about what happens.
Everyone remembers that scene so it must be doing something. What is the main conflict of the story? The empire is building a death star and the rebels have to blow it up. How do they do that? It turns out they can't, not without Luke who must steal the death star plans and learn enough about the force to trust it over machines. What does Chess/Force Training do in the grand scheme of it all?
The team that is going to acquire the death star plans has just formed up, but they haven't gotten to know each other yet. This is a team-building scene in some ways where we see the teams differences, the things they have to overcome to earn the ability to work together. This is a scene with almost the whole team (sans leah) in the same room. We see that Han is a bit flaky, hasn't bought into "space wizardry". That foreshadows his eventual comeback. We see that Luke is frustrated; that learning the force is not easy. We see that the Wookie and Han have a long way to go to accepting the others in their "party".
It also briefly illustrates a before unforseen aspect of the force. We know it can be used to control minds, choke people (at least for evil people) but nothing else except maybe laser swords until this scene. Remember, this is the first movie. No one who watched this knew what the force was. Being able to sense laser bolts and block them is "AMAZING!". So it also has some wow, or what you might call a sense of wonder. And keep in mind that in this movie Luke has zero other force training scenes. After that the death star plot spins up and we lose Obi to provide further training. This is the only scene that sets up the big moment at the end directly for Luke where he has to learn to trust the force in order to win.
As a bonus, it also makes Chewie much more likable. And this is the smart thing about this scene. Take out the chess scene and you just have a set of characters milling about on stage or conveniently off stage. This is usually a failing in books as it removes texture. Having layers can be very good and give a sense of depth. Also, training scenes tend to be a bit didactic, which pushes a consumer away. By adding in some likability and a sub-sub-plot of "are the droids about to get wrecked?" there's a little bit of tension in the scene beyond Luke's whining about how hard using the force is. And there are some giggle's and laughs.
If only we could all write scenes that do this much while seeming to do nothing. I do not doubt that a lot of it came together as it did due to Mrs. Lukas (at the time's) careful cutting. Also, most of the elements of this scene either show us the price that has to be paid to achieve success, the distance one has to go to get there, or the difficulty of actually attaining the thing.
That is not to say this was the perfect way to impart this information. There may have been better more concise ways, but it's serviceable and it passes the "it works test" because we all remember it and we kind of love it. And what we love, we forgive even in the face of evidence of fault.
What Does that Say About Scene "Value"?
If you're looking for the unifying theory: Does this progress the plot? In your example yes it does in key ways identified above. And let's boil that down. A scene is necessary if it helps earn the pivotal moments in your story and/or provides the connective tissue necessary for those moments that earn the plot. You can't hit hard all of the time. Scene-sequel is all about raising the tension and then letting it off just a bit. The scene was "falcon escqpes" the sequel is "oh, these people are together now; what does that mean?"
If your story is about a conflict, with a protagonist on one side an an antagonist on the other. Then a scene is valuable if causes further investment in the stakes of the story and or provides attempts at progress. Progress itself is illusion. You can hand the characters success at any point. But, to earn your progress you have to make it seem reasonably earned and part of that means that your protagonists often need to try and fail multiple times before they succeed; hence try-fail cycles.
However, stories aren't just people doing things. It's often about how those things and people come together and fit to cause the things that happen. In order to make a reader want to get to the end you need to make them care about the people these things are happening to. Remember those deadly words: "I don't care about these people".
So the Chess scene ultimately works because it's about earning, developing, connecting and making us care. Now you might ask: well, how do I know when I've done enough? The answer is probably when you've taken away a piece of your story and it diminishes rather improves the thing. The easy cuts are the duplicitive ones, or the parts where you can't figure out what your story is doing in that moment that is necessary.
Being entertaining and lovable may be enough on its own, especially if you're writing an otherwise downer of a story. Sometimes you just need a bright spot for tone reasons, because it gives the reader something to love and hold on to; a reason to want the characters to succeed and get to the end.
Peter Panning & Tinkerbell
What is Hook about? It’s about a father learning to embrace his inner child in his adulthood so that he can love his family and live his life. Tinkerbell is a foil for him this entire time. She starts with memory he lacks. She knows what he needs to do and she could subsist entirely as the magical all-knowing companion, but the writers/directors/editors smartly chose to examine the major theme. Peter is at risk of forgetting who he is in this scene, which is ultimately a failure. If he becomes all-child, that’s as bad as becoming all-pirate. Tinkerbell represents a temptation: He can forget and never go back. It’s an option, just as it is for his son. This scene makes him remember his wife, his love for his wife and rekindles the fire and refocuses the plot on what Peter really wants: his kids. It’s a really smart scene and one I grow to love more with time. Super adult for a kid’s movie. :)
Ok, So Other Examples?
My favorite from recent cinema is probably the opening game of D&D in Stranger Things. It seems app here because it's also "just a game", but you might argue it does its job a bit better. It is a complete foreshadow for what is about to come. We've just seen something really dangerous, now we cut to innocent children. We get a sense of who they are, that they fight monsters and we start pulling on the sympathy vector. We immediately foreshadow the loss of a party member to the monster. But we're really ramping up on the 80s nostalgia, what it means to be a kid. Complete with begging mom for more time on a school night and riding bikes into the night; betting on comics and for the love of god eating pizza and playing games!
Could we have started with Joyce waking up in the morning, or the kid running in the woods from the monster? Yes. Would the story have been poorer for not having connected us to the characters and making us love them? Yes.
Back to SW for the closing: The Chess scene helps us love the universe. Its short. It's not explosive. And, around the other set pieces it feels small and extraneous. But, it does pull real work and honestly its some of the most important work in the movie for where the movie goes.
This Theory saved Star Wars
Star Wars was a mess the above is a useful resources that shows how they took the knife to the first in revisions. If they hadn't used this theory, the movie would have been a failure.