+1 DPT, yes your characters should be interacting with the setting.
If the "entire thing could have been a phone call", I'd say you have a problem with the dialogue, and the character emotions.
Dialogue itself should be influenced by the setting, and vice versa. Intimate dialogue is seldom exchanged in a loud restaurant, some of it isn't appropriate for a restaurant at all. You make it so the setting influences what can be said, and what characters want to communicate but can't, or can only do surreptitiously. A business meeting is not the place to declare undying love. A bedroom might be.
Likewise, depending upon what is being exchanged, you should devise your setting to, in some way, resonate with the emotions of the moment.
A failure to do that may be a failure of the dialogue, if it is monotoned, always the same emotions, then so is your setting, making both forgettable. People should be communicating for a reason, and dialogue is a form of action, there should be more than an exchange of information, there should be conflict (disagreement, questions, disbelief, confusion demanding an explanation) and emotions or feelings involved. Excitement at learning something, setbacks, wonder at revelations, vulnerability. Anger, disappointment, grief, fear of what might happen. You need to imagine how people FEEL about what they are hearing, they should be feeling something besides "Okay," or "Understood."
If your characters do not have feelings, then your settings cannot resonate with and emphasize those feelings, or create dissonance and clash with those feelings. (Clashes can work; e.g. sitting in a comedy club full of laughing people, and finding out on your phone your father was just killed in a car crash.)
Sometimes settings can influence or steer the conversation. For example, take two partners talking business coming back from lunch. But I want to change the topic to something more personal, to indicate their friendship. So first, a setting conducive to a friendly conversation. A shortcut through the city park. Second, I let their business conversation comes to a natural (comfortable) end. They walk for a minute in their own thoughts, then one of them (Mike) sees a retiree flying a two-string stunt kite in the park.
Mike gestures toward the kite flier. "That's what I want to do. Scrap this whole damn deal, and make myself a kite."
John giggled. "Go fly a kite? Is that a joke? 'Cause I don't get it."
"No joke, I love fuckin' kites, and I haven't flown one in ten years because of this stupid job."
"After We get this deal done, then ..." John spoke while laughing, "you can go fly a fuckin' kite!"
"Right, that's gonna happen," Mike said. Then after a pause, "They have competitions, you know. Make patterns with your kite, or they have battles to cut a string. It's like a sport."
"Uh huh. Like chess is a sport?"
Mike's head was turned right to watch the kite flyer, but they'd passed him. He faced forward, shook his head, smiling. "Fuck off."
If none of your settings actually matter, then you don't have enough variety of emotions in the types of conversations you have. If I want these two characters to have an argument, I'd likely put them in an irritating setting, someplace unpleasant that neither of them want to be. Not a park, maybe someplace noisy, or at least uncomfortable. Too hot, or too cold. Stranded in the middle of nowhere, perhaps. And I will craft the story so it seems natural they end up there, but I want the setting to add to the stress they are feeling by having an argument.