Think of dialog as a weapon your characters wield to attack each other.
If you write dialog as a battle, you have to have people with opposing opinions and it will show in the dialog and help to clarify "by agenda" who says what.
If you had two people battling over something your example might instead be:
"Hey, this is dumb," one of the men replied.
"You're wrong, it's not dumb," the other man said.
"Yes, it is."
The exchange in your example, where two people agree should probably not be "shown." It's easier to write something like:
One of the men felt X was dumb. The other agreed.
They felt X was dumb.
And then move on to the next scene with conflict. (Cut to the chase as it's called...)
Unless it can be cut completely, or maybe shown in a later scene with someone who doesn't agree with them.
Another thing that will help is that dialog cannot only be words and markers for who said them. It also has to be centered in the room by showing the characters interacting with the room, its objects, etc, as well as showing the emotions and reactions of the speakers.
Otherwise, the reader will get a "talking head" feeling after a while.
If they all agree, we can assume they're calmly sipping tea, but if they disagree... there will be a completely different pulse in the dialog and tempers will run hot. Hopefully...
Disagreement doesn't have to be explicit. If the second man is an employee of the first man, he might agree in words only to what the first man says. And you have to use subtle cues to show he doesn't really agree. Unless he's the POV character and his disagreement can be shown in his thoughts.
You can also use character descriptions in cases where you don't have names. For instance "the tall one", "the twitchy one", "dark voice", "squeaky voice", "blue jacket", "the unkempt one" etc.
But as often as possible, you should use action, dialog content, and character voice to show who's saying what.
Cut as many "he said" you can get away with. They may not be noticed by the reader, but if you're lazy with cutting them, the reader might notice that...