In my view the champion of frame challenges is Socrates. His famous method always starts with an attempt at a frame challenge.
The success of the Socratic method stands on the basis that it is based on logic, and it leads to check if a frame challenge exists on the basis of logical contradictions. Socrates poses a lot of importance of factual data, and little importance to beliefs. In fact, he often leads the other party to agreeably reconsider their beliefs on the basis of factual evidence.
Also, when arguing a frame challenge, steer clear of first-person POV's recounts of personal experiences.
...meanwhile in the world of SE
To reword it in the world of SE, if the OP sets some assumptions, a general direction and asks a question to achieve a certain goal, we can ask whether there exist a path that links the two and if the path goes along the stated direction.
The obvious case is that there exists such a path, in which case your frame challenge would be out of place.
If the path does not exist, then:
- show that the path does not exists, i.e. given the assumption and following the desired direction, the outcome is very unlikely (if not outright impossible) to be the requested goal.
- ask whether the assumption is based on facts, or whether the direction is based on facts. If both are factual, then the goal as such is unattainable: see below frame challenge the goal.
- if the assumption is based on facts, state an objectively logical sequence that goes from the assumption to the goal, but follows a different direction: see below frame challenge the direction.
- if the direction is based on facts: see below frame challenge the assumption.
- if neither assumption nor direction are factual: see below imagination is your limit.
A rule of thumb of conflict avoidance is to steer clear of adjectives that carry a judgment. It is perhaps the best example of when "show not tell" can make a difference. If you want the reader to conclude that a certain statement is wrong, you need to show it starting from the facts. Telling alone does not prove it. Also, stay clear of faulty generalizations and other types of logical fallacies: in real life, as in fiction, they are great to raise the tension, but detract from the argument you wish to make.
This is the hardest to frame. The reason being that the OP is about achieving it, and the author has invested their time and emotions in asking how to reach it.
A first person POV with personal experiences is prone to the critique of being anecdotal, no statistical significance, or N=1. On the other hand, a dry third-person POV presenting a series of purely logical steps of the type "if A is true then B is true" makes it easier for the author to follow your reasoning, and to apply it to their question. A successful argument sticks to the author question, free from judgement, and follows it until the contradiction point, where you simply stick to the non contradicting path and reach a different conclusion than the stated goal. There is no need to judge, nor to rub it in the author's face. Perhaps, as Socrates does sometimes, suggest reconsidering, and enjoy your upvotes.
In general, this follows the same advice as challenging the frame of the goal. It is perhaps easier to discuss as the direction is not in the present (like the assumption) and it is not part of the desired outcome, in which the author is emotionally invested. A trick Socrates does sometimes, if I recall correctly, is to gently steer the direction while the dialogue goes on. He could have steered it from the start, but by doing it gently, he causes less attrition. In SE terms, stick to the assumption, follow the direction and identify the contradiction. Instead of changing direction, correct it slightly so that you get closer and closer to the goal while at each iteration of your logical reasoning you continue steering the direction.
Challenging the assumption is equivalent to challenging to ability of the author to understand what they already have at hand, be it their situation, or the hypothesis of their mathematical problem. This is difficult as it may quickly turn into a judgement on the author. Third person POV, avoidance of adjectives are the obvious tools. Also, as in the case of challenging the direction, iterating over the same logical sequence with small changes between iterations, until the goal is reached and the direction is kept unchanged.
Imagination is the limit
When there is no factual evidence, it all becomes simply a writing challenge: write a series of logical events that lead to a given conclusion, following, or not following some initial cue and general plot. The challenge? The style: write it in an iron-clad reasoning, Third-person, impersonal POV, without using any adjectives or adverbs. If you are reading this, you probably are on Writing.SE and you know how to write that.