I am trying to state that a particular question is not annoying, but its frequency is. I have come up with the below sentence, but I feel like it could be worded better.
It is an annoying question only in the frequency in which it is asked.
While the question initially seemed pleasant enough, somewhere around the 127th utterance it took a nasty turn towards annoying.
The question was reminiscent of a top 40 pop song. The lighthearted head nod it inspired at first disintegrated all too quickly into the self-inficted banging of one's head into a concrete wall. (This wall, like most concrete walls, features nooks and crannies seemingly straight from a Thomas's English Muffin. Or are they all nooks? All crannies? After all, who can really tell the difference between nooks and crannies?)
The characteristics of that question's aging are more comparable to milk than to wine. While mostly agreeable in the beginning, it soured over time and failed to develop the delicious subtleties and complex notes of intellectual flavor that embroider deeper interrogatives much like ivy on the walls of an august New England academic building.
The question did not bother me - at least not the first hundred times.
Could there be some way to use indirection? The narrator in "For Esmé -- With Love and Squalor" tells of his training in April 1944:
"Rainy days, I generally sat in a dry place and read a book, often just an axe length away from a ping-pong table."
If we could write like that we wouldn't be on StackOVerflow. But you know what I mean.
If the annoyance attaches to the frequency of asking, rather than the question itself, OP started wrongly in the original version by placing annoying immediately before question. Put it next to the annoying thing...
The annoying frequency with which this question is asked... (if you've got more to say about it)
...or if you just want a simple statement, identify the annoying thing first so your gripe target is clear...
The frequency which which this question is asked is annoying.
I will also just point out that it's "frequency with which" something happens, not "in which".