Naturally in dialogue or thoughts, sparingly in exposition.
Adverbs are communication shortcuts, the reason they are disdained is that they are usually vague and do not evoke an image or experience or feeling in the reader. The 'Y' suffix means 'characterized by', so angrily means 'with anger', 'sexily' means 'evoking sexual attraction', etc. This makes them labels of a state, and a form of "telling" instead of "showing". 'Angrily' might evoke in the reader some general expression of anger, but it depends on what gradation of anger their mind defaults to, so it is pretty non-specific.
Of course many adjectives, metaphors and similes fall into the same category, it doesn't make the writing a whit better to say "Joe said, with anger." It is too broad a word. The same goes for an adjective like "beautiful". If a girl, a landscape, a house and a judo move can all be described as "beautiful", the word cannot be really conveying much meaning beyond a vague pointing well north or well south of something you like (or your character likes).
Another warning sign of this type of adverb or adjective is if in your experience people combine it intensifiers or modifiers: "really very beautiful", or "very, very angrily". If it needs such modifiers, there is probably a better word or description, like "furious", or "Joe's face trembled with fury, as if his mind raced over methods of murder."
However, such adverbs and adjectives, being shortcuts, are very widely used in casual speech, and your character's may do the same. It is common for us to use hyperbole and intensify broad descriptions in our conversations. The most beautiful girl I've ever seen, the sexiest man alive, etc.
So realistic speech can plausibly use such adverbs, and doing otherwise can seem unnatural. In real life we would never say "Joe's face trembled with fury," we'd say "He was very very angry, just furious."
Because in real speech, we tend to restrict our vocabulary to a small fraction of the words we know, to what we can conservatively expect will be a common understanding amongst native speakers. We also tend to want to get the general shape of our thought out quickly, and instinctively accompany this with expression, gestures, vocal pitch or volume, and body language.
None of which can be sensed by a reader, which is why you wish to avoid "label" adverbs and adjectives in exposition; they are much less meaningful without all that non-verbal communication. So good exposition writing should find more specific language that better conveys the images and scenes. Which requires more work by the writer: "Labeling" words tend to indicate the writer has not fully imagined the scene and imagery and feelings of the characters, or they have focused their attention on the information exchanged and forgotten the fact that their characters have eyes, ears, feelings and thoughts, and exist in a setting. (Which is fine for a start, but should be recognized and fixed once the scene is complete (my preference) or in a rewrite (not my preference since the images of the scene not fully captured in the text may have been forgotten in writing many following scenes).
As for thoughts, it is a good idea for thought to follow feeling, thought is often a language generalization of some more specific feeling. In exposition you can describe the feeling with more imagery and detail, and, just like in actual speech, the character's internal dialogue may capture their own feeling with a label. I can describe the effect seeing a dress has on my character, describe it, and then her thought can be labeling: This is so beautiful, she thought.
For an adverb in thought, that would usually apply to somebody else, not themselves: As he said the words, Sandy heard the sharp edge in his voice, the volume raised, his face tensed as if in hatred. Why is he talking so angrily? she wondered, perplexed. (there are obvious ways to rephrase that to avoid the adverb that I would prefer, like "Why is he so angry?", but it is a plausible thought at least.)
Not all adverbs in exposition are bad. "Often" is an adverb I use often. What is bad in writing is being vague when being more specific would bring more clarity of image and experience for the reader. But this requires the writer to be somewhat judicious, we also must pay attention to pace and we cannot address every detail, especially if a broad indication is enough.
I can say "Jerry enjoyed the food at Mike's, and went there often." I might clarify that with a frequency, "a few times a month", but I might not. "often" is specific enough for most readers and the precise detail may not be relevant, or I might feel getting into how often slows down the story for no reason, and delves into the past when I wish to focus on the present: If Jerry is asking a girl out on a first date, focusing on the present might be more important:
"His mind scrambled for a place to have coffee. Mike's is where he went alone most often, but it was more a comfortable studying dive, maybe not a first date place. "I've been wanting to try this brand new place, Grinders," he said. "It isn't far. We could check that out."