I decided to included some offensive language in one of my novels as many novels do. I was wondering however at what point does it become poor dialogue or just plain bad English. Obviously I'm not using every second word as a cuss word but it is recurring.
When it adds no value to the text.
It depends entirely on the context and the setting. Swearing for the sake of swearing is juvenile. Teens do it because they think it makes them appear hard or grown up. Adults tend do it out of frustration, anger or pain, or to add emphasis. Most often frustration.
The trick is how you use it. You don't have to include swearing in dialogue explicitly to have it included in the story. People swear under their breath all the time (in fact, that's usually what I end up doing most of the day at work when responding to emails).
Culture and upbringing also play a role in how a character would respond, and how much they would curse. Australia is rife with this, but most of it's not meant offensively.
I think the only point where it becomes bad is if it stops having a point. As long as you keep the cussing with characters who would normally cuss, or use it in narration to be emphatic on something then it's okay.
There's this series The Raven Cycle by Maggie Stiefvater that I particularly like and one of her characters has a tendency to use the F word. It's okay because he's one of the only characters saying it, and he uses it in a nonchalant way that shows us how little this character cares for society's etiquette and manners. Other than that, there is not much other swearing save for "damn" or "shit" by other characters.
If you have a first person novel where the character is from a rough part of town, has grown up in a rude culture, then having- "I walked down the dark street at fucking 2 am with nothing but a shitty light shining from my phone." - makes sense and the reader may find this amusing and fun. Obviously someone who finds this offensive would stop reading, but that does not make it bad writing.
On the other hand, in my paragraph here if I just started inserting- "It's okay because he's one of the only fucking characters saying it, and he uses it in a nonchalant way that shows us how little shit this character cares for society's damn etiquette and manners." - Then that may come off as a little inappropriate and shocking. I am simply writing a chill response to a simple question on a website. There's no need for harsh tongue here.
The purpose of fiction is to give pleasure to the reader. The use of profanity, like anything else, works when it gives pleasure to the reader.
Of course, certain profanities will displease certain readers, so you will certainly lose some potential readers if you use profanity. The question is, are they readers you would otherwise have had? If not, you have lost nothing by using it.
Equally, the use of certain profanities will give pleasure to certain readers. The question is, will it give enough pleasure to those readers, over and above the other pleasures your story offers, to turn them from non-readers to readers? If not, you have gained nothing by using it.
Logically, then, you should use it when it gains you more readers than it loses you.
But there can be no general guideline for this, for the simple reason that the use of profanity adds to or takes away from the pleasures that a particular kind of story gives to a particular reader. The same reader may appreciate it in one kind of story and reject it in another, because it adds to the pleasure of one kind of story, for them, and detract from the pleasure of another kind of story, for them.
The right degree and use of profanity is always going to be highly specific to particular readers and particular stories, and even to the style of particular authors, one of whom may be able to pull it off where another cannot.
It is worth noting, however, that it is almost always possible to achieve the same dramatic effect without the use of profanity. There is always a risk in the use of profanity, in that most profane words are trigger words for some portion of the reading public. I almost always put a book down at the first bit of profanity, not because I am scandalized by it, but because it is almost always a flag for lazy writing. Unless you are very sure of what you are doing, it is probably safer to attempt to achieve the same effect by other means.
I think of cuss words in my writing like spices. I don't want hot sauce on everything, and I don't want my carrot cake doused in Habanero Death Spiral.
If you use them too liberally, then like spices you overwhelm the rest of the flavor of the dish. In writing, that includes any poetic imagery, turns of phrase, or anything else.
Like hot sauce, it also just numbs the reader to any shock value the words may have, the just become filler. If 'fuck' is on every fucking page for every fucking character then there is no fucking place to go when something truly fucked up happens out of fucking nowhere.
for this reason, I tend to use cuss words very sparingly, and only for characters in extreme emotional states: grief, rage, hot love, cold hatred.
I am not above the standard fare in my own life, and in real life probably use more cussing than average; often in laughter (e.g. that's so fucked up! What a shithead!).
But I don't write that way. Note that sparing use doesn't prevent one character from cussing frequently, the cusses will still be sparing on the page. If you are writing the Joe Pesci character in Goodfellas (an impulsively violent, cruel and out-of-control gangster), you may need it. Like Pesci's character, frequently cussing can be used to show a lack of self control in a character, and aggression in a character: "Fuck you, asshole" is typically used for anger and aggression.
Too much ruins things. Some of it may be necessary in order to comport to reality. If Alice smashes her thumb with a hammer, an exclamation of "Oh, Pansies! Holy savior I think I broke it!" makes Alice a pretty weird and repressed person.
In extremum, normal people curse. I avoid everyday cursing for my characters because I want the extremum cursing to stand taller.
As for racial, ethnic or homosexual slurs: I think it is necessary for some fiction (movies about civil rights issues or racism in America would be pretty ridiculous without racial slurs, as one example), but I don't write that type of fiction, and don't consume it either. That is a personal choice, what I read and write is intended for escape and adventure, I don't need any fictive versions to remind me of real life hatred, abuse and carnage.
Remember the Ad writers maxim: If you emphasize everything, you have emphasized nothing. Now they are talking about bolding, italicizing, caps, boxes, bigger fonts, graphical arrows and badges and such. If the page is littered with them, they are ignored and probably the piece is put aside. if there is ONE big red arrow on the page, people will read what it points to: If there are TEN big red arrows on the page, none of them get read.
Cuss words are intensifiers, but if everything is intensified, none of them have any power. If nobody is cussing and in chapter 7 Alice explodes and says "Fuck you, Bill! Fuck you!" and storms out, that has power. If Alice uses 'fuck' on every other page, that scene has very little power; in fact the only power it has is provided by the context of the scene, pretty much as if the book were written without Alice cussing at all. Which means we threw away our ability to intensify that scene, by creating Alice as a non-cussing character that cussed with vigor in an extreme emotional state.
Anything in dialogue (direct or reported speech) is probably fine if it adds to the character. It also works in first person narration, or where the narrator has a specific character - as you say, it's a character trait.
It also depends how you define offensive language, though this is going to be different for different people. Many will be fine with "curse words", but draw the line at ethnic slurs - these can be tricky even when they help define the character.
If you're thinking in terms of character and context, it sounds like you've got the right idea.
Additional to the other excellent answers is the use of vulgarity for comedic purposes. There are lots of styles of comedy: slapstick, screwball, situational, irony, and so forth. When done right, and in context, vulgarity can lighten the mood of an otherwise deprecating scene, like torture or death, and leave the reader (or viewer) entertained.
And sometimes, such as iconic stand-up comedians like Eddie Murphy, Richard Prior, Andrew "Dice" Clay--it made their careers. South Park, Game of Thrones, and Deadwood, to name a few, do/did just fine with a wide audience.
I've often seen the following strategy used to great effect: if the cursing is relatively mild, it is directly written in the dialog of the character. On occasions it is very harsh, its effects on surrounding characters are described instead ("onlookers were shocked by the sudden outburst of swearing, and its gory and explicit details they don't usually hear every day despite living in a rough part of town")