Is the "genre" a piece of fiction belongs to determined by the author, or by the editor and publishing house? Is there an established definition for each genre, or is it largely ad hoc?
I ask because the Writer's and Artist's Yearbook gives contact details for many publishers, sorted by the type of genre they will publish and/or are known for...
As far as I can tell, what genre your work is published in is usually up to the publisher, as a tool for bookstores and other retailers to label and categorize your work.
I've seen a lot of arguments about particular books being labeled in a specific genre and readers not really considering it to fit there. An example would be "The Sparrow" by Mary Doria Russel, which was published as 'sci-fi' and usually found in that section, but I've heard many impassioned arguments for it to be placed in fiction/literary fiction.
Fantasy/Sci-fi seem to have a lot of books that are both, so you tend to see books with both labels on them.
As for submitting your work publishers, it might be best to see what books they've labeled as what genres in the past. That might give you a better idea whether or not they'll consider your work as part of the genre that you intend it to be, or if they might consider it something else. (For example, if they consider your Steampunk novel to be sci-fi, or fantasy, or speculative fiction, or historical AU.)
quoting wikipedia:
Genre ... "kind" or "sort", from Latin: genus (stem gener-), Greek: genos, γένος) is the term for any category of literature, as well as various other forms of art or culture e.g. music, based on some loose set of stylistic criteria. Genres are formed by conventions that change over time as new genres are invented and the use of old ones are discontinued. Often, works fit into multiple genres by way of borrowing and recombining these conventions.
So genres evolve as they come and go, and based on current cultural state - we could say there is a current established definition.
I read this really good definition by Chris Sunami:
Genre should be seen largely as a way of connecting a writer with the audience most likely to enjoy his or her book based on elements shared with other books.
I read it about 2 minutes ago here: What is the most important characteristic of New Weird as a genre?.
So if you're not sure what to call your book's genre:
You almost have to assign your book a genre in order to describe it to an agent or publisher; the publisher will likely make the final determination of what genre they assign internally, and this can differ from publisher to publisher.
As said before, genre is a marketing tool, it helps agents know which publishers to approach, and publishers know how to market a book. Neither of them will blindly accept what you tell them; they will read your book and make their own judgment.
Because genre is a marketing tool, it has to be very general. It is talking about a kind of story, not your specific story, so it is describing story elements that many readers love to read about. Magic in a Fantasy World, or Magic in the modern world, or various kinds of space-based adventures, etc.
Due to pinpoint marketing (by Facebook, Google, Amazon and others) genres are getting more specific as new target groups are found. Don't expect to be an expert in this. Stick to genre descriptions you can find online, and leave the marketing to people that spend all day, every day, figuring out how to sell books.
Agents/Publishers are likely to cut you a little slack on what you choose as genre; they don't expect you to be a book marketing expert. That's their job. They only expect you to be an expert storyteller and writer.
That said, you should still try to get close, because the genre you claim in your query may be used as grounds for summary rejection. If you claim "X" the agent, knowing the market, may think "X" is a hard sell in this market, I'll pass.
If you claim "Y" the agent may think "Y", excellent, let's see the sample -- Wait, this isn't "Y", this girl doesn't know what she's doing. I'll pass."
I don't think you need to read a bunch of books and say your book is like A, D, and G. At least I don't have time to read dozens of books looking for one like mine.
I think you should be careful and read the genre's themselves, what story elements each genre includes, and pick the one that is closest to your story. That way, the expectations of the agent/publisher won't be far off from the reality of your story, and any refinement they make will actually be just a refinement, not a wholesale reassignment.
Find the elements of genre on the Internet. Agents (for example on Manuscript Wish List (aka MSWL) or Wikipedia's List of Writing Genres). You can also find on MSWL agents interested in your particular genre; and by reading their profiles, you might find even more specifics about what they are looking for.
It's work. A day or two, if your genre is not obvious to you. But don't stray too far into the specifics of story details. Just because your story contains a romance doesn't make it a Romance. Just because it contains an explicit sex scene doesn't make it Erotica. Just because it has a homosexual character doesn't make it LGBTQ+ fiction. You might want an agent that doesn't mind representing erotica or LGBTQ+, but don't claim it as your genre if being homosexual is not central to the story.
Some things listed as genres actually are not completely about story elements; "Children's", "Young Adult" and "New Adult" are more about age groups and content. They need to be supplemented with a genre mostly concerned with story elements. So "New Adult Urban Fantasy" tells you the story appeals to 18-30 year olds, likely contains language and scenes not appropriate for minors (or the topics explored are not relatable or appealing to minors), and is magic in a modern city.
Good enough! You aren't trying to pinpoint your story by listing a bunch of genre labels. That is counter-productive, the more you list the more confusing the sales picture gets. Pick a lane and drive in it, give the experts a good general idea of how it would be sold and let the experts deal with the pinpoint definition of the audience after they decide your story is compelling and can be sold.
Also, a query letter is a very tight box within which to describe your story. The less space you take up trying to pinpoint your exact genre, the more space you will have to sell the story setup (first Act) itself. Even a few more words can be crucial. So give it your best shot within about half a line of text.
