In a lot of books, at the beginning, the authors include quotes from poetry or songs that are relevant to the theme of their story or to their characters' situations. Obviously I would have to ask for permission to use these works, but do I have to pay the authors or writers if I publish? (For clarification, I am asking about situations where fair use and public domain don't apply. I know I can add quotes under fair use, I am asking if I have to pay royalties for excerpts that are not under fair use.)
You sign a contract with the copyright owner. The contract specifies the arrangement.
- Permission to use the quote in your work.
- Permission to use the quote in derivative works (electronic, translations, movies, comics, etc).
- Rights/ownership (more relevant for pieces of work you use that have not yet been published, and for art).
When there is a payment involved, it's almost always a one-time thing for the book. There may be renegotiation for derivative works if the contract allows for it.
My spouse has done contracts like this multiple times to use artwork and other materials. In some cases he hired an artist and now owns everything the artist created for him (especially valuable in the case of the artist that suddenly disappeared mid-work and can't be contacted). In other cases he is inserting an excerpt from another creator and has a contact to purchase the piece or the use.
Always always put this stuff in writing. The only person who has contributed to my spouse's work who he does not have a written contract with is me. Maybe that will turn out to be a mistake later on, but there are at least existing laws that protect us both in the case of dispute, separation, or death, since we're legally married.
If your excerpt is essential to your work, get permission early on. If it would be nice to have but you could substitute something else, let your publisher deal with the legalities.
If you self-publish, early is better in many ways, but has the disadvantage of possibly shelling out money for something you never use. If you do get an early contract, make it flexible, in case you end up not publishing what you're working on but want to use the piece in a different work.
But to answer your direct question, no, you would not pay royalties. Unless you or your publisher create a contract that specifically spells that out. Which would be odd, unless the excerpt is substantial.
If you are using short exerts and acknowledging the source, you don't have to ask permission or pay royalties (pictures are different). A quotation from a poem or a novel at the beginning of a chapter -- quite common -- is fine and you wouldn't pay royalties for it.