Hope the title is clear enough. I'm also looking for any vocabulary related to the act of writing magic ( 'runes' as a magical writing system for example).
To answer your main question. A book containing spells and rituals is a Grimoire. The term is widely used in modern Wicca and other areas.
"Runes" such as the Elder Futhark are alphabets which we tend to think of as used for magic, although that wasn't their only purpose. "Rune" has been used more generically of late. But that is an alphabet or set of symbols - but it sounds like you are looking for a term for the act of writing? Like "inscribing"? I second @chenmunka that the term is "grimoire" for a book of spells and rituals.
Okay, this isn't a writing answer as much as a fantasy/D&D geek answer. This is only a sampling, and looking up synonyms should get you more. Fantasy literature is rich with this stuff, as are games like D&D. just start digging and there's no end to the material. A lot of religious terminology can be applied to magical writing, so this is also a good place to look.
Any old words can give things a magical feel. Magic is old, so scrolls are often described. They aren't printed, so "inscribed" would be how they are written. For example, the Egyptians would inscribe magic spells onto silver scrolls that they wore as amulets. Use ink or pigments, and apply them with a quill or stylus. "Runes" are good as magical symbols, as are glyphs, sigils, seals or even hieroglyphs. I agree that a "grimoire" is a good magic book, as is a spell book or tome and could also be referred to as a codex, libram, manual or folio. Specific spell books may have special names, like the Necronomicon (anything "-nomicon" sounds magical). Many magical things have magical writing on them, like amulets, charms, periapts, phylacteries, talismans or fetishes.