To answer your question, you should start writing as soon as you have completed your story arc. If you can write how many "chapters" you will write and what will be their content, you can start whenever you feel like it.
You should continue reading this, though:
If I take too long, am I at risk of losing my memories
You may not be taking enough notes if you're at risk of losing your memories.
Writing about something which is based in existing literature, like non-fiction or mythology (without re-inventing it), is hard work. And the keyword, here, is work. In this field, having 4 books as source is not a lot. Of course, you may have your favorite works upon which you're basing the majority of your works, but you will probably end up diversifying your source much more than this.
Which brings me to the following point: don't just take note. Take references. When you note something on interest, also write where it's from.
This is important for many reasons, the first being that at some later time you may want more context on these notes you're taking.
Another good reason to note sources is that, over time (and it's probably already happening with only 4 books), you'll realize that some sources will give a different account of the same events. And, at some point, you will have to choose which one you'll use. There are many ways to do this. You may just use the author you trust the most, or the narrative which is the most dramatic. Or anything that suits your agenda, really. It's your book, after all.
To choose which material you will use is way harder than it looks, because these choices will have a direct impact on your story. And, as reality has no need to justify itself, sometimes you will have to chime in and fill in the gaps.
And that is, as far as I'm concerned, one important job. It's both hard and rewarding... unless you want to stick closely to the facts - then stay away from the narratives which would let you too much rope!
Good work, and have fun!