I guess it is a matter of taste, but how can I be sure?
Some stories can be told in either tense and it won't alter the feeling of them drastically, the difference can be quite subtle. Also, some writers hate writing in present tense while some find it's the only tense that comes naturally to them. So it can be a matter of taste.
However, sometimes a change in tense can have a big impact on a story.
For example, I wrote a literary fiction novel about two women who form a relationship over a long journey. I wrote that in past tense because I wanted it to be a lilting tale, told from the point of view of the older woman as she relayed the story of her experiences on that journey to the reader. A lot of time was spent inside the protagonist's head as she narrated her feelings (which you can do when looking back on an event in the past with hindsight).
Later, I wrote a fast-paced psychological thriller in present tense because I wanted a sense of immediacy. I wanted the reader to experience the thrill of the action scenes as if they were actually happening in the moment of reading. Handled well, it can create a closeness with your reader, the feeling that they're experiencing the journey while you're telling it. What I couldn't do was spend huge amounts of time inside my protagonist's head, because it's not realistic that we spend time in a sort of stasis while we mentally process an event happening in real time. But as it was a fast-paced novel, I didn't want the action to be slowed down by that anyway.
So, think about the bulk of your story. Every story is going to have its quiet moments and its thrilling moments, but think about the pace you want to create overall and rather than going on taste alone, think about what's best for your story. If you're slipping in and out without thinking, it sounds as if you're probably comfortable writing in either.
I don't know that you can be sure. You just have to make the best guess at what will work. Sometimes you only know with hindsight once the novel is done. But don't worry, you can always change it later. I've switched tense on a 130,000 word novel. It didn't take as long as I thought it would!!