There are two ways religious concepts appear in speech.
First, there are common expressions: "Oh my god", "go to hell", etc. Those are a natural part of our speech, we hear them all the times and do not give them much consideration. An agnostic or an atheist is likely to use them the same way, without giving them a second thought. You can use those freely. A person could make a conscious effort not to use such expressions, but that would be an exception rather than the rule. (For example, as a Jew, I make a conscious effort not to use "Jeez" in everyday speech. I recognise that it is more natural to use the word, and that my choice is an active one.)
Second there are free associations. Those are not common expressions, but images born of one's mind. "Like the stilts of heaven" and "like the voice of a kindly god" come under this category, I believe. The problem here isn't the mention of religious elements. But searching for an association, a descriptive image, I do not think an agnostic or an atheist would use those particular images; they'd use other images instead. Consider: I live in Israel, I have only ever seen snow a couple of times in my life, while visiting Europe. I can theoretically describe someone's eyes being "blue like an iceberg" - there's no reason for me not to mention icebergs, but it's more likely that my first thought would rather be of the sea on a summer day. The iceberg is not a part of my day-to-day, thus it is not a part of my internal imagery. Similarly, religious concepts (god, heaven) would not be part of the atheist's internal imagery. (Note that I do not disagree here with @Amadeus: a person can use whatever imagery they find convenient. I question the likelihood of a character choosing this language over alternatives.)
On a personal note, "the stilts of heaven" and "the voice of a kindly god" evoke nothing in me: I have no image of a heaven that has stilts in it, nor any idea what a god, kindly or otherwise, would sound like. This could serve as an indicator for you that someone who is not Christian might not use those particular associations.