Write what you want to write: Accept what people want to read into it
This is a balancing act, but ultimately we're in La mort de l'auteur territory here.
Ultimately, readers are going to... read... things into what you have written. Some are going to read "overly" literally and miss your figurative intent. Others are going to reach for innuendo in everything that you, in your own head, meant literally. Subtext will develop. Critiques will form where random things are picked apart and you're left wondering when you even wrote the passage in question, because you don't even recall using that specific phrasing and certainly not with any intent for a particular interpretation of it.
Art has two sides. Any art. Any act of communication. There is the side of the creative experience. And there is the side of the interpretive experience. The more you try to strangle your art into a narrow existence of precise communication, the more sterile it becomes. This works in certain circumstances, but for fictional writing I find it to be generally detrimental.
You need to chose what you're okay with, in terms of what an audience interprets as how they experienced what they read, versus your intent in what you wrote. And you need to understand that if you're not okay with a specific interpretation, maybe you need to not write whatever you believe will lead to it: struggle all you want, get as explicit as can be, and all you've done is call attention to whatever it is in question.
Consider, if you read a passage where an author went out of their way to try to highlight that a banana was merely a fruit, merely meant literally—in a work with other figurative uses for other objects, no less—don't you think you'd find it significant that so much time was spent talking about bananas? Wouldn't it seem to, in its own way, be an act of emphasis? Wouldn't you question whether the author's narrative acts in regards to the banana might not have aspects of deception, in terms of trying to claim it is only meant to be a banana?
You can't win this, and you're the one starting it
People are going to read what they want to read, and their basis for that is what you put in front of them to be read. Writing is communication, and communication is always interpretive between the parties involved. Language alone is interpretive: we used a shared basis for meaning, but even with supposed authorities we still develop different interpretations of individual words, much less the nuances of those words in shifting contexts.
If you're using an item in a scene which has widely known and widely used alternate connotations, puerile or not, your only even marginally safe road if you are concerned is to remove it entirely, substitute something else for it.
One question I would have is, why is there even, explicitly, a banana at all? Why not just "a piece of fruit"? I love detail, but ultimately you're the one in control of what you write with detail and what you don't: and you can't escape the fact that the more detail you give, the more questions of whether there is a reason for that detail will occur in your audience. You put in the assumed effort and care to write it, why wouldn't someone think there's significance to the specific choices in what you wrote?
If you can't beat them, join them?
Sometimes the best foils for defusing something are your own characters. Rather than refuse the interpretation of readers, acknowledge that it will happen by letting your own characters engage in it.
The easiest way to control a narrative, inasmuch as one can, is to own it.
Many people make phallic metaphors out of bananas. They make jokes about it. They tease each other about it. Some people find this exceedingly juvenile. Some court it. Some are oblivious to it.
So rather than run away from this, use it. Your intent might not be to have your use of bananas have any lingering other meanings, but outside of literature, when has that ever stopped anyone from doing so?
Why do you expect it to be different with what you write?
How would your characters, if they were real people, actually act in this situation? You can still speak through them, at which point it's easy enough to have someone pick up on the metaphoric aspects and run with it—teasingly or more circumspectly—and someone else react, perhaps by becoming annoyed that they can't even "eat a banana in peace" (which would undoubtedly get its own reaction, leading in turn possibly to something like an exasperated facepalm and a request to just drop it or something similar).
If you're making choices in wording or scene that you know readers are likely to read into in particular ways beyond your specific intent in writing them, you need to accept that among other things, this is possibly a warning that your characters would also catch on to these metaphors, and skipping over them, particularly with characters developed in ways that would be more likely to pick up on the common metaphorical aspect and especially turn that into some kind of interaction (joking, etc) may actually draw more attention to that, and it will be in ways where you don't even get to have a say about how you feel in regards to that, through how your characters interact in that circumstance.
As a final note, to fall all the way back to Shakespeare, consider the use of (and then consider the ongoing layers of cultural connotations in relation to):
The lady doth protest too much, methinks
There are a number of answers that try various approaches in fulfilling your intent through trying to control or expand the surrounding narrative to try to explicitly set it in a non-figurative direction. I would argue that none of them will work with all of your audience, and most of them may actually exacerbate the situation you perceive as an issue, with the readers who are most likely to make this interpretation to begin with. Even my latter suggestion here (which is similar to @Sara Costa's excellent answer) of just letting your characters naturally play with it (*as she winces at her own wording there*) is going to have readers who still run wild with whether there's some further subtext going on about the sexuality of the characters involved or similar questions. That's part of why I worded the heading for it the way I did.
Pick your battles. If you don't want to have a struggle over a banana, don't stick a banana in it to begin with; or accept that, to some people, that banana is going to mean more than just a piece of fruit. Just know that the more you struggle to push that away, the more people will then find other reasons to read into it, and it still won't stop the readers for whom a banana is always a source of amusing metaphor whenever they encounter one in their lives, nor will it stop those who run wild with conjecture about just why a banana, in particular, was put precisely where it got stuck.
A piece of fruit
But as a final answer to your last question of "can a banana ever be just a banana", I leave you with a question in turn:
As an element of language, what is a "banana"? Actually?
You say it's a piece of fruit.
I say it's just a visual representation (using a set of symbols) of a phonetic structure that is commonly itself used as a symbol to represent a specific "type" of fruit (which in turn varies on how that is interpreted), which has a certain (general) visual to it (however, even the color is not a sure thing), which in turn is similar to other things visually, and of which that visual association is common in certain societies. What makes that phonetic structure a symbol referencing that piece of fruit? Nothing except our common acceptance that it does and further authoritative attempts to declare that this means that it has that meaning. So how does that differ from other symbolic aspects of it? If there's a common phallic association in a given society with a given item for which a given word is associated, how is that not effectively part of its living colloquial definition as a connotative aspect? As humans we run on symbols and patterns, our language (any language) is itself merely sets of symbols: you can't escape symbolic interpretation.
A banana is always just a banana. The error would be in assuming that means it's just a piece of fruit. A banana is always just a banana for however each individual interpreting the word into associated meanings individually perceives it to have meaning. Language is an attempt to form an agreement on the commonality of shared meaning such that intent can be transmitted and interpreted with some small degree of shared basis, but ultimately every piece of language and communication is interpretive based on how we have individually formed related symbolic associations.
It's important, I think, to flip this from a context derived from a conceit of authorial control and perception to being instead about the fact that we are all simply absorbing and interpreting symbols, and therefor as an author your own conception of those symbols is simply that: your own interpretation, individually. Ideally it's at least very close to how many readers will interpret it, but the act of writing is its own individual interpretative act, and even attempts at precision will only go so far, and will carry their own associated interpretations themselves.