If this is not an academic publication and you are not bound by the more severe citation styles like APA or MLA, a common way to reduce distraction in popular non-fiction is to have no in-text references at all and append endnotes to the end of the text that are ordered "chronologically" and give page numbers and text snippets to identify what they relate to.
E.g. this text on page 216:
It has been shown that women are smart. So we must allow them to go to university.
can have an endnote something like this:
216 women are smart Clever, W. (1753). Experiments on the Cleverness of Women. Birmingham: Clever Woman Press.
There are many variants of how to denote what text the citation refers to, like paragraph numbers, but the most common one I have seen is simply quoting the phrase.
This citation method assumes that certain textual markers (such as "it has been shown" or "we found") will signal to an informed reader that in academic publication such a claim is backed up with sources, so the reader can expect to find a source in the endnotes. This means that just as in an academic publication you have to be careful how you phrase your text and make no claims that you cannot back up.
How you style these endnotes (bold, intalics, spaces, etc.) is up to you or your book designer. I would mark up the actual sources according to one common – and relevant to your field – academic style (APA for Psychology, MLA for Language and Literature, etc.).
It is also up to you whether you prepend a short note to your text pointing out this citation style, or whether you let your readers leaf through your book and find the endnotes by themselves. I have seen both, and I think that educated readers will expect some notes and look for them without being told.
Regarding your question after a precedent of legitimacy
As I said, this style is common. I found it in many popular science books by eminent scholars, and actually do not currently remember an example that does not use this style.
The following illustrations are an example (the first page of text and the first page of the notes) from the book Willpower by psychologist Roy F. Baumeister, one of the most cited scientists alive.
These illustrations were downloaded from the publicly available Amazon.com "Look inside!" preview of this book and are provided here in the intent to educate and therefore fall under fair use.