No, no one expects you to cite sources in casual conversation. Nor are you expected to give citations in other casual communications, like a text message or a brief email to a friend.
The only time you need to cite sources is in scholarly writing. It may be considered appropriate in other published writing or media.
Perhaps it would be helpful if I point out the difference between plagiarism and copyright violation.
If you fail to cite your sources in a scholarly paper, that is plagiarism. Plagiarism is an academic violation. If you do it, you can be embarrassed when caught. You might lose your job or find yourself banned from academic publications. But you can't be sued or jailed for it. It's not a crime.
Copyright violation is copying someone else's material without permission. Legally, it is a violation of property rights, like stealing or trespassing. You can be sued for copyright violation.
Note that you could be guilty of plagiarism but not copyright violation, or vice versa.
If you copied fifty pages from someone else's book into your own with no indication that it was not your own work or where it came from, that would be both plagiarism and copyright violation.
If you put it in quotes and gave a proper footnote, it is no longer plagiarism. You have identified your source. But it would still be copyright violation.
If you took fifty pages from someone else's book, reworded it into your own words, and claimed it as your own, that would not be copyright violation. You have not copied someone else's words. But it might well still be plagiarism, as you have copied their information and ideas without giving credit.