1

I'm searching for a tool which finds and lists identical phrases in a long text like a dissertation.

The goal is to find repeating texts which have been created accidentally by copy/paste.

It necessarily has to be an offline tool, as I don't want to use an online tool, where my text is processed and possibly stored on a server under someone else's control.

Improve this question
3
  • 1
    Possible duplicate of Word Clouds and Phrase Clouds. Look at my answer under that question (writers.stackexchange.com/a/14302/5645).
    – user5645
    Commented Dec 11, 2014 at 8:54
  • @what: Thank you - this other question has the same target. However, I don't want to use an online tool, where my text is processed and possibly stored on a server under someone else's control.
    – MostlyHarmless
    Commented Dec 11, 2014 at 9:40
  • 1
    I think plagiarism checkers can also find duplicates within one file, but you'll have to try. Other than than, you can write your own tool (code is available online), or if you want to check for complete sentences only, use regular expressions in a text editor that supports them (like Vi(m)) or from the command line (e.g. using awk), again examples are online.
    – user5645
    Commented Dec 11, 2014 at 10:09

3 Answers 3

Reset to default
2

It depends how much you want to get into it and how big the project is. If it's finding only identical phrases or text, most word processors support this, and finally a lot of programming ide's support regular-expression searching (http://en.wikipedia.org/wiki/Regular_expression). On the other hand if you are looking to find reworded or similar paragraphs and/or excerpts and want to automate this (rather than substringing and searching for variations manually), than I would recommend you look at http://www.nltk.org/

NLTK is a toolkit that integrates a whole range of language manipulation, sorting and tagging tools - I realize it may be more in depth than you want to go (but it is quite easy to learn and has a very good beginner friendly documentation http://www.nltk.org/book/).

I'll let others comment on more generic and perhaps more 'use out of the box' type tools, as I have no experience with those.

Improve this answer
2

This is one of the main functions of the program, ClicheCleaner, which highlights passages in your text that are either cliches, other overly-used common expressions, or phrases of your own that you have repeatedly used within the same document. ClicheCleaner includes a list of nearly 7000 unique cliches and common expressions that are compared against your text.

Currently it only works on text files; a new release currently in progress will allow editing of text, Word, and PDF files within the document.

It runs on all versions of Windows.

Disclosure: I am the author of this program.

Improve this answer
2
  • Thanks for this information (and for being clear about your relationship to the product). Does it allow you to add to the dictionary of cliches too? If I know that I have a bad habit of using a certain expression that isn't one of the 7000 it already detects, can I add that?
    – Monica Cellio
    Commented Dec 18, 2014 at 21:42
  • @MonicaCellio Currently you can't add to the list of cliches, since the original dictionary is highly compressed -- for example, when checking for the cliche "lose one's shirt," the various verb forms lose, lost, and losing are all checked, as well as his, her, my and your being substituted for "one's". That's 15 different versions of just this one cliche. But in the next version, I do plan on adding this capability, however the user will have to put in all forms of the cliche manually that they want checked.
    – tcrosley
    Commented Dec 18, 2014 at 22:23
1

As @user3467349 states, regular expressions (regexes) are your (complicated) friend. There's quite a learning curve involved, but it's worth it if you have to do any significant amount of text searching or modification. Many tools support their use (and some use slightly different dialects for their syntax).

If you have access to a Linux system (where it will almost always be installed by default - I expect it must also be installed on OS X, but I don't use that OS) or to a package of tools for your OS (such as Cygwin for Windows), then the place to start is with the grep (global regular expression print) command. It can find almost anything once you know the basics of regexes. There is also a more obscure command, agrep, which specializes in finding things "like" other things, but I haven't used it yet.

Another alternative is if you have or can install the programming language perl on your system (should be available for almost all OSs), it supports it's own dialect of regular expressions which I believe is the most powerful version. It's easy to access the regular expressions feature of perl without knowing a lot about the rest of the language.

If you do have access to a Linux system, then the command

info sed

will bring up instructions for using the sed tool (which you don't need at the moment). But, if you scroll way down in this help file there's a fairly detailed section explaining how to use regular expressions. This will transfer directly to using them with grep as well.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.