Comparing two files for similarities not differences (inverse-diff)

Question

I need to compare two documents for common sentences and phrases. I cannot, however, find a tool that does this, and I have tried so many at this point: Word, Acrobat Pro, Araxis Merge, BeyondCompare, WinMerge, commandline tools, and others. I am certain I must be missing something super basic in this quest, so I'm really hoping someone here can help. My goals for this effort are as follows:

• Tool must work with at least two plain text files
• It must be able to compare them and return a list of common sentences, paragraphs, and phrases (this could be a manner of highlighting duplicated content or actually returning a list of common content)

If there is a way to do this with standard diff tools by reframing the diff command, that would be perfect, too.

Answer 1

You need an Ngram analyzer that identifies repeating word sequences.

Here is an online tool: http://guidetodatamining.com/ngramAnalyzer/. This tool allows you do search for sequences of two ("bigrams") to five words ("5grams"). There is a link to the Github page of that project if you want to install it locally (it is a PHP script, so you need a server, but you can set one up locally). You may want to search and see if you can find another implementation, for example in Python, that you can run without having to set up a server, or one that searches for sequences of different lengths at the same time.

Searching for trigrams in the body text of your question returns these results:

Using the Ngram analyzer, you would paste the two texts you want to compare into the same text document (or into the same input in that online tool), because this tool finds repeating word sequences within one text. Once you have found the repeating word sequences within your "double text", you can then search for each of the result phrases in your original documents to find where they are.

If you feel prepared to install and work with commandline tools, you can compare ngrams across documents, for example in Python or in R.

Answer 2

I don't know of a tool either.

If you can get the text files to one sentence per line, you can treat the two versions like a software program.

On Linux, use "diff" to find the difference in the files, it searches in a localized fashion to figure out additions,deletions, and changes. If you redirect the output to a third file, the result is a "patch" file, that shows specific changes to lines, or groups of lines, also deletions of blocks, or insertion of blocks. The patch file is usually the minimum number of changes it takes to turn the first file into the second file.

That would be pretty close to the report you want.Knowing how many lines are in the first file, you can find the number of lines that are the same.

You could also edit the patch file, change all the insertions and changes to deletions, and apply that to a copy to find the lines that were the same.

Answer 3

Under Linux, comm -3 FILE1 FILE2 returns the lines in common between FILE1 and FILE2. I am unaware of a visual-comm equivalent. Depending on the output format you want to see, you might need to pipe the output through sed/awk.

You'd probably get better answers on a site dedicated to Linux where you could provide sample input files and expected output. This site is more about the craft of writing and the art of storytelling.

Answer 4

Just use any diff tool. It will highlight the differences line by line so you can visually inspect side A and side B. If you want a file that lists elements (lines, sentences, paragraphs, etc) that are the same you can write a bit of code in one of the many javascript frameworks to have the results of the diff printed into an output file.

If you are looking to ensure something re-written does not include chunks from a source (to avoid accusations of plagiarism) you can use a tool like CopyScape.

A simple text comparison tool like Copy Works Text Compare may work for you.