# Metrics for assessing the persuasiveness of a paper?

Are there any known variables to measure, or questions to ask in survey, in order to asses the persuasiveness of a paper?

For example, I'm not sure if omitting a section from my paper might actually improve it. It's good writing and a good point. So I'm wondering how to take the subjectivity out of the decision so I know I'm making the right choice.

I was thinking I could do an A/B test - I could prepare both versions of the paper, and randomly give one to readers who volunteer to participate in a survey after reading. What kinds of questions could I ask, which would allow me to compare the answers and identify the most persuasive form the of paper?

I think a survey would be easiest for me to execute, but if it's not a good way to decide this, is there something I could look for as the reader is reading? Maybe, track eye movement and see what percent they skimmed? Anything along those lines?

How can I tell which of 2 versions of a paper is most persuasive?

I could ask "On a scale of 1-5 how persuasive was this paper?" And then compare the average rating from each group.

• Anyone know a more relevant tag for this question? – Dingredient Jul 10 '13 at 16:52
• Re: Tags - Does this relate to academic, business, or tech writing? – Neil Fein Jul 10 '13 at 18:39
• Re: Re: Tags - I added tags for tech and business, thanks for the suggestion! – Dingredient Jul 15 '13 at 17:36

"Persuasiveness" is highly subjective. I can't imagine how you would measure it other than to perform an experiment with real people.

You didn't say what the subject of your paper is, which is probably good because that helps us to discuss the question of persuasiveness without being biased by whether or not we agree with you.

I know of three basic approaches to measuring persuasiveness:

(1) Get a group of test subjects. Have them read the paper. Ask them to rate how persuasive it is on some scale, 1 to 10 or whatever.

Catch: People who already agree with you will likely say that it is very persuasive, while people who disagree with you will tend say it is not persuasive at all, no matter how compelling the actual arguments are. You may well be measuring how many people agreed with you before reading the paper then anything about the paper itself. Often people disagree not over the facts themselves but over which facts are important.

For example -- and I'm going to use a real example here because it's hard to make this point purely in the abstract, but I'm trying to concentrate on the meta-issue of persuasiveness and not any particular issue -- so for example, suppose you were arguing against legalizing marijuana, and so you gave evidence that marijuana is bad for people's health. Someone who was pro-legalization might well say that even if true, this is irrelevant and therefore unpersuasive, because people should have the freedom to decide for themselves whether the perceived benefits outweigh the harm. On the other hand, suppose you were arguing for legalizing marijuana, and you pointed out how many people were jailed or lost their jobs or whatever just because they were caught smoking. Someone who was anti-legalization would likely reply that this is unpersuasive because the whole idea of making something illegal is to impose sanctions on people who break the law so that people will be afraid to do it: if these laws didn't hurt people, then they would be ineffective. Etc.

(3) Get a group of test subjects. Give them a pre-test asking their opinion on the subject matter. Then have them read the paper. Then give a post-test. See how many changed their minds.

I think this is the most effective method because it directly measures what you are presumably trying to accomplish: changing people's minds.

Catch: The more controversial the subject, the fewer people will change their minds. With a small test group, the numbers may be so small that they are not statistically significant. People's opinions about controversial subjects are built up over the course of a lifetime, from many many individual facts, ideas, experiences, etc. If a person believes that, say, political position X is good for the country, he likely has hundreds of reasons for thinking so. If you present him with a reason to think it's bad, his first reaction is likely to be that your argument MUST be flawed, because he KNOWS that X is good. Even if you convince him that your argument is valid, it's still one argument against versus a hundred arguments for. He's likely to see it as a curious anomaly, not a game-changer.

So no matter how valid and persuasive your arguments are, any effects may be very subtle, and thus difficult to measure. There may be some number of people who were undecided and who you managed to bring into your camp. But likely there are many more who were previously 80% sure that X was right but who are now only 79% sure. Etc.