How NOT To Do It

Ok, so my weekly issue of The Nation arrived yesterday   The cover in blaring red and white reads "1 in 5 Women Is Sexually Assaulted in College" and so I immediately turned to the cover story.

The writer, Michelle Goldberg, wrote a very important piece about a very important issue, without a doubt.

But she made a mistake.  Perhaps not a big mistake, but perhaps a mistake big enough so that someone inclined to disagree with her outright might use it to discredit her general argument.  The mistake's in this paragraph:

As months of harrowing headlines have made clear, the dimensions of that crisis are staggering. According to an April report of the White House Task Force to Protect Students From Sexual Assault, “One in five women is sexually assaulted in college.” This figure, which comes from the National Institute of Justice’s 2007 “Campus Sexual Assault Study,” has been much disputed by conservatives, but according to a detailed analysis by PolitiFact, “the overall findings in the study were on par with similar surveys conducted over the years that have measured sexual assaults on campus.”

Here's the 2007 report and here's Politifact's analysis of it.  And that's where Goldberg errs.

Whereas it's absolutely true that Politifact says (twice, in fact) that the "overall findings...were on par with similar surveys..." and so on, the analysis also says (and this is the sentence immediately preceding one of the times it's quoted):

Experts we spoke with said that while that statistic is commonly used, the source, a survey of just two colleges, may not be representative of the entire population.

Leaving that part out is the mistake as it misrepresents the entire analysis.

But why might not the study be representative?  Here's why:

[T]he Web-based survey yielded a relatively small response rate of about 42 percent, which the researchers note is lower than other methods, like face-to-face interviews. They hoped, however, that anonymity provided more candid answers and better data.

Additionally, as we noted, only students at two large universities were included in the survey. Experts we spoke with said this is a glaring caveat that makes it difficult to create a national estimate from the results.

"This ‘one in five’ statistic shouldn’t just be taken with a grain of salt but the entire shaker," said James Fox, professor of criminology, law and public policy at Northeastern University.

Large universities may not be representative of experiences at mid-size or small colleges. Further, the two colleges selected may not even be representative of large campuses, Fox said.

Which poses another smaller problem for Goldberg as she opens with this incident as an example of the mistreatment of rape victims on campus:

During her freshman year at Occidental College in Los Angeles in 2010, Audrey Logan says, she was raped on two separate occasions by a young man she considered a friend. Because she knew him and had been very drunk both times, it took a while for her to identify what had happened as an assault. “I really believed rape happened in the dark, by people you barely or don’t know, and weapons or group force were always involved,” she says.

Occidental College describes itself as a "a small, highly selective and diverse liberal arts college in a big city" by the way.

If it's the case that the CSA study, as it's based on web surveys at two large universities, might not be representative of mid-sized or small-sized institutions, then including an illustration from an admittedly small liberal arts college only blurs the lines.  And if the study might not even be representative of large campuses then using it at all only blurs those lines further.

And certainly supporting an argument (even if only in part), by citing a Politifact analysis that doesn't completely support the statistic at the center of that argument is, simply a mistake.

In fact, Mary Koss (author of the 1988 study "Hidden Rape: Sexual Aggression and Victimization in a National Sample of Students in Higher Education") is quoted by Politifact saying:

[T]he Campus Sexual Assault Study "is not the soundest data (the White House) could use."

Needless to say:

"Without a doubt, sexual assault and date rape are of great concern on college campuses," Fox said. "It should not be dismissed. At the same time, we should be careful not to cite national estimates that are shaky, at best."

And I'll leave it at that.