What Fresh Hell Is This?

June 14, 2014

How NOT To Do It

Ok, so my weekly issue of The Nation arrived yesterday   The cover in blaring red and white reads "1 in 5 Women Is Sexually Assaulted in College" and so I immediately turned to the cover story.

The writer, Michelle Goldberg, wrote a very important piece about a very important issue, without a doubt.

But she made a mistake.  Perhaps not a big mistake, but perhaps a mistake big enough so that someone inclined to disagree with her outright might use it to discredit her general argument.  The mistake's in this paragraph:
As months of harrowing headlines have made clear, the dimensions of that crisis are staggering. According to an April report of the White House Task Force to Protect Students From Sexual Assault, “One in five women is sexually assaulted in college.” This figure, which comes from the National Institute of Justice’s 2007 “Campus Sexual Assault Study,” has been much disputed by conservatives, but according to a detailed analysis by PolitiFact, “the overall findings in the study were on par with similar surveys conducted over the years that have measured sexual assaults on campus.”
Here's the 2007 report and here's Politifact's analysis of it.  And that's where Goldberg errs.

Whereas it's absolutely true that Politifact says (twice, in fact) that the "overall findings...were on par with similar surveys..." and so on, the analysis also says (and this is the sentence immediately preceding one of the times it's quoted):
Experts we spoke with said that while that statistic is commonly used, the source, a survey of just two colleges, may not be representative of the entire population.
Leaving that part out is the mistake as it misrepresents the entire analysis.

But why might not the study be representative?  Here's why:
[T]he Web-based survey yielded a relatively small response rate of about 42 percent, which the researchers note is lower than other methods, like face-to-face interviews. They hoped, however, that anonymity provided more candid answers and better data.

Additionally, as we noted, only students at two large universities were included in the survey. Experts we spoke with said this is a glaring caveat that makes it difficult to create a national estimate from the results.

"This ‘one in five’ statistic shouldn’t just be taken with a grain of salt but the entire shaker," said James Fox, professor of criminology, law and public policy at Northeastern University.

Large universities may not be representative of experiences at mid-size or small colleges. Further, the two colleges selected may not even be representative of large campuses, Fox said.
Which poses another smaller problem for Goldberg as she opens with this incident as an example of the mistreatment of rape victims on campus:
During her freshman year at Occidental College in Los Angeles in 2010, Audrey Logan says, she was raped on two separate occasions by a young man she considered a friend. Because she knew him and had been very drunk both times, it took a while for her to identify what had happened as an assault. “I really believed rape happened in the dark, by people you barely or don’t know, and weapons or group force were always involved,” she says.
Occidental College describes itself as a "a small, highly selective and diverse liberal arts college in a big city" by the way.

If it's the case that the CSA study, as it's based on web surveys at two large universities, might not be representative of mid-sized or small-sized institutions, then including an illustration from an admittedly small liberal arts college only blurs the lines.  And if the study might not even be representative of large campuses then using it at all only blurs those lines further.

And certainly supporting an argument (even if only in part), by citing a Politifact analysis that doesn't completely support the statistic at the center of that argument is, simply a mistake.

In fact, Mary Koss (author of the 1988 study "Hidden Rape: Sexual Aggression and Victimization in a National Sample of Students in Higher Education") is quoted by Politifact saying:
[T]he Campus Sexual Assault Study "is not the soundest data (the White House) could use."
Needless to say:
"Without a doubt, sexual assault and date rape are of great concern on college campuses," Fox said. "It should not be dismissed. At the same time, we should be careful not to cite national estimates that are shaky, at best."
And I'll leave it at that.


Heir to the Throne said...

It is not a big mistake. Social Science research plays fast and loose with the any assumptions and data. The only criteria needed to consider it valid is if it proves what the researcher believes.

This never happened

Why Not Name the Hofstra Student Who Recanted Her Rape Story?
"I'm not naming the student out of some mix of pity and sisterhood. She has been suspended from Hofstra. "

Maria Lupinacci said...

Let's not forget the biggest offender of muddying up rape statistics: The FBI.

Until 2012, for 80 years the FBI definition of rape was “Carnal knowledge of a female forcibly and against her will.”

Among other things, the old definition left out cases that involved:
- Anal or oral penetration
- Penetration with an object
- Cases where the victims were drugged or under the influence of alcohol
- Incest
- All cases with male victims

Necessarily as a result, many sexual assaults were not counted as rapes in the yearly federal accounting.

So until 2012, if a woman was drugged to the point of passing out and gang raped, it didn't count.

If a woman was raped with a baseball bat, it didn't count.

If a man was raped--any man--it didn't count.

Etc., etc., etc. Talk about fucked up stats...