A new study finds gender differences in learning, using a methodology guaranteed to maximize gender differences. Bad science in action.
There was a study recently about how people perceive science, particularly erroneous scientific information, when it’s presented in the form of narrative. It came up with one of those gender dichotomies that bad science journalists love, so it’s getting a little attention from that. Unfortunately, a look at the actual experiment shows that they weren’t running the experiment they thought they were, and consequently their results are functionally meaningless. And I can prove it.
First some background. The full study is here, or if you’re in a rush, Colin Schultz has a nice article about it here.
Here, in the experimenter’s own words, is what they thought they were testing and what they believe they found:
This experiment examines the influence of narrative transportation, role of science within the movie, and gender of the viewer on evaluation of incorrect scientific information in fiction. Results show that incorrect science facts accepted as true after seeing identical segments from movies depend on the gender of the participant and a manipulation of the perceived centrality of science to the plot. Men tended to detect more inaccurate science facts when they thought science was central to the plot. Women detected more inaccurate science facts when they thought science was peripheral to the plot, which was presented as a relational story.
Here’s a concise breakdown of how that actually functioned, from Mr. Schultz:
To test their idea, the researchers presented the same film clip to the test subjects in one of two ways, with either the science, or the interpersonal relationships between the characters, being played up.
…
Barriga, Shapiro and Fernandez found that when the science was presented as the main focus of the clip, men caught 3.23 +/- 0.28 (out of 8) of the errors, while women found 2.46 +/- .30 of the flaws. When the participants were told the story was a tale of love and lust, with lasers filling in as the sidekick, men detected 2.68 +/- 0.28 scientific inaccuracies, and women dismissed 3.14 +/- 0.31 of the errors. The men and women tested caught about the same number of errors, it just depended on how they were presented – even after controlling for things like general science knowledge and level of education.
Now, first off, that gender difference is not huge; it may well just be statistical error in a hat. Gender is the only variable anywhere in the study which shows anything approaching statistically significant variance, which suggests that the researchers seized on it because “our study didn’t find anything at all” doesn’t make for a very compelling paper.
But even assuming statistical significance, even granting, rather generously, that their findings are valid, the result is still bullshit because their methodology is so bad it’d produce a gender difference no matter what the actual content was.
The human mind suffers from what are called priming effects, where unconscious associations cause us to make decisions differently, and in particular, if we’re reminded of a role we have, we’ll behave more in accordance with the stereotype of that role. So the only thing this study was actually testing was whether men would pay more attention to a clip they were told was an action movie, and whether women would pay more attention to a clip they were told was a romance movie. All that stuff about science and learning styles and narrative centrality became irrelevant the minute they primed their subjects with a gender stereotype. The only thing they were testing was priming effect, and surprise surprise, they found out it works. And then reported that they found something else, which is just plain embarrassing.
Some might be skeptical, but this kind of priming effect is depressingly well-documented. One study (PDF link) showed that reminding women that they’re women right before giving them a math test produced worse results, because the subjects started unconsciously running the stereotype that women are bad at math. Another one found they could reverse that effect with Asian-American women, by reminding them of their Asian heritage instead of their gender, thus activating the stereotype that Asians are great at math.
Remember this the next time you see a study about how men and women are soooooo different, amirite? If it’s getting reported, it’s because it supports popular stereotypes, and there’s a good chance that those same stereotypes are all that that study’s finding. A neat little cycle, but not one that tells us anything about any reality behind the assumptions.
Photo—jonfeinstein/Flickr
The post Here’s How Gender Bias In Social Science Works appeared first on The Good Men Project.
