A Cult of Quantity
One of the themes that I’ve seen within the skeptical community over time (and, more specifically, in recent discussions about race (here and here) and gender (here and here and here and here and here)) is the hyper-skepticism about social issues and a hyper-rationality when seeking to understand social issues. It often comes in the form of JAQing, typically from people who “just want data.” What they often (but not always) mean by this is that they want specific, quantitative, peer reviewed, top-tier journal published research on whatever specific topic is being discussed, and if that type of research doesn’t exist, then the problem is moot.
In this post, I’m going to address a few of the common arguments advanced by some members of the skeptic community that over-depend on quantitative data and further seek to minimize the usefulness of qualitative data.
Let’s start with a few arguments that I’ve seen around the skeptical blogosphere.
Argument: “The plural of anecdote is not evidence/data.”
Response: That depends on the question being asked. For example, if your question is “how many people experience racism/sexism?”, then anecdotal data are not helpful because you’re seeking a quantity.
However, if your question is “what are some of the ways that people have experienced racism/sexism?” then anecdotal data is absolutely helpful. The answer to this question could certainly be compiled to put a percentage on the ways, but the question itself isn’t focused on quantity but quality.
“But Will,” you might be thinking, “how can we start to fix the problem of racism/sexism if we don’t know the scope of the problem?”
Argument: “We cannot advance a solution unless we know the size and scope of a problem.”
Response: The only thing we learn from quantitative data on the scope of a problem is how widespread the problem is. It tells us very little (if anything) about why the problem is occurring. Qualitative research, on the other hand, is much more useful in helping us to understand why certain behaviors occur, because it utilizes much more focused data collection methodologies. What I mean by “focused” here is that qualitative research seeks a narrow depth whereas quantitative research seeks a shallow breadth.
What this means is that multiple qualitative studies are important for understanding the underlying causes of problems more broadly. In other words, we need more than one qualitative study to be able to generalize. Conversely, we can start with a quantitative study and then move to qualitative data collection to give some depth to the generalized statistical data.
Argument: “Humans are flawed and thus anecdotes and testimonials are not completely accurate; therefore, anecdotes are invalid and tell us nothing.”
Response: This confuses accuracy with reliability. Anecdotes/testimonials can be quite reliable and informative, even if they are not completely accurate. They can paint a personalized and specific picture of a problem that is otherwise overlooked in quantitative data. This can help to humanize social issues (e.g., look at how much American culture has changed with regards to gay and lesbian people over the last ten years—it’s due mostly to the visibility of queer people sharing their lived experiences, not because some statistical dataset was pushed into the public sphere).
Accuracy and reliability are also issues in quantitative, statistical data (obviously, since statistical data is always accompanied with margins of error). But we don’t dismiss a quantitative study because it had a 95% accuracy rate. That would be pretty damned reliable, despite not being completely accurate.
The problem as I see it is that much of the skeptic community highly values quantitative data over any qualitative data regardless of the research question being asked. This is a fundamentally flawed way of approaching research. It seems to me that this comes from an over emphasis on objectivity such that many skeptics feel that they can escape subjectivity. But this is not possible. We are temporally and materially situated beings living in the world. By necessity, we cannot be completely objective, as this would require what Donna Haraway refers to as the “god trick.” That is, we would have to be all seeing and all knowing in order to achieve true objectivity. Our phenomenological situatedness means that we cannot achieve true objectivity.
I point this out because I often encounter people who identify as skeptics that claim to be objective, as if they were somehow able to jettison their situatedness (subjectivity). It is a pervasive thought in the skeptical community that we are somehow above the cognitive errors so common to the rest of humanity, and the irony of this type of thinking is completely lost on those who subscribe to it.
There’s nothing wrong with subjectivity. It’s an important and vital aspect of human existence. We should pay careful attention to our subjective biases and be up front and open with them as much as possible—this includes having them pointed out by others.
See, the important part of skepticism to me comes in the critical thinking. It comes in the recognition that we are all humans who live in an imperfect world because we are imperfect. Once we recognize this and cease to pretend that we can rise above this imperfection, we can begin to work with and around our subjectivities. Though we will never eliminate it, we can (and should) seek to be critical of and minimize our biases in our data. And we can’t do that if we think we’re already objective and free of bias.
So, my plea to my fellow skeptics is this: don’t be afraid of the anecdotal or the subjective, and be skeptical of the idea of pure objectivity. This doesn’t mean we must jettison objectivity or that we should only rely on subjectivity, it just means we must use the right methodological tools to answer the different types of questions we have about the world.