Bad Chart Thursday: Leaving Facebook Makes You as Happy as a Cleanse Makes You Healthy

Does Facebook contribute to unhappiness, an unsatisfying social life, and envy? The Danish Happiness Research Institute recently conducted an experiment to find out. This was clearly a rigorous and serious study as evidenced by how they chose to report the results: an infographic pdf–the Internet equivalent of a picture book.

The presentation doesn’t necessarily mean that it is a bad study. It does mean, at least in this case, that we can’t really tell how good it is because there’s just not enough information.

Here’s what we know about the study from the report. The researchers surveyed 1,095 people in Denmark, asking questions about their Facebook use as well as various questions related to moods, social activity and satisfaction, ability to concentrate, and so forth. The researchers then randomly assigned them each to one of two groups. The “treatment” group stopped using Facebook for one week. The control group did nothing differently, continuing to use Facebook as they did before. Both groups were then asked the same questions to see if there was a difference between the two groups as well as any change from their own previous responses.

The report provides no demographic information beyond telling us the participants were all in Denmark. We don’t know anything about age, gender, mental and physical health status, or other factors that might influence how particular individuals in the group might view and use Facebook and how the results of the study’s average Facebook user might apply to individuals. We also don’t know where participants were recruited from.

To be fair, though, they needed room in the report for this super useful chart:

The study was obviously not blinded, but the blurb about the results on the institute’s homepage express being “surprised by the results,” which were generally that the people who took the break from Facebook reported being happier, less stressed, less lonely, and so forth. The only thing surprising about the results, however, is that the researchers were surprised.

Even with the dearth of information in the report, what’s there indicates that the researchers might have had these results in mind when they designed the study. Simply referring to the group taking a Facebook break as the “treatment” group suggests that Facebook use was seen as a problem to be treated. This could have been just a careless use of two common terms used to describe groups in studies, treatment and control, but it’s one of the ways in which phrasing can even unintentionally bias the conclusions.

Another example of wording potentially skewing the results is in the questions they asked. The study claims to have found an increase in both social activity and social satisfaction in the group that took the Facebook break. The results are charted, with the questions themselves beneath:

First, we see that both groups experienced an increase in social activity and in satisfaction with their social life. The increases are slightly higher for the treatment group, a difference that is exaggerated by starting the y-axis above zero. Most of the charts in the report start above zero, although this is not in and of itself always a bad thing. When all numbers fall within a particular range, and the differences are small, it can make sense to show only the range of actual responses just to see the differences more clearly. Is this what they did? The report doesn’t include any information about the variation or range of responses, so . . . maybe?

Regardless, even with this presentation, the differences between the two groups do not appear to be huge. That both groups reported increases does make me wonder when the study was done, whether the time off Facebook happened to coincide with events, such as holidays, that would have led to people from both groups getting together in real life more often than they had the week before.

But the questions themselves could have produced some or even all of the slightly higher increase in the treatment group. Online socialization is explicitly excluded from the question. It seems self-evident that the people who weren’t socializing on Facebook would be more likely to seek out more social interaction “in the real world.” And even if the offline social life is actually less satisfying than the online social life for the treatment group, we wouldn’t know this because online socializing doesn’t count. So the treatment group could be more satisfied with their offline social life and still be less satisfied overall with their social life.

The questions overall seem to start from negative conclusions about Facebook use–envy, time wasting, concentration, and so forth. Don’t get me wrong. I have no issue with seeking to find out potential negative effects of Facebook use and asking about them directly, but if the questions asked all seem to imply negativity, without companion questions with a a neutral or positive bent (such as, for example, feeling motivated or inspired by other people’s lives as portrayed on Facebook as a companion question to envying people’s lives), how much of an effect will the overall tone of the questions have on how people view their own Facebook use?

If people believe that Facebook has negative effects on them (especially if the questions primed them to think about negative aspects of Facebook before the week off started), then odds are they will report a lessening of these effects after a week not using Facebook. In our discussions about this in Skepchick HQ, Jamie Bernstein described this as similar to starting a diet or cleanse that you believe will be good for you: “In general, I would say that making any change you consider ‘good’ will increase your happiness in the short term. So, the people asked to stay off Facebook for a week could have felt happier for that week because they treated it like a cleanse or diet, and so they reported higher happiness.”

Even if, for the sake of argument, all the effects shown by the study were statistically significant and valid, the most this study shows are extremely short-term effects for some people who stop using Facebook, effects that could easily lessen or disappear over long-term Facebook abstinence. Nothing in this study tells us anything applicable to individuals because we don’t have enough information about the characteristics and circumstances of the people in either group or the variation among the answers.

So the study doesn’t really provide much, if any, information about the effects of Facebook use on happiness, but if you believe its conclusions apply to you, then they probably do. It’s science!

This post is based on a discussion of the study with Jamie Bernstein, who pointed out many of the problems discussed here.