Science Shows How Reddit Users Are Like Sheep

If you’re an active Redditor, you might spend time lamenting the fact that some of your most clever, insightful comments get so few upvotes, and the lamest comments of other users sometimes seem to arbitrarily rise to the top.

As it turns out, a trio of researchers—Lev Muchnik of the Hebrew University of Jerusalem, Sinan Aral of MIT and Sean J. Taylor of NYU—recently decided to apply one of the basic tools of science to investigate this phenomenon: the randomized controlled experiment. And by teaming with a social news site to randomly vote on thousands of comments and closely track how they did afterward, the researchers proved that the very first vote a particular item receives—and not just its intrinsic merit—has an outsized influence on its overall fate.

Their new study, published today in Science, relied upon research they did between December 2010 and May 2011. In the paper, they say that it was conducted on “a social news aggregation web site similar to Digg.com and Reddit.com,” but they don’t disclose which particular site it was, because they say the site’s administrators are nervous about the risk to user privacy.

Nevertheless, they describe a bunch of features (the ability to submit links, make comments, vote up or down each post and comment, and a front page populated by the most popular posts) that are a core part of Reddit, and they even use screenshots of Reddit to illustrate them. Suffice to say that if they didn’t do the experiment using Reddit, they did so with an extremely similar site.

During that five-month window, they analyzed 101,281 comments on the site—all posted by normal, unwitting users—as part of the experiment. The comments were randomly assigned into one of three different groups: those that would receive the “positive treatment” (automatically getting one upvote right after being posted), the “negative treatment” (automatically getting a downvote instead) or the control (simply being passed along with no artificial vote).

The vast majority of the comments (95,290) were simply part of the control group—users of the site interacted with these comments with no outside influence from the researchers. The researchers then split the remaining comments between positive and negative at roughly the same ratio that upvotes and downvotes occur naturally on the site: 4049 comments got the positive treatment, receiving an automatic upvote that had nothing to do with their content, while 1942 comments got an arbitrary downvote instead.

The researchers had a hunch that when the site’s users voted on the comments, they were significantly influenced—whether consciously or not—by the votes that had come beforehand. The very first vote, in that case, would be particularly pivotal, because it’d be the only vote the second voter would see. By influencing that voter one way or another, it could potentially influence the third voter, and then the fourth, with cascading effects that influence thousands of votes and produce what the researchers call “herding effects.”

When they analyzed the overall performance of the comments included in the experiment, as represented by the 308,515 subsequent ratings they got in total, their hunch was confirmed: Getting an upvote at the start made the second vote 32 percent more likely to be positive, as compared to the control. The effect was also passed down the line to subsequent voters in much the way the researchers expected, as at the end of the five months, those in the “positive treatment” group had an overall rating (calculated by subtracting the number of downvotes from number of upvotes) 25 percent higher than those in the control group.

Interestingly, though, when applied to the “negative treatment,” the phenomenon seemed to be reversed: Comments that got an arbitrary downvote were actually more likely to receive an upvote from the second voter. The researchers speculate that this represents users’ desire to “correct” unfair downvotes for a comment that didn’t deserve them for any obvious reason.

The experimenters also analyzed the data based on which of the site’s topic areas (i.e. subreddits) the comment fell within—business, culture and society, politics, IT, fun, economics, general news. Comments in the politics, culture and society, and business areas exhibited the greatest herding effects, suggesting that the phenomenon of upvoting in these topic areas was the subject to being significantly yet arbitrarily influenced by the votes that came beforehand, rather than the content of the comment.

It’s easy to imagine how the findings—basically, that our judgement of something is heavily skewed by our knowledge of how others have already judged it—apply to all sorts of situations that go beyond Reddit, both in real life and online. Previous work has already shown that the comments on a Facebook profile picture can influence how attractive we deem it, and if a news article posted on Facebook garners a lot of “likes,” aren’t we more likely to read it? Politicians, meanwhile, have long known that creating the impression of popularity can often be just as important, in an election, as articulating specific positions that merit support.

But does the desire to correct downvotes reveal something inherently optimistic about our society—that we don’t want to watch something undeservedly crash and burn? Does the herding effect of upvotes mean that if we’re not ourselves successful, we’d like to be on the peripheries of successes, regardless of how deserving that success may be?

For the Redditors, the study proves something they probably already suspected, but alas, have no control over anyway: Getting that first upvote can make all the difference.

Joseph Stromberg | | READ MORE

Joseph Stromberg was previously a digital reporter for Smithsonian.