Does Facebook’s new attempt to distinguish good news sites from bad ones work? Is it fair to all sides? If a new study by two Yale professors has it right, then the answers are no and no.

The study claims to mimic Facebook’s “crowd sourcing” readers to find the most trustworthy news sites. It asked participants whether they’re familiar with a site, then whether they trust that site. It then compared the results to a list of “mainstream media,” “hyper-partisan,” and “fake news” sites to judge at how well readers could tell the difference.

Hyper-Partisan Vs. Fake

The authors explain that “hyper-partisan sites do generally report true events (albeit in a biased or deceptive way), whereas fake news sites typically fabricate content entirely.” They don’t think crowd-sourcing will help Facebook distinguish them from each other. They defined sites as fake partly on how often Snopes dealt with them.

But they don’t think that matters. “Both types produce highly distorted and inaccurate content; and that the distinction between hyper-partisan sites and fake news sites is often far from clear.”

What’s the problem with the study? Its way of sorting “mainstream media” from “hyper-partisan” sites is itself biased against right-leaning sites. It also exaggerates how many people trust the left-leaning mainstream media. The 20 “mainstream media” on the list includes almost entirely left-learning sources, with the exceptions of Fox News, wsj.com, Fortune, and Bloomberg. It includes the very partisan liberal sites Salon, HuffPost and MSNBC.

The “hyper-partisan” sites contain mostly right-leaning sources, like Breitbart, Infowars, the Daily Caller and the Daily Wire. It includes only six left-leaning sources. The best known of those is the DailyKos. It doesn’t include the left wing site Slate.

Republicans Prefer Left-Leaning Sites?

The study separated the results from people who said they were familiar with a site from those who didn’t. The more familiar people were with a site, the more likely they were to trust it, even if it’s a fake news site. For example, people who knew the most popular fake news site trusted it more than they did the mainstream media websites Salon, the Guardian, Fox News, Politico, Huffington Post and Newsweek.

Both Democrats and Republicans familiar with the sites said they trusted the mainstream media sites more than the hyper-partisan ones. The study claims that “these crowd-sourced ratings of outlet trustworthiness do an excellent job of differentiating between reputable and non-reputable sources.”

A Flawed Survey

They probably do. But there’s a problem with the study when it’s applied to judging between sites. It was bound to find that knowledgable people trusted liberal sites more than conservative sites.

Most of the hyper-partisan right-leaning sites listed aren’t the most established, well-known sites. This is the biggest problem. The authors took their list from left-leaning sources like Buzzfeed News, Politifact and Melissa Zimdars’ disputed list. They left out established right-leaning sites like The Washington Times, the Washington Examiner, The Washington Free Beacon and The Blaze.

They include far more obscure sites that someone may have heard of, but may not visit. Fewer will consider them more trustworthy than they would The Washington Times, for example.

The authors of the study put the most respected, longtime left-leaning sites in the mainstream news category. They left out well-respected conservative sites. Then they made the majority of hyper-partisan sites conservative. Finally, they put some of the most obscure and newer right-leaning sites in the hyper-partisan category.

Clearly, informed people will deem “mainstream” sites more reliable than “partisan” and fake news sites. If a study defines most liberal sites as mainstream and most partisan sites as conservative, then their study will find that people trust liberal mainstream sites more than conservative partisan sites.

Well, duh. This is the way they got even Republicans to choose the left-leaning sites as more reputable than the right-leaning sites.

The Problem With the Study

The authors themselves aren’t free of their own bias, as Breitbart noted of one of the authors. Gordon Pennycook retweeted a post calling the Nunes memo a “stunt” born from “right-wing fever swamps” that “recklessly breathe life into conspiracy theories.”

The study brazenly claimed that the hyper-partisan sites have “no editorial norms.” One of the authors of the study, David Rand, backed off this claim in an email to Breitbart. He said he meant “weaker editorial norms.”

The effect, and perhaps the purpose of, this study is to artificially adds credibility to left-leaning news sources. If this is the way Facebook is going to rank news sources, they won’t just protect their readers from fake news. They’ll keep them from hearing the truth they’ve labeled “hyper-partisan.”

Follow Rachel on Twitter at Rach_IC.