In this age of “fake news” and “alternative facts,” how do we know that the so-called fact-checking sources are accurate and honest arbiters of what is true and what is false?
Originally Answered: In this age of “fake news” and “alternative facts,” how do we know that so-called fact-checking sources (snopes.com, politifact) are accurate and honest arbiters of what is true and what is false?
Dr. Julian Bashir: You know, I still have a lot of questions to ask you about your past.
Elim Garak: I have given you all the answers I’m capable of.
Bashir: You’ve given me answers, all right; but they were all different. What I want to know is, out of all the stories you told me, which ones were true and which ones weren’t?
Garak: (slightly agahst) My dear Doctor, they’re all true!
Bashir: Even the lies?
Garak: Especially the lies!
– Star Trek: Deep Space Nine, “The Wire”
Back when I used to teach high school language arts, I did a unit with my sophomores on critical thinking and source evaluation that focused on essentially this very question.
Not specifically Snopes or Politifact, but all sources.
How do you know that what you’re reading or viewing is credible and accurate?
I started the unit by asking how many of my students believed in the Grand Canyon. All of them raised their hands. I asked them how many had been to the Grand Canyon. Usually a few would keep their hands up, no more than three or so. I’d ask how much of it they had viewed. Whether they measured its depth or length personally.
By the time we were done, no student could ever definitively say for absolute sure that the Grand Canyon, precisely as described in scientific literature, existed for sure.
Then I’d ask the students to find some event that they had all witnessed personally, usually something recent. I’d have them write down as many details as possible personally, and then go start comparing on the board.
They’d start to notice inconsistencies between themselves. “Why?” I would ask, while hiding a smile. They were all there, right? Why is it that they remember it different? What’s the truth?
This answer will be long. There will not be a TL;DR version.
Pretty much everything in source evaluation comes down to credibility. You have to decide whether or not you trust that source. You have to decide whether the scientists who measured the length, depth, and width of the Grand Canyon were honest about it. You have to trust that any student had an accurate perception of an event.
There’s just one problem with that trust: human beings really suck at intuiting who to trust.
We all routinely tend to overestimate our abilities to determine whether or not someone is telling the truth.[1] We are very prone to disbelieving facts that do not correlate with our inbuilt implicit biases, or which challenge our core beliefs.[2] And likewise, we are prone to automatically accept information that verifies our core beliefs without challenging it.[3] [4]
Some people are more prone to this than others.[5]
This is why eyewitnesses to a crime are actually surprisingly untrustworthy.[6] Our very memories are not necessarily as reliable as we’d like to believe.[7]
So, who should you trust? And what factors tell you that you can you trust them?
There are some important things to think about whenever evaluating sources.
(1) Do I want to believe it?
Do you know you have more nerve endings in your stomach than in your head? Look it up. Now somebody’s gonna say, “I did look that up and it’s wrong.” Well mister, that’s cause you looked it up in a book. Next time, try looking it up in your gut. I did. And my gut tells me that’s how our nervous system works.
~ Stephen Colbert, The Colbert Report, October 17, 2005
Ask yourself, do I want to believe this? Do I not want to believe this? Where are you predisposed?
Is this another story that confirms your fears about Killary having those with evidence of her wrongdoing in Benghazi knocked off? Is this a story about some new horrible thing that Trump tweeted at 2:30 AM? Do you immediately dismiss it as nonsense because it says something unflattering about something you believe?
How do you feel about the story, right from the title? What is, as Stephen Colbert calls it, the “truthiness” of the story?
The stronger you feel about the story, the more critically you should vet it, either way.
This is because of confirmation bias.
I’ll talk more about biases in general later, but I want to address this specifically as a gateway question in source evaluation.
Confirmation bias is the tendency to accept information that corroborates our worldviews and reject information that is dissonant to our worldviews. The Oatmeal has an excellent strip about this and the “backfire effect” in debate, which is how confirmation bias leads people to entrench themselves more deeply in a belief when presented with contrary facts.
Confirmation bias is the single biggest reason that fake news goes viral. People read something, it agrees with their already-held views, and without even questioning it, they pass it along. Everyone that also already agrees with those views sees that, nods their head, and without questioning it, passes it along.
People don’t do that when it violates their worldviews.
That’s why the first check for evaluating any source ought to be “do I want this to be true or false.”
Being aware of your own predisposition is critical in actively evaluating a source, because it can lead you to overlook or place undue emphasis on other information that disagrees with or corroborates the source.
Intelligence professionals have this hammered into them constantly. It is very tempting to get a piece of intel and then only look to confirm it or dispute it – even when, especially when they don’t know it’s what they’re doing. It’s a great way to get people killed.
This is how the United States got into the Second Gulf War in Iraq: when the intelligence community didn’t tell the administration what it wanted to hear, it created a specialized task force within the military to circumvent the rest of the intelligence community and find enough evidence to support what the administration wanted.
Attorneys like me also have this hammered into them. It’s very tempting to just look for case law or facts that support the client’s position. It affects the very way we construct searches for those pieces of information, and it can cause us to overlook contrary law or facts, and then get ambushed with it. Good attorneys learn to think antagonistically to their own case and look for ways to destroy their own positions, so they can defend against them. That requires being very aware of our own confirmation biases. (One mentor attorney I know routinely plays chess against himself to train his brain to attack his own preferred positions.)
Before moving on to any other steps of source evaluation, this is the first and most critical one. Don’t let the wish be the father of the thought.
Now, some people are more susceptible to confirmation bias than others.
(2) Levels of sources: primary, secondary, tertiary.
The next most important places to start with source evaluation is checking how close to the original fact reporting the source is.
Primary sources are sources that have direct, first-hand knowledge.
Some examples of primary sources would be eyewitnesses, autobiographies, diaries, or photographs that can be authenticated by the person who took them. Audio or video recordings are primary sources. Original legal documents are primary sources.
Secondary sources are interpretations of those primary sources. Scholarly research would be a secondary source, while the data scholarly research relies on would be the primary source. Editorial commentary on a news story is a secondary source.
Secondary sources fall along a spectrum from original fact journalism that gathers up and reports primary source material to analysis of primary facts to outright opinion on the facts.
Tertiary sources are unrelated to the facts themselves and are tools to help interpret primary and secondary sources. Dictionaries, thesauruses, almanacs, fact books, encyclopedias, that kind of stuff are tertiary sources.
Now, the line between primary and secondary sources can sometime be a little fuzzy.
For example, reporting itself can be a secondary source. Good, ethical journalism tries to refrain from editorializing as much as possible, and focuses on just reporting the primary source material without interpretation or commentary.
However, good journalists also understand that just reciting facts without putting those facts into context is also problematic. Trying to add that context can sometimes end up interjecting analysis and opinion into the mix.
A critical reader understands how to sort out the primary source material from the secondary source material.
This is where Fox News gets itself into a lot of trouble, where Politifact sometimes straddles the line, and where Snopes is generally quite good.
Snopes, Politifact, Factcheck.org, and other fact checking sources will cite the primary sources and follow up on those sources. They will link to online sources and vet those sources for verification.
Snopes is excellent at just checking whether or not there is primary source material, and whether that primary source material supports the conclusion. Snopes does a very good job at providing links and citations to the primary source material so that a responsible critical reader can follow up on that source material and verify it for themselves.
Politifact is also very good at linking to its primary sources.
Fox News, Occupy Democrats, Breitbart, Huffington Post, InfoWars, the Daily Kos, and others are not very good at this.
These sources tend not to link to primary source reporting or primary sources at all. If you follow those links, the original source might be buried five layers deep. Fox News cites to World News Daily which cites to Breitbart which cites to The Daily Mail tabloid which simply pulled it out of its ass to start with.
Or worse, these links are circular. Fox News cites to WND which cites to Breitbart which cites to InfoWars which cited to a Fox News article in the first place. In that game of telephone, the information got distorted enough that the new article becomes an amped up pile of steaming bullshit.
This kind of source burying and game of telephone is exactly how an Obama administration “scandal” got started that never actually existed. President Obama went on a trade summit mission to India in 2010. These kinds of visits abroad require a lot of staff and routinely cost several million dollars a day.
On November 2nd, the website for New Delhi TV reported that this trip was going to cost the United States two hundred million dollars a day. It cited an anonymous Indian official. There was no other verification.
The Drudge Report picked up the story and posted a link to it, either believing the story or simply not caring whether it was or wasn’t. Rush Limbaugh, popular conservative radio host, picked up the Drudge Report story, and runs with it. Fox News picks up the Limbaugh broadcast, where then-Fox commentator Glenn Beck adds 34 warships or approximately fifteen percent of the U.S. Navy, and ups the staff count of folks coming along to three thousand people.
This prompts the administration to put out a statement that these figures “have no basis in reality.”
Then-Congresswoman Michelle Bachmann ignored the statement, and restated to Anderson Cooper these entirely unverified numbers and now adding that President Obama’s administration had booked over 800 rooms at the Taj Mahal Hotel – which has only 550 rooms total. When Cooper asked her where they came from, she simply said, “These are the numbers coming out of the press.”
It took less than 48 hours for this story, which was made up whole cloth, to make it up to a Congresswoman to be recited as gospel truth, because nobody in that chain looked for the primary source to verify its accuracy or credibility. It passed through at least six levels of secondary sources. And in that game of telephone, an inflated number of staff, nearly one-sixth of the United States Navy, and double-booking at least half of the Taj Mahal Hotel were all added.
(3) What are the actual, verified facts the source contains?
There’s an old expression: trust, but verify. Look for facts.
Importantly: look for actual facts and not just opinions masquerading as facts. This is becoming an all-too-common problem today. Someone’s unsupported conclusion is taken as fact.
Facts are verifiable. They happened, or they did not. They exist, or they do not.
Opinions carry around a conclusion or a preference with them. A person who says that the color green is their favorite is stating what looks like a fact about themselves, but it’s a preferential opinion: green is the best color. This can’t be verified. This is merely a conclusion.
Facts, on the other hand, are simply neutral. That grass is green. How do you know that? We can verify it. Five people can look at the grass, and come to a consensus about whether it is green. We can settle the matter by using a spectrometer and determining whether the grass reflects electromagnetic waves at 560–520 nm (the accepted consensus definition of green.)
This is one of the reasons that Snopes and Politifact are generally regarded as safe: they verify the facts that various statements rely on.
Some time back, a friend of mine posted an article she ran across in something called the Freedom Daily about Islamberg, New York being raided by a massive counter-terrorism task force who caught more than two dozen Muslim terrorists with various explosives and weapons.
The problem is… it never happened.
The only thing that can be verified is that one Muslim person from a town nearly fifty miles away with no verified ties to Islamberg was arrested for stealing several boxes of ammunition, and in the subsequent search related to that investigation, it was discovered that he had several illegal firearms.
There is absolutely no verification of a Federal raid on Islamberg.
Primary sources such as the county sheriff, someone that would have to be included in something like that, noted that it was familiar with the article and that there was “absolutely no truth” to it.
There was literally no verification of the story.
But what if there’s proof, like photos, right? Surely they can’t be made up!
Even pictures can lie.
Graphic pictures are another area where untrustworthy sources can try to grab you and suck you in. For example, there was a photograph that was circulated in around several evangelical Christian blogs depicting a bloodied woman with her eyes and mouth sewn shut, and touted as a Saudi woman who had been tortured for her Christian faith.
A reverse Google image search quickly reveals that this woman is actually Japanese, and is into “extreme body modification.”
Stock images or composite images are often used in many fake news sites. While some composites are difficult for the average person to detect, others are usually obvious – shadows or lighting that doesn’t match, that sort of stuff. If a photograph is reported in the news source, do a reverse image search if you’re not sure about it. And if it’s graphic or seems extremely damning, definitely you shouldn’t be sure of it.
Even reputable news sources can sometimes re-use old photographs or video, particularly of various natural disasters or extreme weather. A photograph or video of people walking through a blizzard could be years old.
Snopes, Politifact, FactCheck.org, etc., are excellent secondary sources when it comes to pulling out the verifiable, relevant facts from the primary sources. They follow up on the sources from other information outlets, internet stories, or statements of public figures, and find out what the verifiable facts are. The only opinion they make is whether the facts support the conclusions reached by these information outlets, and they’re up front about what their opinion is and how they reached it. They check whether photos are real or not. They do the reverse image searches. They follow the links. They make sure to vet the material before presenting it.
(4) What inferences can reasonably be drawn from the facts, and do you have all of the facts to draw accurate inferences?
Inferences are conclusions that people make after looking at the facts.
This is where people mostly get into trouble. This is also where the spin doctors make their money. NPR, MSNBC, and FOX can all report the exact same piece of news, and all three of these sources could result in three entirely different and competing narratives. Why? Because of the why. Each will present the story slanted to let the reader come up with their own context about what those facts mean, why the story happened that way. The sources, the context, all of it is designed to lead you to a certain conclusion.
Facts on their own are just trivia. People naturally try to put facts into a narrative. It’s just how we think, how we operate as human beings.
That is why people will make viral a Natural News article that shows that a study “conclusively proved” that there is usually 1 part per billion of glyphosate in wine.
Why does that matter?
Because glyphosate is a boogeyman to a large group of people. It could (very disputably) be a carcinogen. Never mind that your typical wine contains approximately 130 million parts per billion of a known carcinogen (ethanol) because that’s not the story. The narrative is something sinister is amok. The article invites you to draw a conclusion from the facts they present, namely that Big Wine is in the pocket of Monsanto who doesn’t want you to know they’re spraying grapes with Roundup because Monsanto wants you to have cancer. Why? Because they’re eeee-ville!
What logically follows from the facts? Not what you want them to mean. Not what you wish them to mean. This is where you have to walk back over to step 1 and ask yourself what do I want to hear?
Logical inferences are created by applying facts to rules.
For example, what happened here?
It probably seems pretty obvious to most people that this guy’s wife snuck up behind him, smashed him with a bottle while brushing his teeth, and then tried to make it look like he slipped on some soap coming out of the shower, right?
There is a whole set of rules about the evidence we see in this picture getting applied. What does a bathroom typically look like? How do shoe prints get there, and what rules do we have about the shape of shoe prints? What do we know about monogrammed towels and their typical usage? There are a whole host of rules that get subconsciously applied here: only women wear high heels. A man doesn’t use the “hers” towel – he uses the “his” towel.
These rules are the critical link between the facts and the conclusions we can draw from those basic facts. The facts here are not in dispute. We can all look at those facts and agree what the facts are.
But the rules we impose on those facts to draw inferences might not be accurate.
When assessing the credibility of a source, it’s essential to be self-reflective and personally critical of the rules that you are applying to reach a conclusion. This goes back to that confirmation bias that I referred to earlier.
Even certain pieces of evidence might be inferred, or conclusions stacked on top of each other to make larger inferences.
For example, if there’s a missing person, a pool of blood, and drag marks, and you don’t have a body, would you not typically infer there was a dead person somewhere? So, if it’s somewhere, where is it? And what happened?
It seems obvious that someone was murdered, right? But that’s merely an inference, which filled in certain missing pieces of evidence.
This is also why it’s important to have all of the facts before making an inferential conclusion.
Are you using all the facts, or ignoring ones that go against the narrative you want to hear?
During the 2016 U.S. election cycle, a black church was burned, and the words Vote Trump spray painted on the side.
Factually, this happened. Depending on how crazy you want to get, you could either trust the photos, or have bought a plane ticket and driven down to Mississippi to go see it with your own eyes. It’s verifiable, authentic.
What came after that is the problem. Assumptions and inferences ran amok.
Not without good reason, mind you.
It was a historically black church in Mississippi. It’s not hard to add a little into the fact pattern. It’s not a stretch to find a context. A lot of people wanted it to be Trump supporters. They wanted them to be as vile and deplorable as they were led to believe Trump supporters were. A few facts were enough data points combined with some historical precedence to reach a plausible conclusion. A lot of liberal people wanted that story to be true.
Then, it turned out later that the church was burned down and tagged by a black man, who was a congregation member.
Immediately, conservatives went crazy. This was vindication, based on that one little fact alone. They tweeted and posted and blogged immediately, crowing their victory. They wanted a different story to be true, one in which they weren’t the bad guys. Black guy trying to set us all up, we knew it all along! See?! See?! Y’all called us deplorables, and look how mistaken you were!
And so they ignored (or didn’t bother to look for) any other facts. They went straight from a little bit of fact to an inference about what those facts showed, because it supported their narrative.
I admit, when the story first broke originally, I wanted it to be true. Black church, Mississippi, plenty of white supremacists at Trump rallies, sure. It wasn’t a stretch. It wasn’t implausible. And it said something about Trump supporters that I personally liked, that some of them were terrible enough people to firebomb a church and tag the ruins with self-supporting graffiti. I wanted that to be true.
Still, I cautioned myself and others at the time after a day of thinking about it to wait for the investigation to come back. Let’s wait for the facts. All of the facts.
Turns out, after we got those facts, everyone was wrong. Everyone drew inferences from scant facts, and every one of those inferences was wrong.
It turned out that the man arrested wasn’t doing it for racially motivated reasons at all. He was taking advantage of the charged, polarized environment around the election to cover up a burglary he committed at the church. It was a convenient cover for him. He would have gotten away with it, as well, if it weren’t for good police work that went beyond the initial theory of the case, and looked for all the facts. (And those meddling kids!) Not just facts that supported one version of events, but all the facts.
If all the police did was look at the early facts and didn’t investigate more, didn’t follow up on everything, they might very well have reached the same conclusions the internet did. They might have reached the same conclusions that many of their predecessors in the same town might have reached a hundred years ago, and those people a hundred years ago typically ended up lynching someone over it.
Remember: there are two parts to inferences, the facts and the rules.
In my profession as an attorney, we talk about “garbage in, garbage out.” If there is a breakdown in the facts, either inaccurate or incomplete facts, or if there is a breakdown in the rules, either inaccurate or incomplete rules, then the inference drawn from those facts and rules will also be inaccurate.
When evaluating a source, evaluate the inferences it makes, and the inferences you draw from the source, and ask whether it is logically supported, whether it is based on incomplete or inaccurate data, or whether it is based on incomplete or inaccurate logical rules.
Furthermore, you need to be self-aware of any rules that you are imputing into the situation. Again: go back up to the picture here and think about how you arrived at any conclusions. What rules did you apply to those facts?
Those rules didn’t come out of nowhere.
Our brains are constantly creating a framework of rules for how the universe operates. We literally couldn’t function without them. Many of these rules are intuited from our observations.
In the Tragedy in the Bathroom, some of those rules might include:
- Women are the ones who usually wear high heels, not men.
- A man is likely to use the “his” towel and a woman is likely to use the “hers” towel.
- Bare feet don’t leave the same kinds of footprints as shoes and certainly don’t leave the kinds of footprints as high heels.
- Additionally: high-heeled shoes leave a distinctive footprint pattern that looks like this.
- A person who suffers a slip-and-fall-injury leading to death would probably be facing parallel to the place where they would have slipped and fallen, not perpendicular.
You could chart these out like this:
If you have evidence and a conclusion, have you consciously articulated how you got there? If not, stop and think about that for a moment: what rule did you apply that bridged that evidence and conclusion?
And then ask whether that rule is valid. Is that rule an assumption? Or based on assumptions?
For example, if you concluded that the poor schmuck in the photograph was killed by his wife, what assumptions did you likely make?
- Did you assume that whoever owned these “his” and hers” towels were married?
- This assumes that only people who are married cohabitate. This assumption may or may not be true, depending on where you live and the culture there.
- This also assumes that it was whoever used the “hers” towel that killed the guy.
- This neglects possibilities such as a jilted mistress who just found out that the guy was married, or that this poor schmuck was at the mistress’ place and his wife found out. The deceased might have even been at a crappy run-down motel for a tryst.
- For that matter, maybe the guy was killed by a femme fatale assassin looking to set up another woman?
The key here is to be self-reflective: if you’ve come to a conclusion, do you have both verifiable evidence to support that conclusion, and valid rules? Make sure to think critically about those rules and whether they are built on any assumptions of how the world works. Question those.
This is quite difficult in practice, and often best done by bringing in an objective third party to question those rules and assumptions.
(5) What is the authorial bias?
I imagine most of you thought we ought to start here. If the source is biased, we can just dismiss it, right?
Absolutely not.
Every source is biased. And that’s okay.
Bias is not inherently a discrediting problem.
There are a couple of common forms of authorial bias.
- Confirmation bias. I discussed this earlier. It’s the tendency to accept things we already like and reject things that we don’t.
- Partisan or “statement” bias. This is what we most often think of as bias, or “spin,” where the author engages in active advocacy. This kind of bias is where the author goes beyond simple primary fact reporting and interjects a preferential or partisan slant on the material. They present the facts in such a way that it clearly favors a perspective.
- Selection or “gatekeeping” bias.Some news sources go out of their way to try to be balanced in their presentment of a news story, such as NPR, but have a different kind of bias: what stories they present at all. This is selection bias. By cherry-picking only certain events or facts, even if the story is presented in a balanced manner, the overall source is biased.
- A subset of this is “ventriloquism,” where a source edits experts or witnesses out of context so that their quotes agree with the author’s bias, or only show the parts of the expert or witness that agree with the author’s bias.
- Coverage or “visibility” bias.Some stories are juicier or more popular than others, and so get more coverage. Hundreds of children are killed by their parents every year in shaken baby incidents or abuse. But the Casey Anthony trial ended up becoming incredibly high profile, dominating the national news for weeks and captivating the country, because it had a special twisted kind of appeal to it.
- Concision bias.In the TL;DR culture today, sources are having a harder and harder time with complex, nuanced reporting and analysis. Investigative journalism can take months if not years of research; see the Boston Globe’s Spotlight team’s investigation of the clergy sexual abuse in the Catholic church. There is a lot of information that needs to be put into context.
And people are less patient to read that. They want the quarter-page summary. The 140 character tweetable explanation.
The McDonald’s hot coffee case is a perfect example. The original story was long and complex. The jury found that McDonald’s was not only negligent, but actively reckless in how it stored and served coffee, having broken regulations on safe temperatures. The woman involved was not driving; her son was. They had pulled over into a parking stall; they were not in motion when she spilled the coffee. She had received third degree burns to much of her body. The jury award was not only for her medical bills, but punitive damages because McDonald’s had acted so egregiously.
- This is bias for the extraordinary over the ordinary. Sometimes also called “yellow journalism,” it is the tendency of media outlets to hype up an otherwise less exciting story to get better ratings or circulation, thereby giving the impression that relatively rare events (such as an illegal immigrant committing a violent crime,) are more common than ordinary events (American citizens committing violent crimes.)
- Modern mass media’s newest form of sensationalism is what we call “clickbait.” If you see articles about “X Politician Blasts Obama!” or “21 Unconstitutional Things Obama Did – You’ll Never Believe #17!” it’s just sensationalized clickbait.
- The Spanish-American war was started because of a circulation war between Joseph Pulitzer’s New York Worldand William Randolph Hearst’s New York Journal. On February 15, 1898, the S.S. Maine blew up in the Havana harbor after what a Spanish investigation determined to be a spontaneous coal bunker fire that spread to the ship’s main magazine. The Navy tried to push the narrative that it was a Spanish mine, but openly ignored their own internal investigators that believed the Spanish conclusion was accurate and the explosion was initially caused by firedamp released by the use of bituminous coal over anthracite coal.
Decades later, further exploration of the wreck of the Maine and investigation of firsthand accounts leaned heavily to confirm the conclusion that the Maine had been destroyed by a coal explosion and not a Spanish mine, in part because many other ships of the same construction type and class using the same coal suffered similar nearly catastrophic explosions until the Navy quit using coal and switched to fuel oil.
But at the time, Pulitzer and Hearst hyped it up as an attack on the Maine by Spanish forces to sell more papers. Quickly, the chant “Remember the Maine; to hell with Spain!” became a rallying cry that shut down ongoing diplomatic negotiations and instead goaded a hawkish Congress into declaring war on Spain two months later in April of 1898.
- False Balance or “fairness bias.”Sometimes sources try to present facts in a balanced manner to avoid the appearance of bias, but this can lead to giving the appearance of fairness at the expense of artificially making it seem like a side of the story is more based in reality than it is.
- There’s a good little exchange in the television show The Newsroomthat illustrates this:
Maggie: How can you be biased towards fairness?
MacKenzie: There aren’t two sides to every story. Some stories have five sides; some only have one.
Tess: I still don’t underst…
Will: Bias towards fairness means that if the entire congressional Republican caucus were to walk into the House and propose a resolution stating that the Earth was flat, the Times would lead with “Democrats and Republicans Can’t Agree on Shape of Earth.”
- There’s a good little exchange in the television show The Newsroomthat illustrates this:
- False Timeliness. This can come in two main flavors:
- Presenting an old story as new. This is more common to hoax sites or actual fake news, recycling old scaremongering and just updating the release dates. This happens a lot with crimes; something from five or ten years ago is brought out as having freshly happened.
- Bringing up an old story as suddenly relevant because a similar event has now occurred, or without specific context and thus implying that the old story is new or timely.
The key to evaluating a source for bias is understanding what bias it might have, and then accounting for it when verifying the source material or judging the credibility of the source.
I don’t ignore Fox News, but I know very well that if I see a Fox News article, I’m going to have to do a lot of extra digging to get the rest of the facts that they inevitably decided weren’t relevant (and usually are,) and spend a lot of time sorting out the spin from the primary source material and isolating the rest of the biases.
Now, most of the time I’m just too lazy to do that much work on it, so I don’t bother with Fox News in most circumstances.
There’s an excellent chart out there called the Media Bias Chart.
This is a good chart, but it really only covers partisan or statement bias, and where along the secondary source spectrum a source tends to exist on average.
This chart doesn’t tell you much about selection bias, coverage bias, or other authorial biases.
Some of the sources at the top of that chart are highly reliable sources in terms of their partisanship and original primary source reporting. The AP and Reuters are some of the gold standard in journalism.
But the AP and Reuters can also still be subject to selection or concision biases without ever being partisan. The AP doesn’t often do extremely in-depth investigative journalism; it tends to report shorter stories of current events.
ProPublica, The Economist, USA Today, or even Time Magazine are all reasonably centrist in partisan bias, but these tend to interject a great deal of selection bias simply due to format of presentation. Most of these are periodicals for their long-format pieces. They don’t do much short reporting. They do more analysis. It might be non-partisan analysis, but because they only have so much space and time, they have to pick and choose what to write about. That inherently results in some selection or coverage bias.
That’s not necessarily anything bad. It just means you might not get other important, relevant facts or stories that could shed relevant context on the original story.
Snopes, Politifact, and other fact checkers are also susceptible to these kinds of biases, particularly selection bias.
Politifact is criticized by conservatives as liberal-biased because of this. Politifact tends to fact-check more conservative claims than liberal claims. As a result, a broad survey of the site makes it seem like it is biased towards liberalism.
And conservatives sometimes gripe about the site simply not bothering to check on certain liberal claims. That can be a valid criticism.
Now, there’s also something to be said that conservatives might just also make more and more incendiary claims, and make more erroneous claims.
For example, conservatives argue that the bulk of the media outlets must be liberally biased simply because they tend to report much more negative press about President Donald Trump than positive stories.
But, that doesn’t inherently indicate partisan, selection, or any other bias. It could simply be that President Trump keeps consistently lying out every corner of his mouth and constantly picking fights with the press over policies that are generally criticized by a majority of the public figures out there.
If Republicans do a dozen crazy things and Democrats do one, it isn’t unbiased to make up eleven crazy stories about Democrats to balance it out. That would just be introducing a fairness bias, instead.
As I said, every source has some degree of bias. That’s fine. Apply the appropriate corrective lenses, and evaluate the facts and inferences on their merits.
(6) What degree of institutional integrity does this source have?
Good journalistic outlets know that the only resource they have is credibility. If a journalist is not credible, she is finished in the industry. Lying, just once, even by mistake, is (usually) a career death sentence.
Sources that are reputable and credible take this very seriously. If they have to issue a retraction, they have probably fired the journalist who reported the story and that guy is likely blackballed from everything with more integrity than Buzzfeed.
Look at how the source handles corrections and retractions when it turns out they’re wrong about something. That will tell you a lot about the credibility of the source right there. FOX virtually never issues corrections or retractions, and never for its flagship primetime shows hosted by what they go out of their way to call entertainers, such as Tucker Carlson and Sean Hannity. Even if these guys are straight up factually wrong, provably so, they never issue an apology for getting it wrong. That should tell you something about the institutional integrity of the source; namely, that it has virtually none.
CNN is likewise slow to correct itself when it does. The Huffington Post doesn’t readily admit mistakes, either. I’ve never seen the Daily Kos issue a significant correction or retraction about anything.
WaPo and the Times, in contrast? Generally, they’re quick to admit mistakes and issue corrections, updates, or even retractions. The AP, Reuters, NPR, The Wall Street Journal, The Economist, CBS, ABC, NBC, The Atlantic, really most major journalistic outlets are pretty good about this.
There are some red flags you should immediately recognize for any source, article, or media outlet that should alert you to credibility issues. Some can be “immediately discount this source and do not engage further with it,” while others can be “proceed with caution.”
- You are barraged with advertising and pop-ups the minute you get to the site.Even reputable sources make money off of advertising, but if your screen starts to look like the house from Chevy Chase’s Christmas Vacation, especially if it’s advertising to lots of clickbaity things like “This one weird trick in your town can lower your insurance by 75%!” it’s less likely to be a reputable, credible source.
Run away and don’t come back to these sources.
- Listicles and articles that require you to click through 15 pages to read all of it. If you’re in a source that requires you to wade through 30 ads a page and click on “next” a dozen times to read the whole thing, it’s a red flag that this is not a reputable, credible source, and is just a steaming pile of clickbait trying to extract advertising dollars from you.
Again, run, don’t walk, and never come back.
- Number Seven Will Shock You!These are another form of clickbait closely related to listicles. Spoiler alert: number seven probably won’t shock you.
- This is why sources like Buzzfeed, Twenty-Two Words, etc. are generally crap. Don’t use these for news or credible sources.
- Headlines that don’t match the articles.You’d be amazed at how often shitty sources will try to grab you with a sensationalized headline that has little or nothing to do with the article. “Joe Biden Actually Registered Member of KKK!” turns out to be a story about some idiot that just happens to be named Joe Biden that lives in Mississippi and not the former vice president.
One time here or there and it’s minimal? Flag it and be skeptical of anything it publishes. If a source does this a lot, or it’s really egregious (like the Biden example above), just stop using it.
- Headlines with graphic pictures or promises of graphic pictures.This is often clickbait. The source is trying to lure you in with the promise of a novelty. That picture may not even be in the article.
- This is again why sources like Buzzfeed, Twenty-Two Words, etc. are generally crap. Don’t use these for news or credible sources.
- Also, be really aware of photographs. Some of the pictures of kids in cages published during the Trump Family Separation Crisis were either from the previous administration or, when zoomed back, were random kids on a street behind a chain link fence who were not being held in detention at all. There are many sources, even reputablesources that as of August 2019 are using old stock images of forest fires for headlines about Amazon rainforest wildfires; at least two normally reputable sources used photographs that weren’t even of the Amazon.
- Headlines that are questions.This is also clickbait. The source is trying to lure you in by offering a question, not a fact. The article probably doesn’t answer it. No, it’s not a “thinkpiece.” It’s usually just a pile of trash designed to generate ad dollars.
- “Native advertising,” or pieces written by advertisers, not the actual content providers.These can be really hard to spot sometimes. They look like actual articles. They have headlines and graphics, sometimes infographics, and can very accurately mimic a real piece. But in actuality, they’re written by advertisers, not journalists.
- Sometimes this is called “branded content,” or “sponsored content,” or “featured partners.” It’s the same thing.
- Some sources try to take good care to make this very explicit. The Timesusually has a big banner that reads “Paid Post.” But even this may be misleading to the reader, if they are not aware that not all content is written by the outlet itself, and these don’t look like ads at times – they look like articles.
- If you see these kinds of articles, and you might have to look carefully, it may not be an instant disqualification for the source itself, but you can safely discount that entire article.
- Vague attacks or generalized references.If you see something about “Washington” or “The White House,” or “Trump supporters” or “Bernie fans,” you can safely discount it by at least 50%.
- Anonymous sources. Be careful with these. They canbe reputable. Mark Felt was the whistleblower who brought down Nixon, and nearly until his death in 2008 was an anonymous source known only to the public as “Deep Throat.”
But anonymous sources can also be extremely disreputable. Look to see if any of the facts can be verified by independent sources. The “Deep Throat” information was all vetted very, very carefully through independent sources before the Times ran with it.
- Anonymous sources. Be careful with these. They canbe reputable. Mark Felt was the whistleblower who brought down Nixon, and nearly until his death in 2008 was an anonymous source known only to the public as “Deep Throat.”
- Future speculation.If the source is speculating on what might happen, be wary of it. Unless military scientists have something with time travel not generally available to the public, it’s pretty difficult to get accurate fact reporting from the future. Be wary of sources talking about what’s going to happen as if it were fact.
- “Lawmakersays [insert shitty foot-in-mouth statement here]” or “Lawmaker proposes bill to [insert extremely stupid or divisive issue here.]” Turns out that “lawmaker” is a pretty generic term that can apply right on down to a city alderman somewhere that has about as much national level political clout as the secretary of the local PTA. Sources use “lawmaker” because it’s not a Congressman or Senator and it makes the person sound more important.
That “lawmaker” is probably a low-level junior freshman politician that has absolutely zero chance of passing his “ban breastfeeding for public morality” bill. But a disreputable source might try to make it sound like he’s the governor of New York or Speaker of the House.
There are over seven thousand state-level elected legislators, some of whom won their seats while getting fewer than 1500 people to vote for them. New Hampshire alone has over 400 elected legislators.
- The same is true of “advisors” or “officials.” These are basically no better than anonymous sources. The source is trying to make them sound more important than they probably are. Take it with a salt lick.
- The same is also true of [insert celebrity figure with zero expert qualifications here.]
Let me get this off my chest:
Ted Fucking Nugent is not a reliable source regarding literally any goddamned piece of information, including music. Ted Nugent is not a politician. Ted Nugent is not a policy expert. Ted Nugent doesn’t have any expertise in any area except how to sing “Cat Scratch Fever” and avoid getting drafted by the military. Ted Nugent does not hold a single degree above a high school diploma nor have any relevant professional experience that would qualify him to be a credible source of anythingexcept for how to look like human-rat hybrid experiment that went horrifically wrong.
Stop giving this moron any sort of credibility. He is not a credible source of information.
The same is true of all sorts of other celebrities who have decided that for some reason or another, their opinion matters and should be taken serious. Just because you have a million followers on Instagram does not make you a credible source. Just because your parent was a senator doesn’t qualify you as an expert on foreign policy or politics.
- It’s on the blog portion of an otherwise reputable source’s website.Forbes and Reuters are respected, usually highly credible news sources. However, both Forbes and Reuters have third-party-blog portions of their websites that are not vetted and edited by Forbes or Reuters. I could go get a blog on Forbes. Reuters has a section of its site dedicated to unvetted, unedited press releases that literally anyone can publish. Contributors are solely responsible for the content. These sources do no fact checking about these posts. Make sure its from the journalism side of those generally reliable, credible sources before trusting it, not Joe’s Totally Unbiased Blog hosted on Forbes.
Avoid these sources, or very carefully review them.
- Look for weasel words.“Many experts agree” or “polls indicate” is often a way to weasel out and report something that isn’t really credible without actually saying it is. “Many experts” could turn out to be four guys in a bar in Djibouti. The source ought to tell you who those experts are and where they work.
Follow up and see if they’re actually credible sources and experts in the field they’re talking about. - Recent/new study shows [insert inflammatory conclusion or health benefit of otherwise unhealthy habit, etc.] This is another classic piece of clickbait. News sources are notoriouslyscientifically illiterate and rarely report on methodology, confidence levels, what “statistically significant means,” the fact that correlation is not causation, and a lot more. Take everything that involves studies with a salt lick unless it’s from a peer reviewed journal and you actually know how to read the published results. Which also leads to…
- Watch out for “journals” that aren’t actually peer-reviewed, credible sources, or are actually just think tanks, foundations, and institutes that spout made up bullshit.
There is an ever-increasing number of sources that really lookand sound like they would be scientific, prestigious journals, and are anything but.
For example, the American Journal of Engineering Research sounds like a pretty reputable source from the title, but it’s nothing more than a predatory journal that does not publish anything even approaching peer-reviewed quality scientific research.
There are also various organizations, think tanks, and “institutes” that sound like credible sources, but publish conspiracy theories, pseudoscience, hoaxes, and other entirely made up nonsense. It’s really easy to fall down the rabbit hole these days, especially on Facebook, YouTube, Twitter, etc.
Avoid these like the plague on humanity they are.
- Public Survey Push Polls. Unless it’s conducted by a reputable pollster who specializes in this kind of statistical work, polls are not always a very credible source of information. Worse yet if they’re a poll conducted by a local news outlet by putting a question up on Facebook or Twitter.
Discount them as credible sources of anything except how many really angry people use Facebook or Twitter.
Polls canbe useful if they’re well conducted by otherwise reputable sources. Gallup, for example, is generally a pretty good pollster. Marist and Quinnipiac are fairly highly regarded. Zogby… not so much.
- Fox News, surprisingly, is a pretty good credible pollster. They source this out to two firms, one Democratic and one Republican (Anderson Robbins Research and Shaw and Company research, respectively). These firms are quite respectable and their methodology usually quite sound. While Fox News itselfmight be a dumpster fire of credibility, their polls are usually fine. (With thanks to resident stats expert Mac Tan on this one.)
- Check the URL.There are hundreds of fake websites that look almost exactly like actual news sites and are off by just a letter or two. ABC News is an actual source, and generally highly respected. There’s a website at the URL abcnews dot com dot co, and it looks very similar, but is loaded with actual fake news instead.
Conventional source evaluation used to hold that .edu or .gov sources were likely to be reputable. Many educational institutions today host web content that is not reviewed or fact checked, much like Forbes and Reuters above. Government sources are still likely to be mostly credible, but the current administration has a tenuous grasp at best of what constitutes credible, reliable fact-based information and those running various agencies are little better.
Your mileage may vary here.
- Nobody else is talking about it.If your source’s headline or article isn’t reported literally anywhere else except a handful of blogs or some threads on Reddit, that’s a good clue that it’s not credible. There is no validity to the idea that there is a massive conspiracy to hide Teh Truth!™ from all of us by the mainstream media, only to be thwarted by a plucky band of YouTube commentors and Redditors.
Look for at least local news articles. Most small towns have at least a local radio station or weekly newspaper that publishes local news, even “Cow Crosses Road; Traffic Halted for Hour.”
If it’s something that seems nationally important and even the local herald doesn’t report on it, odds are it never happened.
- View foreign government-run sources with a healthy skepticism. Some of my conservative friends have started sharing a lot of things from RT. RT is literally Russia Today, a Russian state-run media corporation. For the love of God, don’t. This is literallya propaganda arm of the Russian government.
Now, not all foreign sources are bad. The Times of India, British Broadcasting Corporation (BBC), and Der Spiegel are excellent journalism and well-respected sources.
But don’t rely on foreign state-run media as a rule.
- Edit: Al Jazeera is a bit of an oddity here. It’s generally pretty well-respected and credible, but it’s also the state media arm of Qatar. It’s not without controversy for that precise reason. I take it seriously as a source, but I always follow up to see if anyone else is also reporting it and whether there’s anything missing.
- Edit 8/26/19: Some Indian residents are telling me that the Times of India isn’t what it used to be and has become far less reputable.
Snopes, Politifact, and other fact checkers are usually very good at avoiding these kinds of red flags.
Snopes, Politifact, and other fact checkers are generally accurate and honest arbiters of truth because they don’t engage in shady journalistic practices, verify and disclose their sources so anyone can decide that if they don’t want to take the fact-checker’s word for it that they can follow up and do the math themselves, and have reputations for correcting mistakes.
They have good journalistic integrity and that is why they are regarded as high quality, reputable, credible sources.
Mostly Standard Addendum and Disclaimer: read this before you comment.
I welcome rational, reasoned debate on the merits with reliable, credible sources.
But coming on here and calling me names, pissing and moaning about how biased I am, et cetera and so forth, will result in a swift one-way frogmarch out the airlock. Doing the same to others will result in the same treatment.
Essentially, act like an adult and don’t be a dick about it.
- Getting cute with me about my commenting rules and how my answer doesn’t follow my rules and blah, blah, whine, blah is getting old. Again, ornery enough today to not put up with it. Stay on topic or you’ll get to watch the debate from the outside.
- If you want to argue and you’re not sure how to not be a dick about it, just post a picture of a cute baby animal instead, all right? Your displeasure and disagreement will be duly noted. Pinkie swear.
I’m done with warnings. If you have to consider whether or not you’re over the line, the answer is most likely yes. I’ll just delete your comment and probably block you, and frankly, I won’t lose a minute of sleep over it.
Debate responsibly.
Footnotes
[2]You’re not going to believe what I’m about to tell you – The Oatmeal
[3]Liberals and Conservatives Are Both Susceptible to Fake News, but for Different Reasons
[4]https://www.apa.org/news/press/releases/2018/08/fake-news
[5]Dunning–Kruger effect – Wikipedia
[6]https://www.scientificamerican.com/article/do-the-eyes-have-it/
[7]Revisionist History Season 3 Episode 4
109.3K views
View Upvoters
View Sharers