Obsidian Roundup Newsletter
Join the newsletter to receive the latest updates in your inbox.
Let’s say you’re taking your first steps into the wide world of learning stuff independently, and are being inundated with all sorts of “fake news” and propaganda, questionably accurate pop-sci books and outright lies, clickbait headlines and the inevitable result of memetic games of telephone played out on social media. How do you find sources you can trust? How can you evaluate information without devoting your entire life to becoming an expert in every subject?
For a variety of reasons, teachers in a schoolhouse setting often have a hard time teaching this skill. This should not surprise anyone; teaching critical thinking is hard and so is teaching notetaking. One reason source evaluation is tricky is that teachers must navigate the political waters of our district and parents can get pretty angry if teachers criticize their preferred news outlet. Another is because we’re required to teach fine-grained distinctions between primary, secondary, and tertiary sources … but due to time and cognitive constraints it gets oversimplified down to “primary sources are best!” even though primary sources are harder to interpret without context. A third reason is because students doing formal research projects are highly incentivized to “play it safe” by sticking to obviously reliable sources and therefore get very few chances to learn by experimenting with edge cases, because then they might “fail” and failure has consequences; lost time, lost prestige, and shame, among others.
But where does that leave folks after graduation?
A lot of smart folks recommend writing and notetaking and learning in public because it’s easier to keep yourself honest when you’re doing stuff in front of people whose opinions you care about. It’s the same reason that it’s a good idea to sign up for a couple of competitions if you’re learning ballroom dance; deadlines and social pressure are better at forcing you to improve than vague goals.
But since nobody likes embarrassing themselves unnecessarily, here’s my quick-reference guide for evaluating the reliability of stuff you read on the internet.
Does the reference cite its sources?
My biggest recommendation is to look for references that cite their sources, or are at least willing to do so when asked. Plenty of accessible writing references more academic texts and even if you have difficulty parsing them, the fact that they’re actually basing their information off of real scholasticism is a good (although not sure-fire) sign.
My pop history books are liberally annotated with things like “ok WHICH scholars say that? Don’t just say scholars says!” and “ORLY” and “double-check.” I don’t think this makes them bad books, it just means that it is a book, not a research paper. It would render the book unreadable to cite every claim even if you didn’t count how much it would increase the wordcount—the book would balloon in width way past what bookbinders can handle. Technically this is speculation, but I’m basing it off of how liberally cited my education research papers were during my master’s program, and how often I have to wade through lengthy inline citations when I read journal articles. Most people don’t enjoy having to skip a third of the line for every sentence.
For that matter, I analyze pop books differently than I analyze pop articles. Pretty often pop articles don’t bother with citations and references at all, so it’s an easy top-level way to separate the stuff worth reading from the crap that’s not even worth my time. The rest of the time, they’ll link to the original study and I’ll try to read that instead. That’s what we mean by “primary sources are more reliable” — a lot of science journalists have a shaky understanding of what reports are actually saying, or go out of their way to sensationalize things. This is true for politics too, by the way, which is why I like to read actual proposed laws myself.
The danger, of course, is that often you will lack the expertise to correctly analyze such a primary source. Non-experts lack context. I’m a critical consumer of historian-created media, not a historian myself—although I am a history teacher, the credentialing process is a lot more focused on the teaching pedagogy than my actual knowledge of history, which was tested with… literally one test that I hope an average high-schooler could have passed.
Are the sources reliable?
You can usually get a sense of how reliable a pop science book (or news article, or tweet, or whatever) is by looking at the size and reputation of the publisher. Big 5 Publishing Houses and major universities can’t always be relied on (they will sometimes ‘chase that dollar’ or let a mistake slip through) but you’re less likely to publicly embarrass yourself citing a book put out by Harvard University Press than a small publisher known for pushing, say, holocaust denialism.
Media sources can often be evaluated on the basis of what their incentives are. The replication crisis uncovered in the early 2010s was not usually the result of bad-faith liars, but it is still a pretty big problem. Journals aren’t typically punished for publishing bad papers, universities sometimes outright refuse to investigate blatant frauds, and academics are pressured to publish regardless of publication quality. Journalists are incentivized to publish sensational things. Propaganda outlets are trying to get people to buy into a particular worldview, for whatever reason. This doesn’t automatically mean they’re lying or that the information is wrong (a man going into space can be sensational and true), but it does raise the bar for a double-check.
This is one reason I tend to stick to learning about relatively obscure eras of history & neat animal facts. Reasonable people might disagree about how horseback riding got invented, but it’s usually pretty easy to figure out what perspective my sources are coming from. At a certain point, you just have to decide whether the author is acting in good faith and then take responsibility for confirming any claims that you are planning to make.
Don’t repeat claims you haven’t confirmed
If someone makes a claim that you feel inclined to repeat, triple-check it. There aren’t enough man-hours to confirm everything you read, whether it matches your pre-existing biases (and everybody has them, they’re a natural consequence of having a knowledge base) or not. But if you’re going to repeat a claim or integrate it into your worldview in such a way that you might reference it in a discussion, it behooves you to be able to explain with dignity where you learned that little tidbit, and no one respects you when your response is “well I read it from some guy in a Twitter thread.”
Just because a claim was published in a respected textbook that was recommended by an actual historian doesn’t mean that every other historian agrees with it. Especially if the source is older, go see if there have been any updates in the literature. Just because multiple sources repeat a claim doesn’t mean you’ve confirmed it, by the way — If every source you look at repeats the exact same claim in the exact same way, chances are you’ve run afoul of some kind of SEO article farm that’s just regurgitating stuff from Wikipedia or random other articles. More on this later.
Can you track the claim back to its origin?
I recommend trying to read widely, because it will help hone your instincts. If somebody makes a claim that doesn’t “feel” right, track it down. But even if you have no context to evaluate the truth of a particular claim, you can evaluate how broad the claim is. Extremely broad claims like “every single culture has a 10% rate of mothers lying about paternity” are almost certainly false.
Find out what other people say. This is good practice anyway, because even people who aren’t trying to peddle bigotry and nonsense are often wrong because of perfectly normal biases that aren’t, like, awful. Robert Heinlein once said, “if everyone knows thus-and-such, then it ain’t true, but at least a thousand to one” but if all the comments on a Reddit thread are pointing out why the original post is incorrect and they’re all providing links to journal articles that seem to accurately support their claims (double-check this, because sometimes people will post links to “support” that actually directly contradicts their claims!) — well, maybe there’s something to what they’re saying.
Also, pay attention to who created the source. Google them. Are they a professor at a respected institution? Are they at the center of any Twitter controversies (this doesn’t immediately disqualify them from being reliable, but it should give you a sense of their positions)? What does their “About” page say; is it a satire account or a widely published Yale professor? What does their purpose for writing seem to be? Do they come across as someone who loves a particular time period, or do they seem to be really interested in making a polarizing point (which again doesn’t automatically mean they’re wrong, but it is a factor to consider). For example, if a particular article is dead set on talking about Vikings in a particular way… is that person writing a whole bunch of articles making the same point, and really hyper focused on that point, or are the rest of their articles looking at other facets of Viking life that don’t make that point?
Who agrees with them? Are the people agreeing with them engaging in something that feels academically or intellectually rigorous, or are they just happy to have “someone who sounds smart” agreeing with their pre-existing biases? Do they generally seem willing to engage with alternative viewpoints and admit they are wrong?
So, let’s use an example as a practice exercise.
Pretend I’m looking for information about when waterwheels were invented, I’m going to start by googling it, because I’m not writing a dissertation or anything I’m just looking to learn something about a history thing I found interesting after having visited the Netherlands on vacation.
I’m not a Medievalist or anything, and I’m definitely not an expert on whatever you need to be an expert on for waterwheels to fall under your purview, so let’s say I get hits for 3 books and 4 websites. I skip the books for now because I’m looking for an immediate answer to a simple question.
One article is from what looks like an encyclopedia website with a bunch of articles, but, drat! It’s paywalled because I already read 4 great articles from there. I skip it for now — although if I hit this paywall often enough, I’ll buy a membership.
The next article doesn’t have academic references, but it has a couple of hyperlinks to what look like sources and seems to be made by a hobbyist. Paydirt!
The third article is from a website trying to sell landscaping rocks that used to be millstones, so it’s definitely not academic and they’re definitely trying to convince me to spend money, so I know that whatever they’re claiming, I’m going to need to double-check, but I still read it because maybe I’ll get some useful search terms out of it.
The 4th article reads like a ranty screed trying to convince me that a particular ethnic group is the best because they invented waterwheels and a bunch of other early technologies. I stop reading after the third paragraph and make sure to click back as fast as possible, so Google knows it was a crappy result it should de-prioritize in SEO.
Then, because I’m interested now, I go back to the books. First, I look at the cover. Does it look accessible, or does it look like somebody’s dissertation? I’m smart, but dissertations are long, filled with a bunch of stuff I don’t usually care about, and not usually written to a layman audience even when they’re about something interesting. I skip the one that looks like it’s intended for a super dry academic audience. (Note: I do sometimes read dissertations. Next on my list is The Life Cycle of Disability in Ancient Greece by Deborah Sneed)
The next one looks interesting and is written at my level and is even filled with citations, but the reviews are a mix of creepy and scathing. I leave it off my wishlist.
The third one looks more promising. The reviews seem enthusiastic about how this book addresses new archaeological finds, it’s making a claim I haven’t heard before, and it’s been published by a reputable publisher. The sample pages appear heavily footnoted. It even has a neat cover! I put it on my wishlist and get it for Christmas and start reading it.
But whoa, some of the claims it’s making are not things I’ve ever heard before. The author addresses the previous state of scholarship and even sounds convincing about why they’re right and their predecessors are wrong. (I know of no new scholarship about waterwheels, but I did recently get two “popular history” books for Christmas that do exactly this — the Amazons by Adrienne Mayor and The Golden Thread by Kassia St. Claire and I am actually in the midst of going through this process with them).
So now what do I do?
First, I make sure to read critically. I annotate heavily, making liberal use of the ORLY owl and notes-to-self to follow up on because honestly, even the absolute best, most well-considered, most thoughtful and well-intentioned books is going to say stuff that’s outside of the expertise of the author.
My husband was listening to a podcast that goes super into depth about I think WWII (I don’t recall, but it was definitely an American war) and the guy was clearly an expert on that war. But he made some throwaway comment about dragoons that my husband took on faith and like, I knew it was wrong, dragoons were not invented by Napoleon or whatever outlandish thing he claimed, and I wound up in a long argument with my husband about it because he then felt like he couldn’t trust anything the guy said and was really upset that a source he thought was reputable had recommended this podcast and like now he felt like he couldn’t trust anything from that source and…
… look, nobody’s perfect. So before you go getting excited about some hot new thing you learned, even if it’s from a source that really seems great, double and triple check it. Because that waterwheel story I was telling?
Every single one of the sources I found online claimed that they were invented in 4000 BCE. Which is insane. The wheel was invented in 3500 BCE in Mesopotamia and I know that off the top of my head because I happen to teach a unit on Mesopotamia (I’m a social studies teacher, but everyone will have different bases of knowledge). The waterwheel is attributed to one of Alexander the Great’s generals in the same breath as the 4000 BCE date.
So because I am terrible with dates, I google Alexander the Great, who it turns out was active around 400 BCE, and get my answer: somebody typo’d.
Anyway, once I’ve found a claim that I think I might ever want to repeat, I go looking for verification. Half the time I don’t bother with the actual source cited by the book, because that stuff is usually behind a paywall I can’t access, unless my Google Fu fails me. Then I try to get access to the source, or email the author (authors are often thrilled to respond!) for more information, or ask over on a heavily moderated subreddit like r/AskHistorians. I once found a historian able to help me explain all the weird differences between sources claiming to explain the origins of coinage — apparently people like to erroneously claim that their ethnic group is responsible for that invention.
If you come across a claim you’d like to integrate into your worldview or repeat, try to track it back to its source and evaluate the original claimant. This goes doubly for claims that seem surprising or unintuitive but can’t be dismissed out-of-hand. Consider: does the original claim match the one you read? If so, try to figure out if the original claimant is a crackpot or a respected institution, and whether or not their claim is considered controversial. How likely is it that they are incentivized to publish something that isn’t true?
How embarrassed will you be if you get corrected in public? You can’t do this kind of rigorous fact-checking for everything, so a good rule of thumb is to consider the return on investment. If you’re wrong about whether horses were first ridden before or after domestication, probably no one is going to be harmed by you being wrong in public. But it might be pretty dangerous to accidentally spread misinformation about, say, safe food preservation practices.
Be careful out there & good luck!
Note: There are a couple of affiliate links & codes scattered around, but these always come from links I was already recommending and usually I share them because they benefit you too (i.e. getting you extra time on trials).