AI coverage is extraordinary — in both senses of the word. Some of the most important technology stories of our era are being reported in real time. But the same stories are frequently misframed, overstated, or quietly funded by the companies being covered. Here's how to read more carefully.
Eight Red Flags in AI Coverage
These aren't signs that a story is wrong — they're signals to slow down and read more carefully.
-
"Breakthrough" without specifics
In AI coverage, "breakthrough" often means "better than the previous benchmark on this specific test." Ask: breakthrough at what? Compared to what? Does it generalize beyond the test conditions?
-
A CEO is the only quoted expert
Press releases dressed as journalism. When the only people quoted about a model's capabilities are the company that built and is selling it, the story lacks independent verification. Look for third-party researchers who have actually tested it.
-
Benchmark numbers with no context
"Scores 92% on the MMLU benchmark" sounds impressive. What's human performance? What does the benchmark test? What does it not test? Benchmarks are measuring instruments — they measure what they measure, not general intelligence.
-
"Scientists" or "researchers" without names or institutions
Vague attribution is a flag for unverifiable claims. Real research has authors, institutions, and usually a paper. If the source isn't named, ask why.
-
Anthropomorphization without qualification
"The AI felt confused," "the model wanted to help," "it understood the task." These framings aren't necessarily wrong, but they carry implicit claims about machine experience that aren't established. Good coverage uses more careful language.
-
A demonstration video with no independent replication
AI demos are often carefully curated. The famous Google Gemini demo that showed real-time understanding of video was, on closer inspection, produced from still images with significant post-processing. Watch demonstrations critically and wait for independent tests.
-
Timelines stated as facts
"AGI by 2027." "Full automation of X jobs by 2030." AI timelines are among the most contested predictions in technology. When stated as near-certainties, they usually reflect the source's incentives more than the evidence.
-
Existential fear without technical grounding
Fear-based AI coverage often suffers the same problem as hype-based coverage: it outpaces the actual technical evidence. Serious AI risk arguments are detailed and technical. "AI will destroy humanity" without mechanistic explanation is not one of them.
A Vocabulary of Vagueness
Certain words in AI coverage carry more implication than information. Here's what they often actually mean.
A Rough Hierarchy of Sources
Not all AI coverage is equal. Here's a rough tiering, not as a definitive ranking but as a starting framework for evaluating what you're reading.
The gold standard. Read the abstract and conclusion even if you skip the methods. Check whether it's been independently replicated. Preprints (not yet peer-reviewed) are one step below published papers.
MIT Technology Review, Wired (technical pieces), 80,000 Hours, LessWrong (for safety debates), The Alignment Forum. Writers who name their sources and engage with technical detail.
The New York Times, The Guardian, FT — when they're reporting on AI with named researchers and independent verification. Good for context and implications; occasionally shaky on technical specifics.
Useful primary sources for what labs say about themselves. Read with appropriate skepticism — they're communications strategies, not independent assessments. Often technically detailed and worth reading, but not neutral.
Huge range. Some excellent, much poor. The incentive structure favors engagement over accuracy. Apply all red flags listed above. Never let social media be your only source on a consequential AI claim.
Common, especially for smaller publications. If the story could have been written without any reporting — just by reading the company's announcement — it probably was.
A Quick Checklist for Any AI Story
- Who is making the claim? What's their incentive?
- Has this been independently verified or replicated?
- What specific task or benchmark does this apply to?
- What does this NOT tell us — what are the limitations?
- Is there a link to the underlying research or data?
- Are experts outside the company quoted by name?
- Does this story require me to believe a claim that would be remarkable if true?
- Is the headline stronger than what the body of the article supports?