Attraction to But

High contrast, low meaning

Counterintuitive ideas have staying power, even when they’re false. Napoleon wasn’t particularly short, humans use far more than 10% of our brains, and Einstein was a fine math student. Myths like Einstein being a poor student persist because they’re so surprising; what would it mean about our education system, or how we identify talent, if the smartest guy of the 20th century almost fell through the cracks?

More specifically, many debunkable factoids create a sense of contrast. You’d expect powerful generals to be physically large, but one of the greatest ones was actually small. You’d assume that you already use your entire brain, but actually our potential far outstrips our current performance. If I walked around spreading misinformation that Einstein had thin hair, or that Napoleon had longer than average toes, nobody would pay attention. Just like babies enjoy looking at high contrast patterns and mystery novels sold in airports feature dozens of plot twists, the human epistemological apparatus is enticed by the counterintuitive.

I’m not sure this bias is solvable, or even desirable to solve; when a surprising claim is true, it’s good to reflect on it and notice what it implies about your worldview. But it’s worth noticing when the machinery of contrast is unearned, and an idea is being compared to something that, on reflection, is just vapor or straw.

Good and Bad Buts

Contrast is an atomic component of human perception and experience, so if your guard went up every time you saw the word “but”, “although”, or “however”, you’d become an anxious mess. But (eek!) these words are a signal to confirm that both sides of the contrastive pair are meaningful.

For example, let’s look back a couple of paragraphs. The first sentence:

I’m not sure this bias is solvable, or even desirable to solve; when a surprising claim is true, it’s good to reflect on it and notice what it implies about your worldview.

and the second:

But it’s worth noticing when the machinery of contrast is unearned, and an idea is given extra credence by being contrasted against something that, on reflection, is just vapor or straw.

I posit that this is a good but, because the two sides really are in contrast, and neither is otherwise useless padding. The claim that we shouldn’t try to stop being allured by surprising stuff can stand on its own two legs, separate from the claim that sometimes, the mechanism of surprise is co-opted to prop up weak ideas. The two sentences are stronger for their interrelation.

So what would a bad but look like? On the same topic, suppose I’d written:

Everyone loves a good surprise, but it’s important to stay vigilant against misinformation.

Slop! Balderdash! When you look closely, these two claims are flimsy cards that can only stay up by leaning on each other. “Everyone loves a good surprise” is either false or tautological, but either way it adds no actual informational content. Likewise, “it’s important to stay vigilant against misinformation” offers no, well, information about how to stay vigilant, or why it’s important. It’s easy to read (or write) a sentence like this in a low-attentional state, nodding along, without noticing that the contrastive structure is all there is.

This sort of low-information-density is a hallmark of sermons and occasionally poetry, where I’ll begrudgingly say it’s fine; if you know a piece of writing is merely attempting to stoke your passions, so be it, that’s a fine form of entertainment. But when a piece of writing purports to actually say something, each side of a contrasting pair should be compelling on its own.

(As to whether the previous paragraph passes the very test it sets out, well, I’m on the fence. Close reading is not for the faint of heart.)

The Vibes Machine

LLMs are masters of form, to a far greater extent than content. They very often will construct rhetorical flourishes that sound profound (or at least cogent), but that have no actual insight. This is a big part of what constitutes justly maligned “AI slop”, and once you start looking for it, it’s hard to miss.

For instance, here’s a GPT-4o-generated sentence about the topic of widely believed falsehoods:

While skepticism is valuable, outright dismissing those who believe misinformation can be counterproductive; instead, engaging with curiosity, presenting well-supported evidence, and acknowledging why the false claim seems plausible can foster more productive discussions and critical thinking.

If you’re new to generative AI, and simply trying to evaluate how smart these systems are, then sure, this is impressive. In particular, its structure is impressive - it is shaped like an insight-rich piece of text. But every individual claim within that structure is… not quite there.

While skepticism can be valuable

I guess? Valuable to what end?

outright dismissing those who believe misinformation can be counterproductive

It just seems like it has to depend, and any actual insight on the topic would come from the nitty gritty of what it depends on. Like, you almost certainly should “dismiss” someone who is bellowing their “misinformation” about the government in a CVS parking lot, while you shouldn’t merely “dismiss” a coworker who, on the basis of “misinformation”, wants to take a doomed approach to a shared project.

instead, engaging with curiosity, presenting well-supported evidence, and acknowledging why the false claim seems plausible can foster more productive discussions and critical thinking

Still nothing much here. It’s fine advice to “present well-supported evidence”, “fostering more productive discussions” seems like a nice thing to do, and everybody’s happy to laud “critical thinking”, but none of this is actual insight. And indeed, I think a lot of the backlash to AI-generated text comes from the contrast between the vague positive feeling that comes from the structure of that text, and the eventual realization that none of it meant very much.

There are lots of other ways that AI nails structure even when it offers no concrete insight; for example, AI often tees up lists of three where each item is vague. But pointless contrast is a particularly common hallmark - I only had to ask for a single paragraph on a single topic to have a ready example.

As a final warning, this is one of those pet peeves that you end up stuck with pretty quickly. Once you start seeing bad buts, they’re hard to unsee. A lot of text starts seeming soggy, fluffed up, mere junk food.

The best stuff, though? Not one but out of place.