AI loves splashy contrast. When a chatbot identifies something, it’ll first identify what it’s not. That’s not shirking your duties, it’s practicing self care. That’s not empathy, it’s Machiavelli hiding behind a fig leaf of therapy speak. That’s not… you get the picture. (They also like clustering examples in threes.)
Everything in modern AI is a big simmering multicausal gumbo, so there’s not just one reason for this tendency. Some of it is probably because it signals active listening, and the models are trained to be good listeners; if I say I’m worried about X, correctly labeling X before comforting me is a strong conversational move. Some of it is probably because it feels (well, used to feel) human and natural, and models are trained for that too.
But enough of this pussyfooting. It’s time for the real reason, the one true reason, which I have intuited and will henceforth elucidate:
It’s greedy1 token sampling, doinkus!
What?
The way LLMs (that’s the cooler letters for what we call “AI” these days) work, is they think one word at a time. Technically, it’s actually closer to one syllable at a time, but I’ll use “words” in this post to simplify, and jam all the complexity into a footnote.2 LLMs actually take the entire conversational context, everything that anyone has said so far, every time they pick a single word. This is very unlike humans, who unless very drunk tend to think in chunks and nail down several words in advance.
What does this have to do with things being Not X, But Y? Let’s consider a sample exchange.
User: I’ve been really struggling to get out of bed lately, and it’s got me feeling really bad. Switching from night shifts half the time to checking up on my mom in the nursing home when she’s at her most lucid in the early mornings on my off days is… oof. I just end up on my phone scrolling Insta for an hour at the start of every day. Wasteful.
AI: Wow. That’s
What comes next? Let’s trace out some really likely next sentences; after all, an AI’s job is to pick a word that will most reliably approximate the sorts of sentences that might show up in the wild on the internet, and also get a thumbs up (or objectively correct answer) from a human grader.
Here are a few options.
That’s understandable! Taking care of an ailing parent is draining even without factoring shift work.
That’s really tough. You might already know this, but anyone in your position would be struggling.
That’s brutal. The time with your mom is so valuable, but a night shift job probably makes it really hard to be present in the mornings.
These are all fine, and would likely get good scores as continuations. So “understandable,” “really,” and “brutal” are all plausible picks.
However, here are a few more options:
That’s not wasteful, it’s being a human person with universal limitations.
That’s not just a sleep disturbance issue, it’s an emotionally charged family obligation on top of a perpetual circadian disruption.
That’s not one major life challenge, but two on top of each other in a trench coat.
These all would also score pretty well, but they all have the same next word! Even if these three options were all less well-scoring than the first three as sentences, when picking the next individual word they’d all count as a single bloc.
If LLMs computed sentence-by-sentence, they’d choose whatever sentence fit the best. But since they compute word-by-word, they’ll tend to choose whatever specific word fits with lots of plausible sentences. “Not” (or “isn’t” or whatever) is a very strong candidate. You’re never going to paint yourself into a corner saying “not” next, just like you’ll never get fired for choosing IBM.
A good metaphor is vote-splitting in a primary election. A straightforward next word might have several possible synonyms, and they split the vote for that sentence structure. But all the negation sentences rally behind the word “not,” so “not” beats the competition.
What Else?
A lot of LLM-isms kinda click into place, if you look at them with the “one word at a time” frame. An em dash is a very versatile punctuation mark; you can throw it in before many different sentence structure pivots, and all those pivots get counted as a bloc when figuring out what to type next. When you’re listing examples, three is the most likely number for a typical sentence, so the LLM will decide to add another example after each of the first two, then close it out with the third. If LLMs operated at the paragraph level rather than the word level, they might throw in more two-example or four-example sentences for variety, but when you’re greedily picking one word at a time, it’s very difficult to choose anything but the most common structure every single time.
The pattern is most obvious in AI fiction. AI just picks the same names over and over and over again. “Kael” is the most likely single name if there’s a male character with a fantasy vibe, so it just always picks Kael. If it’s a modern guy, it goes with Marcus. It feels insane to ask for ten unrelated stories and get the same protagonist name every time. But if it’s the most likely single word for that slot once, it’s going to be next time, too.
This is a big part of LLM smell that is hard to wash out. You can train the em dashes out of your AI, or (maybe) get ChatGPT to stop using the word “goblin” three times in every conversation, but whatever new attractor basin it falls into will be fresh exactly once, then a little grating, then unbearable. This is true for more abstract LLM-isms too. Even if you can’t put your finger on it, your instincts eventually do notice a colorful, splashy anecdote always appears on the exact same beat, and get bored.
In other words, it’s not incidental, it’s architectural. And that matters.
Yeah I know, the technical term “greedy” is incorrect because temperature isn’t 0, and it’s actually a matter of softmaxing surfacing negation as a plausible option disproportionately often relative to the underlying distribution at the sentence or paragraph levels. So sue me!
Even more technically, what they do is construct a vocabulary of the most common combinations of letters, give each a number, and pick the number associated with whichever one comes next. These combinations of letters are called “tokens.” So “the” is a likely token, because “the” is very common. But “commotion” might split into three tokens, “com”, “mo”, and “tion”, because “commotion” isn’t that common a word but “-tion is a common ending to a word. If you’re lost, don’t worry about it. “Words” is close enough.
what can i say, it is not just a substack post, it is a very nice explanation of a common LLM phenomena :)
Many authors on substack show this pattern. I thought they were copying each other