Very thoughtful. Just the sort of ideas I’m struggling with as I both finish up my first novel (written over 5 years the old fashioned way) and explore AI fiction through my Substack TAILS. I too have noticed over the course of my blog that, yeah, it’s getting better. Lit journal worthy? Not quite, IMHO, but I’d give a creative writing student an A. It is a weird moment indeed for us writers.
Yeah, I suppose to be clear I was thinking of fairly low end lit journals, like ones that accept ~10% of submissions and don't pay. The more exclusive ones, I'd be surprised.
I think I’m both more impressed than you are with the outputs of older models and less impressed with these, which gives me a different sense of the rate of improvement. When I look back at some of nostalgebraist’s old posts, including the one from 2019 featuring GPT-2 fine-tuned on Nabokov, I’m struck by just how gorgeous some of the imagery is. Of course it struggles with coherence even within a sentence, which makes it nearly useless at writing stories, but there’s something there. (Newer models can write much more coherently, but I don’t think they’ve rid themselves of the problem entirely.)
I preferred Kaj/Claude 3.6’s Barthelme pastiche about the pigeons to the sci-fi/fantasy stories he generated with Opus 4. Like a lot of LLM writing, the former has an overload of arbitrary whimsy, but there were moments I found genuinely delightful. (“They aimed for the moon but landed in Staten Island, which they declared close enough,” for example.) The sci-fi stories were more coherent and perhaps better-written in some formal sense, but they also felt ponderous and repeated the same imagery/themes over and over. (The two stories are also remarkably similar—were they both generated using a specific prompt?) I also thought the Sam Kriss pastiche from Jasmine Sun/Deepseek R1 was decent—there were a lot of empty words, to be sure, but it was a fine pastiche and that’s not unimpressive!
I’m caught between the awe that machines can do any of this at all and the realization that, were any of these stories written by a random human, I would have no interest in them. If my best friend sent me the story about Kai and Sam, I could tell them sincerely that I liked it; if I were a high school creative writing teacher and a student handed it in to me, I would want to have a chat with them about their potential as a writer. But if it were the first story in a collection I was thinking of buying, I would put it down. When I read Kaj/Claude’s Barthelme pastiche next to the Barthelme story that Kaj chose as inspiration, there’s no comparison. And whenever I start to think that current LLM writing is getting really clever, I turn to whatever book I have nearby (Virginia Woolf, Patricia Lockwood, Penelope Fitzgerald, Réjean Ducharme…) and I still generally feel like there’s no comparison. That’s not to deny that there’s something truly remarkable about how far LLMs have come, but I still don’t generally want to read what they produce.
It does seem like a fool’s game to try to predict just how good AI creative writing will get in the coming years, so I don’t want to assume that Claude won’t be scaling Parnassus. I do somewhat suspect that, if the goal is to have LLMs produce long-form works with an original voice that appeal to a discerning, literary audience, the path from GPT-2 to where we are now is easier than what’s ahead. But perhaps even next month someone will figure out how to prompt Opus 4 to get something undeniably brilliant—so I prefer to hedge my bets.
Thanks - this is pretty close to how I feel, too, with the exception that I am extra annoyed with overwrought/too cute writing in general, and thus more impressed with recent progress. But it's a slight difference.
I very much agree with the awkwardness of "wouldn't read a book of these stories, would be bewildered to find such a book lauded, would be relieved if a friend sent it to me (I could honestly say it was pretty good!), would be impressed if an undergrad submitted it to me."
What to do with that is mysterious. My actual gut says that REALLY good fiction, or even fiction that would make me feel like "ah, it has surpassed me" is still pretty far off. But it moved faster than I expected in the last three months, so well, hmm.
Very thoughtful. Just the sort of ideas I’m struggling with as I both finish up my first novel (written over 5 years the old fashioned way) and explore AI fiction through my Substack TAILS. I too have noticed over the course of my blog that, yeah, it’s getting better. Lit journal worthy? Not quite, IMHO, but I’d give a creative writing student an A. It is a weird moment indeed for us writers.
Yeah, I suppose to be clear I was thinking of fairly low end lit journals, like ones that accept ~10% of submissions and don't pay. The more exclusive ones, I'd be surprised.
I think I’m both more impressed than you are with the outputs of older models and less impressed with these, which gives me a different sense of the rate of improvement. When I look back at some of nostalgebraist’s old posts, including the one from 2019 featuring GPT-2 fine-tuned on Nabokov, I’m struck by just how gorgeous some of the imagery is. Of course it struggles with coherence even within a sentence, which makes it nearly useless at writing stories, but there’s something there. (Newer models can write much more coherently, but I don’t think they’ve rid themselves of the problem entirely.)
I preferred Kaj/Claude 3.6’s Barthelme pastiche about the pigeons to the sci-fi/fantasy stories he generated with Opus 4. Like a lot of LLM writing, the former has an overload of arbitrary whimsy, but there were moments I found genuinely delightful. (“They aimed for the moon but landed in Staten Island, which they declared close enough,” for example.) The sci-fi stories were more coherent and perhaps better-written in some formal sense, but they also felt ponderous and repeated the same imagery/themes over and over. (The two stories are also remarkably similar—were they both generated using a specific prompt?) I also thought the Sam Kriss pastiche from Jasmine Sun/Deepseek R1 was decent—there were a lot of empty words, to be sure, but it was a fine pastiche and that’s not unimpressive!
I’m caught between the awe that machines can do any of this at all and the realization that, were any of these stories written by a random human, I would have no interest in them. If my best friend sent me the story about Kai and Sam, I could tell them sincerely that I liked it; if I were a high school creative writing teacher and a student handed it in to me, I would want to have a chat with them about their potential as a writer. But if it were the first story in a collection I was thinking of buying, I would put it down. When I read Kaj/Claude’s Barthelme pastiche next to the Barthelme story that Kaj chose as inspiration, there’s no comparison. And whenever I start to think that current LLM writing is getting really clever, I turn to whatever book I have nearby (Virginia Woolf, Patricia Lockwood, Penelope Fitzgerald, Réjean Ducharme…) and I still generally feel like there’s no comparison. That’s not to deny that there’s something truly remarkable about how far LLMs have come, but I still don’t generally want to read what they produce.
It does seem like a fool’s game to try to predict just how good AI creative writing will get in the coming years, so I don’t want to assume that Claude won’t be scaling Parnassus. I do somewhat suspect that, if the goal is to have LLMs produce long-form works with an original voice that appeal to a discerning, literary audience, the path from GPT-2 to where we are now is easier than what’s ahead. But perhaps even next month someone will figure out how to prompt Opus 4 to get something undeniably brilliant—so I prefer to hedge my bets.
Thanks - this is pretty close to how I feel, too, with the exception that I am extra annoyed with overwrought/too cute writing in general, and thus more impressed with recent progress. But it's a slight difference.
I very much agree with the awkwardness of "wouldn't read a book of these stories, would be bewildered to find such a book lauded, would be relieved if a friend sent it to me (I could honestly say it was pretty good!), would be impressed if an undergrad submitted it to me."
What to do with that is mysterious. My actual gut says that REALLY good fiction, or even fiction that would make me feel like "ah, it has surpassed me" is still pretty far off. But it moved faster than I expected in the last three months, so well, hmm.