Technically, ChatGPT can do (some of) the work of a human editor. But an experiment comparing three separate human edits of a literary short story to edits by ChatGPT exposes AI’s serious limitations.
Writers have been using AI tools for years – from Microsoft Word’s spellcheck to the passive-aggressive Grammarly. But ChatGPT is different.
The first version had been rejected by literary journal Overland, but its fiction editor Claire Corbett had given generous feedback. The next version received detailed advice from freelance editor Nicola Redhouse, a judge of the Big Issue fiction edition . Finally, the piece found a home at another literary journal, Meanjin, where deputy editor Tess Smurthwaite incorporated comments from the issue’s freelance editor and also their typesetter in her correspondence.
We chose it because we believe the literary genre, with its play and experimentation, poetry and lyricism, offers rich pickings for complex editorial conversations. Interestingly, ChatGPT did not pick up that the story was now published and attributed to an author. Raising questions about its ability, or inclination, to identify plagiarism. Nor did it define the genre, which is one of the first assessments an editor makes.
Stage two: AI writes Next, we provided a second prompt, responding to ChatGPT’s initial feedback – attempting to emulate the back-and-forth discussions that are a key part of the editorial process. The author’s unique descriptions were changed to familiar cliches: “rugged beauty”, “roar of the ocean”, “unbreakable bond”. ChatGPT also changed the text from Australian English to US spelling and style .
The biggest problem is that final transition – I don’t know how to read the narrator. Her emotions don’t seem to fit the situation. I’m not 100% sold on this line. I think it’s a little confusing … It might just be too much hinted at in too subtle a way for the reader. Skills include confident conviction, based on deep accumulated knowledge, meticulous research, cultural mediation and social skills.
For the follow-up, we asked specifically for an edit that corrected any issues with tense, spelling and punctuation. This makes sense when you know the datasets that drive ChatGPT mean it explicitly works toward the word most likely to come next. ChatGPT came back with suggestions for how to revise specific parts of the text, but the suggestions were once again formulaic. There was no attempt to offer – or support – any decision to go against familiar tropes.
Our typesetter has queried one thing: on page 100, where “Not like a thing at all” has become a new para. He wants to know whether the quote marks should change. Technically, I’m thinking that we should add a closing one after “not a thing” and then an opening one on the next line, but I’m also worried it might read like the new para is a response, and that it hasn’t been said by Elsie. Let me know what you think.
A major problem that emerged early in this experiment involved ethics: ChatGPT did not ask for or verify the authorship of our story. A journal or magazine would ask an author to confirm a text is their own original work at some stage in the process: either at submission or contract stage. As we’ve seen, when writing challenges expected conventions, AI struggles to respond. Its primary function is to appropriate, amalgamate and regurgitate – which is not enough when it comes to editing literary fiction.