By Misha Bereznyak
(This isn’t a topic I typically write about, but I think some of you might find it interesting because practically all of us are exposed to AI, and it touches on the ability to use it to form and refine ideas.)
Recently I started experimenting with using AI to help my writing beyond just minor editing, and this revealed very interesting limitations of its potential – at least for the type of writing I am doing.
My idea going into this wasn’t to give an LLM a one-line prompt and for it to generate a full written post – that would obviously be completely useless because it will contain none of my thinking about any subject. (If anything, usually I would start with something much longer than the final result, and then trim it down.) Instead, the idea was for AI to assist me in the places that are most time or energy-consuming.
So I set out to build a system that would achieve that.
The results have been quite counterintuitive for me. It appears that AI can be quite useful for some tasks where I thought it would struggle. But other tasks, the ones I thought would be most naturally suited for it, turned out to be completely impossible.
When developing the system, I provided a substantial volume of my work, and the AI was able to do a deep analysis of it to understand my writing style – not only on the surface level, but also my structural thinking. Theoretically, it gained enough understanding to be able to recreate the same type of writing.
The process itself was divided into a few distinct steps.
Step 1, which wouldn’t be relevant in all cases, is to develop an argument that is not formed yet. Usually in such cases, the process will start with a very long and poorly organized text, and it’s not clear what the main thesis is and which details are relevant, or what major ideas are missing.
This step would usually start with the AI reviewing the material and stating what it thought the main argument was, and then it would go through back-and-forth with me, developing the arguments and filling any gaps in logical sequence, reasoning, or details.
I found AI to be quite effective for this step at identifying potential directions, probing me to decide or refine them, and looking at high-level gaps in thinking. I even found it effective when reviewing messy transcriptions from a long time ago, to help me quickly identify what ideas there might be worth looking into further.
Step 2 includes organizing an effective structure and producing a clear structure map of what will later expand into the full text. The map includes the bare points and all the relevant details from my original text and from what was developed during the conversation. This step worked too, though here it was more cumbersome, and only somewhat better than doing it on my own.
Step 3 is turning this structure map into text. This will be our main focus and we will be back to it shortly.
Step 4 includes editing and refining a roughly completed text for minor improvements. Besides some use of the previously derived understanding of my terminology and style, there is nothing special about this step, and this is likely something that most of us have experimented with when writing.
Now back to step 3, where I take the structure map and convert it to text. When I am writing on my own, if I have the information about the structure and the relevant details, and the points I am trying to convey are mostly settled, putting it in writing is relatively straightforward – if a bit time-consuming.
If the point I try to convey is not fully resolved, I am more likely to struggle putting it in writing – but that was the purpose of having a back-and-forth with the LLM to work out what exactly I wanted to say.
The LLM did have this information, and also a very detailed understanding of my way of writing. And since writing and transforming text is the most basic task for an LLM, this should have been very straightforward.
But it wasn’t straightforward at all.
In fact, it turned out to be a complete disaster.
And upon more investigation it turned out to be completely impossible.
For each part of the text, I would usually get one of two results: most often, generic AI nonsense, and sometimes, a restatement of the same information without any added value.
When the point that had to be written did not carry enough information, the LLM would fill it with invented things that I never claimed, while also using a generic voice. When the point carried a lot of detail, it would make minor transformations without developing or explaining it.
Many of the points in the map were developed through the back-and-forth in earlier steps, so it made sense to expect that the LLM would not write them in exactly the same way as I would – though in practice, even these already developed points were not just “not exactly” how I would write them, but rather had zero relation to anything I would want to say.
But the more surprising failure was with the points in the map that were already well-formed in the original draft. All the original meaning was there when they were condensed into points in the map. The LLM even had access to the original text as a reference right there in the map. And yet, the LLM was equally unable to expand the points back to anything that I would ever say.
I tried a lot of different ways to overcome this. A lot of the focus was on trying to explain to the AI that it should simply say the things that it’s trying to say rather than convey ideas through some kind of complex writing. But none of that helped.
Why was this the case? After trying for a while to refine the system and improve on it, I realized there is a deeper issue and set out to understand what was going on.
This included looking into specialized AI tools for writing and checking how well they handled this type of task, and more importantly, looking at what others who were trying to build and implement similar systems found out, and what did and did not work for them.
The consistent finding, across both the specialized tools and the people who had tried to build something similar, was that no one had actually solved the problem of expanding an outline of points into text. They found some workarounds: specifying each point in much more detail, providing the model samples of their own writing, training the model on their work, and going through more passes; these workarounds reduced how much was left to fix, but never got close on their own to what they would actually say.
There were two issues, both related to the underlying ways in which LLMs work: one small, the other fundamental. The small issue is the problem of voice, by which I mean how the text sounds – not word choices that carry the exact meaning. The problem is that these models are tuned for text that sounds typical, and as a result, they slide towards a standard register. Even if the model has a full understanding of your voice, it is not capable of reproducing it. It is possible to push it closer, but it’s not possible to push it all the way. Maybe more specialized tools would be able to train on somebody’s writing and solve part of this issue. But again, this is only a small part of the problem.
The fundamental issue, however, is the problem of meaning and substance. An LLM, even if it has the full description of the point and the full detail that needs to be included, is not able to generate the text with the meaning that I would write. The reason is simple, though it was far from obvious to me.
The text itself is the meaning.
There is no meaning that a sentence or paragraph encodes, that is external to the text itself. There is no such thing as a condensed message that lives externally to the text, that can then be expanded into a full text. Except, of course, if the expansion is done by a person who develops the meaning or already knows it.
An LLM can transmit existing meaning that precedes the words – it can condense, summarize, and explain. But it cannot create new meaning that only comes into being in the words themselves – because for it, that meaning does not exist yet.
That meaning cannot be carried in an outline, and the only real way for an LLM to get to that meaning is to provide it with a text that’s already very close to exactly what I myself would say – which practically means doing step 3 on my own, and keeping the LLM for minor editing.
For a quick thought experiment – while no LLM was trained on my writing specifically, most LLMs have been trained on a lot of Shakespeare’s writing and would easily be able to generate texts that would sound like Shakespeare, including in subtle ways. If Shakespeare had access to such an LLM, it could maybe help him with structuring, minor refinements, and proofreading – but it would likely still be useless to him for the writing itself, because it would not be able to say what he wanted to say in the way that he wanted to say it.
Where does it leave us? I still find structural organization and a back-and-forth to test ideas and find gaps quite effective with AI. And of course, it is very useful for minor editing. This is because in those cases, it does not need to come up with meaning. But the text itself, it cannot produce – and this appears to be a fundamental limitation.