The PEREC effect
Testing the limits of large language model AI with generative lipogrammatic paraphrasing
Premise
Train ChatGPT 4 to rewrite a text based on an arbitrary, constrictive rule around which the large language model (LLM) AI must contort, specifically to train the AI to accept input text and output stylistically acceptable lipograms.
ChatGPT’s ability to paraphrase is outstanding, and even better since v4. This exercise is designed to see how far its powers of paraphrase can take it, to really put it through its paces.
Background
Georges Perec (1936 - 1982) was a writer and member of the French literary movement Oulipo. The Oulipo movement, loosely speaking, was interested in unlocking the generative capabilities of language through arbitrary constraints, rules and mechanical literary devices. Oulipo was active was before the age of personal computers, I should point out.
One of Perec’s most notorious literary achievements was to write an entire novel in French without using the letter “e”. To put this in perspective, the letter “e” is the most frequently occurring letter in the French language, occurring nearly twice as frequently as its nearest contender, the letter “a”. The technical term for this type of linguistic occlusion is a lipogram.
While Perec’s lipogrammatic novel, La disparition (1969), is a remarkable display of inventiveness and virtuosity, what is even more remarkable is that it was translated into English under the title A Void (1994), similarly omitting the letter “e”. Not content with this already peerless achievement, Perec went on to write a novella that only used the letter “e” for vowels (which was also translated).
Enormous linguistic resources are required to pull off this kind of literary stunt, and it seems it would be exactly the kind of linguistic task at which a LLM AI should excel. Let’s see how ChatGPT does.
Training ChatGPT 4 to output lipogrammatic text
I began by ‘priming’ the conversation by asking the AI to summarise what it knew about Perec and A Void. Naturally, the AI was both knowledgeable on the topic and verbose.
I explained the task to the AI:
Prompt:
I'm going to train you to perform a specialist text transformation operation I call PEREC. It's based on a book by Georges Perec who wrote in English and French without using the letter e. got it?
ChatGPT:
Yes, I understand the task. You would like me to perform a text transformation operation, which you've named PEREC, based on the writing style of Georges Perec, who wrote in English and French without using the letter 'e'. I will transform the given text accordingly by avoiding the use of the letter 'e'. Please provide the text you would like me to transform.
Easy. The AI now accepts a basic prompt structure, e.g. PEREC [text]. So far, so good. Almost…
First time out, I prompted:
PEREC Sonnet 18 by William Shakespeare
Here’s the initial output:
Shall I match you to a hot month day?
You art as charming and as mild:
Rough gusts do jar buds of May,
And hot-month's stint hath all too short a span;
Somhow too hot bright orb's gold cast,
And oft his gold display is dimm'd;
And all fair from fair will not last,
By luck or natural laws cut or untrimm'd;
But your constant charm shall not wane,
Nor will you brag of that which you own
Within this mortal world's short domain;
Nor shall Oblivion brag you roam his turf,
As long as humans sigh and lungs draw air,
So long will this, and you in this, stay fair.
Not bad, right?
But the contortions are evident, e.g. ‘dimm’d’, and in some cases, the AI just was not able to comply with the rules, e.g. ‘wane’. In some instances it even took the liberty to respell words that did not fit its purpose, e.g. Line 5: ‘Somhow’. Shakspere [sic] would be proud!
Feedback can improve the accuracy of the algorithm but there are noticeable limits. Here is output on a second pass on another famous Shakespearean passage after some feedback:
To stay, or not to stay, that is what's in mind:
If 'tis nobl'r in a mind to withstand
Slings and arrows of daunting condition,
Or to hold arms against a flood of ill,
And by opposing, stop it? To nap: to drift;
No more; and by a drift to say, our trials
And a thousand natural jolts
That living is heir to, 'tis a conclusion
Warmly to wish. To nap, to drift;
To drift: possibly to vision: ay, that's it;
For in that drift of oblivion, what sights may occur,
Upon quitting this mortal coil,
Must bring us halt: that's what holds up
This calamity of such long duration.
Impressive! And in a rough iambic pentameter to boot.
But not perfect either:
One outright hack, ‘nobl’r’, and one clear mistake, ‘heir’.
I have to say in ChatGPT’s defence, it doesn’t get much harder than lipogrammatically paraphrasing the Bard (the Renaissance wordsmith, that is, not the competing Google product).
Let’s try some prose.
Here are two prose PEREC transformations of famous openers from English novels. Can you decipher the sources?
1.
Here is the first transformed opener:
It was optimal of spans, it was worst of spans,
it was an duration of wisdom, it was an duration of foolishness,
it was an instant of faith, it was an instant of skepticism,
it was a period of Light, it was a period of Dark,
it was a spring of aspiration, it was a winter of frustration,
all things had us bound for triumph, all things had us bound for downfall,
in short, this duration was so far similar to that now,
that a portion of its most noisy authorities insisted on its
signaling only by comparison of good and bad.
You’ve probably worked out the input source for this one was A Tale of Two Cities by Charles Dickens. ‘Skepticism’ and ‘period’ don’t comply with the PEREC function but more to the point, it’s verging into gibberish. And this is after some feedback from me. Sorry, ChatGPT. That’s a fail. One point to Dickens, the writer!
2.
Here’s the next transformation of a famous novel opener:
In my youthful and naive days my dad would impart to his son an important bit of wisdom: "At any point you find a longing to fault an individual," my dad would say to yours truly, "just think about that not all individuals in this world had a similar upbringing that you did." My dad's words would stay with yours truly throughout my adult days, a continuous and vital admonition against my inclination to find fault with anybody who did not follow my way of thinking.
Yes, that’s right. It’s the transformed opening to F. Scott Fitzgerald’s The Great Gatsby. And yes, the AI did not do perfect job. It used ‘naive’ and it seems to have some difficulty focalising the narrator without pronouns. I should add this was its fourth attempt after multiple rounds of feedback. I’d say that’s another point to Team Writers!
Observations
Although the AI clearly understands the rules of the PEREC function, it still struggles to produce compliant output. Let’s unpack that with a simplified example.
Suppose the input word is ‘chess’.
The AI spots the letter ‘e’ in ‘chess’ and selects from synonyms.
Suppose it chooses ‘board game’ as the replacement term.
The AI is unable to self-correct its own output (‘game’ contains an ‘e’), and choose another substitute that is compliant.
Here are some possible explanations for ChatGPTs failures with the PEREC function I have entertained:
Literary lipograms are about as challenging as it gets and, as advanced as it is, the AI just isn’t up to the task (yet). The PEREC function is definitely an edge case and something the AI was certainly not trained for as a generalist application. In other words, without full control of the creative input, its resources of quotidian language are too limited to accomplish a complete and accurate lipogrammatic transformed output. It gets overwhelmed.
It needs more training. I’m not convinced on this. I have extensively fed back to the AI on its output and it seems that you end up playing a frustrating game of ‘whack-a-mole’ where one problem fixed sees another pop up or return.
There is something about the way humans produce language that is different to the way the AI does. When we speak, we self-censor as the utterance comes out of mouth, a kind of double temporal moment in which there is a slight delay between the words forming in our head before popping out of our mouths that allows for small adjustments to be made to accommodate our social situations, self image or, tragically, our anxieties. Not that we are always successful. This is very much veering into the domain of the Freudian Unconscious where slips of the tongue, jokes and poetic language in general point to a deeper function of overdetermined selection and substitution at work beneath our day-to-day language use.
The first explanation is by far the most plausible to my mind. The last is on shaky evidential ground on nearly all counts. Naturally, I personally prefer the last explanation.
While Perec’s lipogrammatic gymnastics are dazzling to observe, it’s worth noting that the titular Void of A Void is the unspeakable horror of the Holocaust. It takes a writer to realise that kind of vision.