Researchers at Kyoto University discovered that people find it difficult to distinguish AI-generated haikus from those written by human poets, and that human/AI collaborations led to the most interesting haikus. But perhaps this only works in Japanese.
Researchers at Kyoto University discovered that people find it difficult to distinguish AI-generated … [+]
AI is getting better at creating visual art, and it’s getting pretty good at realistic conversations, but how does it fare with poetry? Kyoto University researchers put it to the test by asking hundreds of volunteers to read haiku poetry and rate it aesthetically. The volunteers also had to indicate whether they thought the poems were written by humans or by an AI, and here’s where things got interesting.
Of the eighty different haikus used in this experiment, forty were written entirely by human poets, and the other forty were written by an AI. But because the researchers knew that the quality of AI poetry could be improved by having humans choose the AI-generated elements, they split the AI-generated haikus in two groups: half (20) were entirely created by AI and the rest had some human involvement.
When the volunteers rated the poems, they didn’t know which ones were AI-generated, which ones were made by AI and humans together, and which ones were entirely written by human poets. The ones that they rated as the most aesthetically pleasing were not the poet-written ones, but the haikus that were generated by an AI and then selected by humans. This suggests that a combination of humans and AI could result in more interesting haikus than humans alone.
The volunteers were asked to indicate which poems they thought were AI-generated and which ones were written by humans. That could help isolate any biases the volunteers had, and that turned out to be quite interesting. “They were supposed to be unbiased but instead became influenced by a kind of reverse psychology,” study leader Yoshiyuki Ueda told Kyoto University. “In other words, they tended to unconsciously give lower scores to those they felt were AI-generated.”
Despite that, the AI poems scored at least as well as the human-created ones, because people were very bad at guessing which haikus were created by AIs or not.
The study was done with Japanese haikus, but it inspired me to ask OpenAI ChatGPT to write some haikus in English. It didn’t go entirely as planned, mainly because syllables are far less predictable in English than they are in Japanese. When I specifically asked ChatGPT to write me a 5-7-5 haiku about AI’s ability to write haiku I got this in return:
AI’s skilled hand
Crafts haiku with ease, it seems.
Natural as can be.
In most English accents, that first line is decidedly four syllables, and the last line six. (Although “natural” can be squeezed into two syllables – natch’ral). I asked ChatGPT if it knew what a 5-7-5 haiku meant and I got a perfect explanation, so that wasn’t the issue. But when I asked “How many syllables are in the sentence ‘AI’s skilled hand’” I got a confusing answer. It wasn’t four or even five, according to ChatGPT, but SIX. Here’s what it said.
“The sentence “AI’s skilled hand” has six syllables [sic]. In English, syllables are typically counted by the number of vowel sounds in a word or phrase. In this sentence, the first word “AI’s” has two syllables [sic] (“AI” and “s”), the second word “skilled” has two syllables [sic], and the third word “hand” has one syllable. Therefore, the total number of syllables in the sentence is six [sic].”
Image showing that ChatGPT incorrectly counts syllables, explaining why it has trouble with haikus … [+]
(Notice that besides the wrong number of syllables in two of the three words ChatGPT also calculates 2+2+1 incorrectly)
So I don’t think it will be quite as hard to guess which haikus are AI-generated if they’re done in English, but here too there is potential for humans to improve on the output and only select ones that are structurally correct. I’m sure poets can think of fun and creative ways to use AI in their work.
