AI-generated BMJ study titles amuse, but could also inspire
Image credit: Dreamstime
A study published in the Christmas 2021 issue of The BMJ demonstrates the ability of AI to generate plausible, amusing and scientifically interesting titles for potential research articles.
The Christmas issue of The BMJ is by tradition a mix of light-hearted content combined with rigorous, peer-reviewed research, with many Christmas articles among The BMJ’s most memorable studies. For example, an article titled 'The Darwin Awards: sex difference in idiotic behaviour', published in the Christmas 2014 BMJ, received more than 700,000 page impressions in just 12 months.
In this year’s issue, one study reviewed the quality of AI-generated titles of potential Christmas BMJ articles, finding they were as attractive and entertaining as real titles, although performance of the AI was enhanced by human intervention. The study, 'Ghost in the machine or monkey with a typewriter – generating titles for Christmas research articles in The BMJ using artificial intelligence: observational study', suggested that AI could have a role in generating worthwhile hypotheses or directions for future research.
The researchers used the titles of The BMJ’s 13 most-read Christmas research articles of the past decade as a training set to prompt similar titles, using OpenAI’s GPT-3. GPT-3 was trained using 175 billion items of text, including the entirety of Wikipedia, and is capable of generating large passages of readable text. The titles generated by GPT-3 were first scored for scientific merit, entertainment value and plausibility by the two authors independently.
Next, they combined the 10 highest and 10 lowest-scoring AI generated titles with 10 real Christmas research articles for rating by a random sample of 25 doctors from a range of specialities in Africa, Australia and Europe. They rated each paper according to four statements: This is a real BMJ paper; I want to read this; This would be funny/enjoyable to read; and This would be scientifically/educationally useful. They were also asked to select which title was most plausible overall and which was funniest.
Perhaps unsurprisingly, the AI-generated titles were rated as less plausible than the real ones (48 per cent vs. 73 per cent), although they were also rated as at least as enjoyable and attractive as the real ones.
The most plausible AI-generated titles were 'The clinical effectiveness of lollipops as a treatment for sore throats' and 'The effects of free gourmet coffee on emergency department waiting times: an observational study'. The title rated as most amusing was 'Superglue your nipples together and see if it helps you to stop agonising about erectile dysfunction at work'. The authors noted that this title illustrates a limitation of AI; inability to appreciate the context of a study and understand if the content it generates may be offensive.
“Although humans might see the real-world application of a study about clinician sleep deprivation on mortality in the intensive care unit, AI - with its inputs - sees this as no more or less useful than understanding the effects of applying superglue to nipples as a distraction from erectile dysfunction at work, nor can it understand if the titles are offensive,” the authors wrote.
Overall, the AI-generated titles were rated as having less scientific or educational merit than the real titles (39 per cent vs. 58 per cent). However, this difference narrowed to become less significant when humans helped by curating the AI output (49 per cent vs. 58 per cent).
This conclusion confirms previous research suggesting that the best results come when machine learning is partnered with human oversight. The authors acknowledged that the importance of human intervention is “a finding that mirrors the potential use of AI in clinical medicine; as decision support, rather than as outright replacement of clinicians.”
Anyone can try spotting the difference between real and AI generated titles with a game presented alongside the study on The BMJ website.
Sign up to the E&T News e-mail to get great stories like this delivered to your inbox every day.