Babylon claims chat bot provides health advice ‘on par’ with doctors
Image credit: dt
Babylon, a digital healthcare company, has announced that its chat bot performs as well as human doctors at providing patients with health advice, following testing using examinations for trainee GPs.
Babylon is best known for its ‘GP at Hand’ service, which connects patients with GPs for remote appointments, for either a monthly fee or on a pay-as-you-go model. More recently, the company has been working on an artificial intelligence (AI) backed chat bot to pre-screen patients’ health conditions with a symptom checker. The chat bot can then offer medical information and advice based on these symptoms.
The company used diagnostic questions from exams used by the Royal College of General Practitioners (RCGP) to assess trainee GPs, in order to test the effectiveness of the app.
According to Babylon, its AI scored 81 per cent on the RCGP exam, whereas the average mark for human trainee GPs from 2013-18 was 72 per cent. The company carried out further tests with other organisations, including the Royal College of Physicians, which demonstrated scores of 80 per cent for accuracy, while the handful of doctors it compared its results with fell in the range of 64-94 per cent. The AI also received high scores for accuracy when assessed for safety and for conditions seen most frequently in primary care. The company announced its results at an event at the Royal College of Physicians in London.
According to Ali Parsa, founder and CEO of Babylon: “Babylon’s latest [AI] capabilities show that it is possible for anyone, irrespective of their geography, wealth or circumstances, to have free access to health advice that is on par with top-rated practising clinicians.”
“Tonight’s results clearly illustrate how AI-augmented health services can reduce the burden on healthcare systems around the world.”
However, the RCGP pointed out that Babylon used MRCGP exam preparation questions, which are for revision purposes and not necessarily representative of the full range of questions that trainee GPs face in the MRCGP exam. Those questions are not in the public domain and had not been provided to Babylon.
This element of the chat bot’s assessment used two of the three examinations sat by trainee GPs in the UK; the multiple-choice applied knowledge test (AKT), which assesses knowledge, and clinical skills assessment (CSA), which assesses ability to gather information, then make and communicate evidence-based decisions with the patient.
“[Babylon] is talking about the doctor’s ability to diagnose problems, but being a good GP is about far more than this,” Dr Andrea Clay, a member of the RCGP, GP trainer and CSA examiner, told E&T.
“The CSA exam is the final component of the MRCGP and this is a face-to-face exam judging the doctor’s ability to diagnose, assimilate information and make sensible, safe, competent decisions. We are testing so much more than how to make a diagnosis. It’s about attitude, communication skills, picking up cues and listening to the patient to make patient-centred management decisions that are appropriate, acceptable and effective - all in 10 minutes.”
Professor Martin Marshall, Vice-Chair of the RCGP, said that Babylon’s claims were “dubious”.
“The potential of technology to support doctors to deliver the best possible patient care is fantastic, but at the end of the day, computers are computers, and GPs are highly-trained medical professionals. The two can’t be compared and the former may support, but will never replace, the latter,” he said in a statement.
“No app or algorithm will be able to do what a GP does.”
In a statement to CNBC, Mobasher Butt, Babylon’s medical director, described the RCGP’s criticism as focusing on “shoring up an outmoded and financially self-interested status quo which solely works to the benefit of a limited number of partner GPs, rather than celebrating a scientific achievement which has the potential to improve the lives of patients and clinicians globally.”
This is not the first public spat between Babylon and professional medical bodies. Its GP at Hand service attracted criticism for increasing the heavy workload of GPs and causing some patients to be re-registered at different GP practices. The planned rollout of the service beyond London was put on hold following a formal objection lodged by NHS England.