Food-tracking AI system to reduce malnutrition in care homes

It is estimated that more than half (approximately 54 per cent) of residents of long-term care homes are either malnourished or at risk of malnutrition.

At present, food intake is primarily monitored by staff who manually record estimates of consumption by looking at plates once residents have finished eating. This approach is laborious and subjective, limiting clinical inference capabilities.

A new smart system, developed in Canada by researchers at the University of Waterloo, the Schlegel-UW Research Institute for Aging and the University Health Network, uses artificial intelligence software to analyse photos of plates of food after the home's residents have eaten.

The researchers proposed a novel deep convolutional encoder-decoder food network with depth-refinement (EDFN-D) using an RGB-D camera for quantifying a plate’s remaining food volume relative to reference portions in whole and modified-texture foods.

The team trained and validated the network on the pre-labelled 'UNIMIB2016' food dataset and tested it on its own two novel long-term care-inspired plate datasets (689 plate images; 36 unique foods).

The sophisticated software examines colour, depth and other photo features to estimate how much of each kind of food has been consumed and calculate its nutritional value.

“Right now, there is no way to tell whether a resident ate only their protein or only their carbohydrates,” said Kaylen Pfisterer, who co-led the research while earning a PhD in systems design engineering at Waterloo.

“Our system is linked to recipes at the long-term care home and, using artificial intelligence, keeps track of how much of each food was eaten to make sure residents are meeting their specific nutrient requirements.”

Robert Amelard, a Waterloo alumnus and postdoctoral fellow at University Health Network, said the subjectivity of the human observation process results in an error rate of 50 per cent or more. By comparison, the automated system is accurate to within five per cent, “providing fine-grained information on consumption patterns.”

Researchers collaborated with personal support workers, dieticians and other long-term care workers to develop the system, which saves time as well as improving accuracy and would ideally be added to tablet computers already used by front-line staff to maintain electronic records.

The system is designed to provide improved transparency and to approximate human assessors with enhanced objectivity, accuracy, and precision. This also assists human staff by helping with early detection of malnutrition in long-term care homes and hospitals, where time and personnel resources are often constrained.

“My vision would be to monitor and leverage any changes in food intake trends as yellow or red flags for the health status of residents more generally and for monitoring infection control,” said Pfisterer, now a scientific associate at the University Health Network Centre for Global eHealth Innovation.

A research paper on the team's work - 'Automated food intake tracking requires depth-refined semantic segmentation to rectify visual-volume discordance in long-term care homes' - has been pubpished in the journal Scientific Reports.