3D selfies created using neural network photo transformation
Image credit: Dreamstime
Computers scientists based at the University of Nottingham and Kingston University London have created a web app which can – for the first time – transform a single 2D photograph into a 3D model of the face pictured.
The research was led by PhD students in the University of Nottingham’s Computer Vision Laboratory, in collaboration with staff from Kingston University’s School of Computer Science and Mathematics.
Attempting to construct the geometry of a face from flat images has frustrated graphics and vision experts for years. Systems which perform this task today require multiple, specific images of the face from different angles, and often face problems such as non-uniform lighting and varying facial expressions. The result of these problems is that the final products often look uneven and unrealistic, much like different 3D jigsaws meshed together.
The University of Nottingham researchers bypassed these issues by handing the task over to an artificial neural network.
“The main novelty is in the simplicity of our approach which bypasses the complex pipelines typically used by other techniques,” said Professor Yorgos Tzimiropoulos of Nottingham’s School of Computer Science, who supervised the project.
“We instead came up with the idea of training a big neural network on 80,000 faces to directly learn to output the 3D facial geometry from a single 2D image.”
Convolutional network networks are a type of artificial neural network – a system which learns to detect patterns and perform tasks by processing huge amounts of data rather than being specifically programmed – often used to analyse images and videos.
For instance, a convolutional neural network could be trained to recognise the likeness of Ed Balls by being given many thousands of photographs of faces, some of which are labelled as ‘Ed Balls’. Although these deep learning techniques are in widespread use, artificial neural networks are often described as ‘black boxes’: similar to human brains, it is extremely difficult to so much as begin to understand how these highly complex systems work.
The researchers trained a neural network on a dataset of tens of thousands of photographs and 3D facial models, and using this information, the neural network was able to reconstruct the geometry of a 3D face from a flat image.
The resulting app is able to reconstruct a reasonably good 3D model from a single 2D photograph, and can construct the non-visible parts of the face based on its training data.
“Our [convolutional neural network] uses just a single 2D facial image, and works for arbitrary facial poses (e.g.: front or profile images) and facial expressions (e.g.: smiling),” said Aaron Jackson, the PhD student who led the project.
Visiting the project’s website, you can upload a selfie, and in less than a minute, receive a 3D model of your face. Although the technology is at an early stage, and the resulting models often fall into ‘uncanny valley’ territory, it is a significant step forward for the field and has proved popular among internet users: more than 400,000 people have tried it so far.
According to the researchers, this technique could prove useful not just in face and emotion recognition, but also to personalise augmented reality and video games with 3D avatars with the faces of the users. It could even have some medical applications, such as in simulating the results of plastic surgery to share with a patient.