Hogwarts-style enchanted portraits created from art and photographs
Image credit: Dreamstime
Computer scientists based at the University of Washington have developed a technique for transforming 2D images into 3D animations, allowing a subject to appear to 'leap out' of their photograph.
Previously, it was thought impossible to animate a person leaping out of a photo on the basis of a single image, due to the lack of photographs taken from different angles.
“This is a very hard fundamental problem in computer vision,” said Ira Kemelmacher-Shlizerman, a professor at the University of Washington’s school of computer science and engineering. “The big challenge here is that the input is only from a single camera position, so part of the person is invisible. Our work combined technical advancement on an open problem in the field with artistic creative visualisation.”
The algorithm, 'Photo Wake-Up', can take a person from a 2D photo or artwork and make them appear to run, walk or jump out of the frame. The system also allows viewers to watch in 3D using augmented reality tools.
The algorithm begins by identifying a person in an image and creating a mask of the body’s outline. It then matches a 3D template to the body position, marking its limbs and other body parts necessary to identify for animation. In order to ‘warp’ the template to look convincingly like the person in the photo, it then projects the 3D person back into 2D. According to the researchers, it is very difficult to manipulate objects in 3D without errors being made very obvious during animation.
Once the template has been warped to match the shape of the person, the algorithm generates the back of the person’s body using information from the image and template, then binds the two pieces together to create a 3D figure which can be seen from all angles. The background hidden by the figure in the original image is filled in using information from the rest of the image, so that a blank patch is not left behind when the person moves away from their original position.
While the animated people move in a manner that can be slightly unnatural, like the movement of a character in a modern video game, the results are impressive.
The technique can be applied not just to photographs but also to abstract depictions of people in artworks. While the algorithm can animate some photographs in which the subjects have their arms crossed or blocking parts of their bodies, it is not yet capable of animating subjects with their legs crossed.
The researchers have tested out their algorithm on a photograph of athlete Stephen Curry; a photograph of Neil Armstrong standing on the surface of the Moon; the cover of The Beatles’ album Help!, and various artworks, including Matisse’s Icarus. They envision this algorithm being used in numerous applications, including to allow people to interact with paintings and photographs in museums, bringing children’s drawings to life and to create avatars for gamers based on their own appearance.
Sign up to the E&T News e-mail to get great stories like this delivered to your inbox every day.
