Rolling back the years: is de-ageing the future of film-making?

When Industrial Light & Magic’s (ILM) visual-effects technology converged with world-renowned film director Martin Scorsese, the result is the much-talked about Netflix-backed film ‘The Irishman’. But while there’s been considerable hype around the venerated director’s return to the mob violence genre, together with his reunion, after nearly 25 years, with Robert De Niro and Joe Pesci, joined in the film by Al Pacino, the real buzz has been around the so-called ‘youth’ technology used to de-age the famous septuagenarians.

The technology in question is a new markerless performance-capture system, developed by ILM, that’s based on light and texture, together with a new software system, FLUX, that alters the shapes of faces without animation. The leap forward has been in its ability to deform and change the younger faces with rendering and compositing, while bypassing traditional keyframe animation – the drawing or ‘frame’ used in filmmaking, marking out which movement the viewer will ultimately see.

Scorsese wanted to free-up the actors from traditional performance capture so they could truly be themselves, while incorporating tricky flash-back/fast-forward sequences portraying characters at different ages in a story arc spanning over four decades. All this, on three of the most iconic actors of our times, whose faces are wholly familiar to audiences at the various stages of their long careers, in films like ‘Taxi Driver’, ‘Raging Bull’, ‘Heat’ and ‘Goodfellas’.

Did Scorsese’s risk pay-off? Have these time-defying, much-debated regenerative facial effects been successful and, if so, what does it mean for telling film-stories in the future? As this game-changing tech gets better over time, what other implications will it have?

Based on Charles Brandt’s book ‘I Heard You Paint Houses’ – a reference to the blood spatter across walls and floors at the site of a mob killing – ‘The Irishman’ chronicles the life and times of Frank Sheeran (De Niro), a World War Two veteran-turned-hitman whose story intertwines with that of the notorious head of the Teamsters labour union, Jimmy Hoffa (Pacino) and Philadelphia crime boss Russell Bufalino (Pesci).

It’s in Sheeran’s memories and flashbacks that we see the youthification tech at play. Perhaps it’s in Scorsese’s resolve to have the characters’ faces de-aged, rather than the actors’ faces themselves, giving ILM a broader canvas with which to work, that has meant the risk has paid off, pleasing viewers and critics alike... mostly. In terms of success, its recent triumph at the Golden Globes, as well as ILM’s place on the Academy’s longlist for the VFX Oscar, seems to shout ‘yes’.

Led by Pablo Helman, ILM’s visual-effects supervisor/second unit director, the ILM team took as their focus, in line with Scorsese’s vision, the arc of the characters’ stories, with a view to de-aging the actors to their characters’ younger selves, rather than the other way around. Another significant issue for Helman was the actors were not to be restricted by intrusive headgear, allowing them to perform on set in the normal way. To do this, ILM had to resolve the tracking and face replacement with some major innovations in technology.

Helman explains: “‘The Irishman’ presented a specific set of challenges that were met with FLUX. How do you take the technology away from the director and the actors so that performance is the only thing you go for? Once the actors are left without intrusion, their performances are closer to the ‘truth’, and are 100 per cent believable.”

FLUX is ILM’s proprietary markerless on-set software for facial capture. Helman says “it allows the actors to be on set under theatrical lighting and wear no markers on their faces, or helmet cams on their heads. There are no restrictions on space or camera movement for the director. This system is currently the only system available in the world to allow for markerless, on-set lighting facial-performance capture.”

In preparation, the actors were all shot separately, everyone performing a variety of facial movements, and then repeated on a special light stage to capture a range of lighting conditions. The resulting data was used to generate 3D models of each actor. In addition, Helman’s team researched and amassed thousands of historic photographic and film references of the three main actors’ younger performances at specified ages, and developed an AI-based Face Finder program to show them, in real time, the closest match when de-aging.

Another innovation on set was a brand-new camera rig. Helman realised that if the actors weren’t going to be wearing tracking markers or head rigs, then the team had to obtain the maximum amount of information from the actors’ texture and lighting: “To complement the software, ILM designed a three-camera rig, which included two high-​resolution ARRI [Arnold & Richter Cine Technik] infrared modified ‘witness cameras’ attached to and synched with the primary RED [Red Digital Camera] director’s camera.

“A proprietary infrared ring was placed in front of the infrared cameras to light the actors’ faces and neutralise unintended shadows while remaining invisible to the production camera. This system captures facial performances from multiple angles and provided a full set of data of every frame of each performance and all the on-set lighting and camera positions that would be needed later. All performances were procedurally captured and solved by ILM’s FLUX,” he adds.

Without conventional markers, how did the ILM team ensure they captured enough data so the de-aged faces looked authentic, and didn’t end up looking weightless and waxen or, worst of all, inadvertently comical?

Helman explains: “The capture is accomplished by leveraging texture, shading and lighting cues in the image to infer geometric deformation. The base geometry is deformed on a frame-by-frame basis and the result is a fully rendered 3D geometry mesh that is fully textured and lit. Because no markers are used on the actors’ faces, the software interprets the pixel movement on the face and the deformation creates a more realistic render than other methods.”

However, it’s not just about the faces. You can make a group of 70-somethings look young, but how do you make them ‘act young’ in order to create an authentic representation? Fail, and the performance becomes slightly off-kilter, especially for an audience that remembers what a young De Niro looked, sounded and moved like.

Since the film didn’t use body doubles, the ILM team digitally altered the actors’ bodies. Helman also realised the actors’ hands would need a lot of attention. Hands age very distinctly and the ILM team worked on adding and altering numerous digital hands.

“There were hundreds of shots that were touched up in CG to create the illusion of a younger body,” he explains. “We also had a ‘movement analyst’ on set to help the actors, day in and day out, to get to grips with the different challenges of acting the appropriate age.”

Yet for all its digital wizardry, has ‘The Irishman’ got away with it? The effects aren’t perfect and, alongside the accolades, the film has received plenty of flak too. However, perhaps the limitations of its CGI have worked to its artistic advantage in this instance? Framed as a morality tale based on the memories of an old man looking back over his life, the film version of the ‘young’ actors could be said to mirror the hazy, neither-here-nor-there realm in the way that memory often works.

A new technology heralding incredible new storytelling opportunities it may be, but what other implications does it have? We’ve come a long way from techniques like smearing Vaseline on camera lenses to create a soft, dreamy aesthetic in a film.

It could be said the turning point took place in 2006 with the release of ‘X-Men: The Last Stand’ when effects company Lola Visual Effects, using a toolbox of physical and digital techniques, de-aged actors Patrick Stewart and Ian McKellen in a flashback scene.

Lola Visual Effects remains at the forefront of de-aging technology to this day, working almost exclusively on the Marvel and X-Men series, reversing time on actors including Robert Downey Jr, Kurt Russell and Michael Douglas. The company was also the studio behind the mind-boggling 100-year age transformation of Brad Pitt in ‘The Curious Case of Benjamin Button’. Then there are the other well-known visual effects companies such as Weta Digital, famous for The Hobbit franchise, and, of course, ILM itself, which, alongside ‘The Irishman’, has worked on numerous blockbusters including the Star Wars and Marvel franchises; all are now major players in the world of narrative de-aging.

For all its artistic and technological brilliance, what does it mean for the entertainment industry where total human animation is now within reach and long-dead actors of the past can be routinely resurrected? In recent years we’ve seen well-known actors like Peter Cushing brought back to the screen as Grand Moff Tarkin in Star Wars spin-off ‘Rogue One’, and, of course, Carrie Fisher as Princess Leia in ‘Star Wars: The Rise of Skywalker’, but with the news that James Dean is set to star in a new film, 64 years after his death, what does this mean for actors who are currently alive and trying to earn a living, and what are the legal and ethical implications?

Although the film’s directors have the approval of Dean’s family, the decision has provoked a noisy backlash across social media. Dean’s character has no link to the actor himself or to the story arc of the film, with many suggesting his ‘casting’ is nothing more than a marketing gimmick.

For Helman, the answer lies in asking important questions at the very beginning of the process: “Does the VFX methodology support the story? Why are you doing it? Where does the performance come from? These are hard questions to answer. Once you do, you’ll know if it’s worth doing.”

The flipside of the de-aging coin might spell an extraordinarily long acting career for some, but does it mean that audiences are starting to prefer illusion over reality? The technology can only get better, and entertainment law is already starting to shift to take this into account. States including California have introduced legislation granting individuals the right to state how their image can and can’t be used for up to 70 years after their death.

The issue of fake news and the implications of deep-fake technology are ever-expanding, becoming part of public discourse as they rise up the political agenda. It’s therefore no surprise that other areas of our lives are permeated as the boundary between illusion and reality becomes increasingly blurred and the spheres of human law and ethics seek to catch up.

The experience of ‘The Irishman’ seems to point to youthification technology as the ultimate film-making tool, freeing up the story-teller’s imagination, as well as the actors themselves, to tell stories unburdened by time and physical limitation.

As Helman concludes: “Markerless facial-capture technology is going to change the way we capture data on set. It will unburden the actors and free them from old restrictions. Performance will benefit.”