‘The whole point of the process is to tell a good story’: Matt Brown, Imaginarium Studios

“For all the technology we have around us, the whole point of the process is to tell a good story.” Matt Brown is taking me on a tour of London’s Imaginarium Studios, which creates computer-generated performance motion-capture (mocap) for film, games and, more recently, mixed reality. If you’ve seen any of the ‘Lord of the Rings’ movies and are familiar with the character of Gollum, played by Andy Serkis, you already know what ‘mocap’ is. If you’ve seen Serkis’s performance as Caesar in the recent ‘Planet of the Apes’ franchise, you’ll know how far it’s come. “Andy’s performance as Caesar is completely driven by performance capture,” says Brown, “and that’s the sort of work we typically do here.”

Put simply, mocap creates digital renderings of real actors doing real things, put together by a combination of motion-capture cameras and reflective body markers. Serkis founded the Imaginarium Studios back in 2011 and its CEO is the 40-year-old Anglo-American Brown, who is keen to point out that while mocap might have made its name with Gollum, there’s more to the story than bringing Tolkien’s mythical creation to life. Today, among other things, Imaginarium is developing content for the Magic Leap spatial computing headset, putting the studio at the frontier of performance capture in mixed reality.

“Many of my conversations with clients and partners begin with the idea that they have a story to tell,” says Brown. “Often, they’ll say they’ve seen a studio do something similar in a particular way but will be wondering if we can bring their characters to life in a way that suits them. Often what we are asked to do has never been done, which means that we must find the right tech integration to enable that story to be told in a way that is not just interesting, but affordable and deliverable.”

As an example, Brown creates the hypothesis of a client wanting to capture a stage musical production. “That’s not been done before, so we have to set about bringing together the right mixture of technologies to motion-capture that and make it happen.”

For Brown this is different from making a film or a TV programme, where the process is “essentially performing to a camera, where the performance is locked into a string of ones and zeroes that becomes a moving image. When we capture something on our stage for computer-generated moving imagery, we literally capture every angle you can possibly imagine.” The Imaginarium ‘mocap volume’ (industry jargon for ‘motion capture space’) is designed so that more than 50 cameras capture the performer. “The workflow on the stage is a performer in a suit that has 53 reflective markers on it. These points are connected to make a digital skeleton. The movement of the actor then manipulates the skeleton, to which we can then apply muscle, flesh and costume, meaning the actors can then authentically puppeteer digital versions of themselves.”

For Brown, the place where performance capture is most likely to lead his technology is mixed reality, while in the entertainment space its most likely application is theatre. “Theatre is very much a mixed medium. You have human beings performing on a stage in costume in relation to set pieces. The digital proxy of that is what the spatial computing device Magic Leap does. With this device, you still see the world around you, but the digital content you see is spatially aware of the world you are in.” Hence ‘mixed reality’, rather than virtual reality, a development Brown explains with the scenario of when “a virtual object can walk across your real coffee table. Only when it walks behind your coffee mug, it is occluded by it. You see both worlds simultaneously. We are working out ways to develop content for this platform.”

Creating working mixed-reality demos for theatre is only part of the potential for the technology. “Ultimately, I can see mixed reality replacing your mobile phone,” says Brown. “Think about all the things we do now, looking down at that little screen in our hands. Well, with a mixed-reality device such as Magic Leap, you’ll be able to do all those things while looking ahead.” At this point, Brown demonstrates the Magic Leap headset with the qualifying statement that its current form factor is the first public iteration of the device. He says it is helpful to extend the mobile phone analogy: “When they first came out they were huge, with multiple batteries and so on. That’s essentially where we are now. But ultimately, the product will look very much like a regular pair of glasses. This is the point at which it will definitely be a mass-market, mass-adoption device.”

The level of physical and software engineering that has gone into Magic Leap has come on a long way since the days when early prototypes “were the size of a double fridge. Today we have a 200g headset tethered to a computing device smaller in diameter than a compact disc, capable of rendering high-quality graphics in real time. Huge strides have been made in a short while. For us, what’s exciting is that it’s a totally new platform – a new way for people to experience entertainment content. Yet, the functionality of the device also plays in lots of other verticals.”

Brown predicts mixed reality will have future applications in education, enterprise and health, where the gesture control function will allow user interaction. “Fundamentally, this changes how we can work with this digital content while, unlike VR, not blocked off from the world.

“Given what we already do, we were very keen to see how performance-capture content could work on the device. It wasn’t something that Magic Leap had ever tried before. When we were given the opportunity to develop content for the device, we were essentially creating a pipeline for how you do that. Nobody had any idea of how you could create a digital asset that could be rendered in a visually pleasing or technologically possible way for the device. We had to figure that out in a short space of time by taking our knowledge of real-time rendering and applying it to a mobile platform.”

After an iterative process in which Imaginarium developers found their facial expression and body rigs were ‘too heavy’ for the device, an approach was developed that satisfied requirements of how to drive characters with mocap data on the Magic Leap headset. The result was the recent announcement that Imaginarium had been named as the “the first performance-capture partner of Magic Leap”.

Brown describes the Imaginarium Studios as a “prominent resident” of the legendary Ealing Studios in west London, famous worldwide as the spawning ground of a host of post-Second-World-War classic black-and-white movies such as ‘The Lavender Hill Mob’, ‘Passport to Pimlico’ and ‘The Ladykillers’. These accolades are writ large and proud on the studio’s exterior walls, making it easy to forget that Ealing Studios also has an emerging 21st-century heritage that can boast being home to many of the interior sets on TV hit ‘Downton Abbey’. The Imaginarium Studios are integrated into one of these historical buildings that date back to the 1930s, and the sense of British film-making history is palpable.

Imaginarium was established by actor Serkis in 2012. “After his time with Peter Jackson, playing Gollum in ‘The Lord of the Rings’, followed by his work on ‘King Kong’, Andy came back to the UK.” One of the first things Serkis spotted, says Brown, “was that no-one was using digital performance-capture technology to anywhere like the degree that had been used in ‘The Lord of the Rings’. He found a few like-minded people that were interested in building up a studio in the UK, which gave birth to this place.” Brown gestures around a modest green room festooned with posters for ‘The Hobbit’, overlooking the mocap volume. “When we started, our studio here was a concrete floor and black painted walls. Everything you see here today was brought in by the desire to find out what we needed to tell great stories, bring creative ideas to life.” To get where Imaginarium is today, “we took a long look at what hardcore technology was needed so that we can deliver on these creative ideas”.

Brown has been with Imaginarium since 2016 and is keen to point out that while his route to becoming the CEO of what he describes as “an engineering think-tank” might seem unconventional, it obeys its own logic. “As a kid I was interested in music,” but being – in his own words – a very mediocre bass player, he soon realised he’d never make it as a musician. “I switched to photographing bands. But, while I was doing that I did a degree in international relations, which exposed me to more bands. This gave me the opportunity to see what it was like to be signed by a major record label, do the big first album release and all that sort of thing. That deepened my interest in the music business. Then for my master’s degree, I did a course that applied business thinking to the creative industries, which led to me spending a few years helping artists and managers to understand what technology was doing to their ability to make a living. At the time, the entertainment landscape was fundamentally changing due to download technology. Streaming wasn’t even on the horizon.”

Brown says two decades ago the business was in flux: from a technological viewpoint, from an intellectual property point of view, legislatively, as well as the challenge of working out how to monetise product. “The artists we were working with were trying to strike a balance between having a direct line of communication with their fans through email and websites and so on, while maintaining their creative position.”

Brown says the result of these shifting sands was that he ended up in a “weird place between tech and creativity. I set up a trade body to represent the interests of recording artists and worked with musicians such as Pink Floyd’s Nick Mason, Blur’s Dave Rowntree and Ed O’Brien from Radiohead. There were all these amazing musicians that were recognising the way their careers had started didn’t exist anymore.”

Having spent so much time lobbying on IP and legislative issues on behalf of recording artists, “one thing led to another, and I ended up working for Innovate UK, funding R&D projects and early-stage technology. I had to look ahead to what would be the market-driven challenges for businesses three, five, ten years out and try to come up with funding programmes that would lead to really interesting R&D projects that could have a transformative impact many years down the line.” It was a difficult task, says Brown, who went on to fund 300 projects in his five years at Innovate UK.

“That’s when I first met the Imaginarium Studios and got to know some of the team,” leading to Brown joining the organisation as head of business development “for the things that didn’t quite fit the traditional film and TV industry. Things like VR, mixed reality and so on.” This led to a further opportunity, to join forces with Serkis, “to make the studio a more self-contained entity” moving it away from the traditional set-up of a production company to a mocap studio. “Those two parts of the business were restructured, making us very much a standalone production company and a standalone studio, with Andy Serkis as the player with a foot in both camps. I run the day-to-day stuff of the studio, while Andy gives creative direction. He’s also good at opening doors, while being an ambassador for the technology.”

For Brown, the key aspect of mocap’s ventures into new platforms such as mixed reality is starting to see how the technique is unleashing potential that has been in the wings for more than a decade. “It’s an exciting developmental phase for mixed reality now and it is a case of ‘you ain’t seen nothing yet’.

“It’s going to be much more accessible and common by the end of 2019. I think one of the key places we’re going to be operating in is bottling theatrical performances and helping them to have another life, in a different context, after the theatrical run is over. Potentially on physical versions of the stage set at home, where you can see a mixed-reality performance happening on stage right in front of you.”