A team of Doctor Who devotees has for decades been advancing digital restoration techniques while recovering vintage episodes from some often obscure sources.
Although BBC television managed to keep an untransmitted version of the first episode of Doctor Who recorded 50 years ago, the organisation wiped more than a hundred others made up to the mid-1970s. A series of appeals, including one launched this year to coincide with the show's 50th anniversary in November, have turned up videotapes and film recordings from around the world. But just over 100 episodes still remain 'missing believed wiped'.
The reasons for the lack of tape retention are not necessarily down to carelessness or wilful vandalism on the programme makers' part. Accurate figures are hard to ascertain, but when Doctor Who began in 1963 each hour of Quadruplex 2in videotape cost £100-£120 new – which, adjusted for inflation, is equivalent to more than £1,000 in 2013 terms. Used videotapes also took up precious space in what passed for the BBC's archives in the mid-1960s. There was at the time little prospect of them being used ever again, and the series started during an era when TV was generally considered to be a largely disposable medium. Many 405-line recordings were junked following the advent of UK colour TV broadcasting from 1969.
Repeats were possible, but deals with performers' trade unions placed a time limit on how long after initial transmission they could be used even if such a deal were in place – once that period expired, there was no chance of retransmission without new agreements. It should also be remembered that the first domestic video recorders would not appear until the late 1970s, and there was no market for home video for close to the first two decades of the show's history.
After the end of Jon Pertwee's tenure as Doctor number three in 1974, the BBC took better care of its recordings and other televisual elements. However, it was not until 1983 that the BBC's then retail subsidiary was ready to sell VHS videos of old serials as retail items.
Hunt for the Tomb of the Cybermen
As soon as the decision was taken, the problem of satisfying demand surfaced: a fan convention voted that the serial 'Tomb of the Cybermen' (1967), with second Doctor Patrick Troughton, should be the first to be put out; but it was missing from the archives. Instead, 'Revenge of the Cybermen' (1975) with fourth Doctor Tom Baker was issued.
This substitution fired the starting pistol on a race that would become very familiar to afficionados of heritage television: uncovering any possible source of missing material around the globe. In 1991 telerecordings of all 'Tomb of the Cybermen' episodes were finally recovered from the Rediffusion Television in Hong Kong. Although the original 405-line standard videotapes had long been wiped for reuse, programmes were commonly rerecorded onto 16mm black-and-white film stock for overseas sale (where a foreign broadcaster might have a telecine equipment for transmitting film, but not pricey video apparatus to record and play programmes on tape). Telerecording – transferring video content to film – is the media by which most surviving TV from the 1950s and 1960s that was originated on video has been preserved.
Survival-by-telerecording was even the case for the serials produced after Jon Pertwee took over as the Doctor in 1970, and the PAL colour system was standard. Colour television was still a rarity internationally, and the BBC worked on the basis that syndication overseas was most likely to be to stations that broadcast in black and white. The BBC did not take much better care of these film reels – a number of those that were not sold overseas were apparently trashed to recover office space. Many of the recovered film reels have been found in the archives of overseas stations or found their way into the possession of TV staff who kept them when the broadcasters asked for them to be discarded. Legend has it that other programmes were salvaged from rubbish.
Bringing colour back to the 16mm film versions of Pertwee-era recordings is where much of the recent ingenuity in restoring Doctor Who has been focused. No other TV series has been able to call upon a restoration effort in the way that Doctor Who has. The series' sometimes fanatical followers have delivered a series of solutions to the problem of how to resurrect TV shows when much of the source material was deliberately recorded over or trashed.
The network of fans has deployed increasingly complex techniques to put the colour back into Jon Pertwee's cheeks and even recreate scenes excised by the original production team. In doing so, the work has revealed a lot about the processes used in the 1960s and 1970s to convert from video to film and back again – and how sticky analogue data can be despite best efforts to destroy it.
A bit of technical knowhow
Only one serial was preserved in the form of colour film – the consequences of a studio-based strike during production. Because the four episodes of 'Spearhead from Space' (1970) could be filmed as outside broadcasts – and the medium of choice for this was film – the director chose to shoot the entire story on 16mm colour film. This has allowed a release on high-definition Blu-Ray – it may be the only adventure of the first 25 years of the series able to receive this treatment, because it is comparatively straightforward to rescan all-film elements to the HD format.
When it came to preparing a VHS release of 'The Daemons' (1971) in the early 1990s, the only broadcast-quality source available for all but one of the episodes was a set of monochrome 16mm film negatives.
However, according to Steve Roberts, who was a senior post production engineer for BBC Post Production & Graphic Design, two fans of the series convinced the broadcaster to fund a novel way to 'recolour' the serial. Ralph Montagu, a graphic designer at the BBC, and James Russell, an equipment designer at Rank Cintel (which produced film-to-video converters), were aware that another fan had arranged for a colour transmission in Los Angeles in 1978 – seven years after it first broadcast in the UK – to be recorded off air on a U-matic format video recorder. Admittedly, the transmission used the lower-resolution NTSC format standard to the US, but at least it was in colour.
"I'd contacted the BBC's archive selector in 1992 to ask if he would allow me access to the black-and-white 16mm film prints and off-air NTSC colour copy of the story in order to see if I could combine them into a broadcastable colour version," says Steve Roberts. The archivist put Roberts in touch with Montagu and Russell. "After viewing their test, I could see lots of ways to improve the process, so we collaborated on a full colour restoration of 'The Daemons' for BBC2 transmission, and two more stories for VHS release."
Montagu and Russell teamed up with Steve Roberts to transfer the colour signal from the NTSC recordings and combine that with the brightness information from the higher-resolution film version. This process takes advantage of the eye's lower sensitivity to colour detail, although it involved a number of digital transformations to marry the subtle distortions that resulted from the transfer to NTSC. Some of the recolouring had to be done by hand using specialised video-effects equipment such as the Quantel Paintbox computer graphics workstation. This kind of work would move to the PC with software such as Adobe's Photoshop in later years. The work on 'The Daemons' led to the creation of a more formal network of fans dedicated to a programme of restoring Doctor Who episodes: the Restoration Team.
"The name came about when I needed something to make us sound official in a letter I wrote to TV Zone magazine – but it seemed to stick," Roberts explains. "The 'Doctor Who Remastering Team' would be more accurate."
It has worked to recover and remaster the bulk of the serials released since the mid-1990s, some of them from the videotapes, but many of them sourced from film versions of the originals. Using film as a replacement for video presented a more subtle problem – the overall look of the production was different to the way in which TV viewers saw it on their sets in the 1960s and 1970s.
VidFire and motion estimation
Series fan Peter Finklestone developed a software system called VidFire (Video Field Interpolation Restoration Effect) that processed the recording from the film to provide the 'look' of video as it would have been seen by viewers of the original broadcasts.
Film displays each image as a single contiguous frame, running in the case of UK recordings at a speed of 25 frames per second. The standard UK video's frame rate was the same but split the image into two fields – drawing alternating horizontal lines on each field update which takes place 50 times per second. The higher update rate for fields meant that motion has a less jerky quality than with film although it results in a comb-like effect for fast-moving objects.
"The original VidFire process used motion estimation to create a new frame midway in time between pairs of existing frames on the film recording, then this new frame was interlaced with one of the film frames to create an interlaced video frame, returning the 50 picture per second live video feel. So, if the original frames were 1, 2, 3, etc., VidFire created new frames at 1.5, 2.5, 3.5, etc. Over the years this process has been developed and updated as more powerful motion estimation systems become available," Roberts explains.
"One problem with the original process was that the motion-estimated frames were softer than the original film frames, meaning one field was sharper and grainer than the other. These days, both fields are created from motion estimated frames at 1.25, 1.75, 2.25, 2.75, etc., to avoid this problem. It's worth pointing out that an enormous amount of manual retouching and clean-up has to be done after the automated processing."
The VidFire system has also been used in the restoration of archive programmes other than Doctor Who. These include the 1950s BBC production of Nigel Kneale's landmark sci-fi serial 'Quatermass and the Pit' and 1960s episodes of comedy series such as 'Sykes' and 'Dad's Army'.
Hidden colour
The black-and-white telerecordings turned out to hold a lot more recoverable information than expected. BBC Research engineer James Insell says he first noticed colour information might be preserved on the supposedly monochrome-only film recordings when he saw patches of saturated red appearing in a rerun of the Pertwee serial 'The Ambassadors of Death' (1970) transmitted on UK Gold in 1994.
Despite being recorded on monochrome media the colour information had not been stripped out. Insell contacted Roberts who confirmed that recording video to film would not totally filter out the colour signal. Black-and-white film captures colour information from a video signal because of the way that the colour signal is mixed into the video fields. Patented by Georges Valensi in 1938, the video signal is the combination of two high-frequency subcarrier waves – one of them delayed in phase by 90° – that use changes in amplitude to carry information. Black-and-white sets would decode the luminance or brightness subcarrier but ignore the second 'chrominance' signal. Colour sets would decode the chrominance to display a colour image.
The problem with dealing with the merged carriers is that it does not carry a reliable clock signal. A simple, single-carrier signal has an implicit clock signal because the modulated wave crosses the zero point each half cycle. By overlaying a second, phase-shifted subcarrier, this zero-crossing becomes far less frequent. The US NTSC system attempted to solve the problem by transmitting synchronising bursts at the start of each line but the TV decoder still ran the risk of drifting away from the implied clock which had the effect of making the colours themselves wander away from what they were meant to be. It did not take long for NTSC to come to stand for 'never twice the same colour'. PAL attempted to deal with this problem by inverting the phase of the colour signal and retransmitting it every other line, helping the TV receiver to recover from phase errors by cancelling them out.
Although the effective colour resolution is lower, the eye is less sensitive to colour detail than it is to luminance.Without filtering, film captures both signals. The film used was typically very slow, providing it with a very fine grain able to capture minute details. If not filtered out, the colour information appears as a fine pattern of dots or stripes overlaying the image – an effect similar to that of newsprint photographs. Viewers would not normally see this as filtering during rescanning to video before transmission would remove them.
Unfortunately, because of the differences in the way that film and video work the film recording process is not directly reversible – simply playing the film as-is to a colour decoder does not result in viable colour video. The result is generally flashes of spurious colour. The film image is often distorted, which alters the position of the colour dots, which in turns shifts the hue of the decoded colours.
Former BBC engineer Richard Russell developed software that could filter the colour information and correct distortions. It remaines a manually intensive process as the subtle changes in the chroma pattern can throw off the automated decoding.
"The quality of the colour-restored output is dependent on many factors, some of which still elude us," explains Roberts. "An episode recorded one week might decode beautifully, while one that looks similar on the surface recorded the next week might decode terribly."
Richard Russell adds: "I have tried one or two experimental modifications but with little success, so my suspicion remains, as it has always been, that there's not much room available for improvement."
However, the process is faster than hand painting, which the BBC has used on some serials where there is no recoverable colour information. Even hand colouring can be semi-automated. The BBC engaged US-based Legend Films, which has recoloured a number of films, to work on a 90-second fragment of 'The Silurians' (1970) in 2006. Much of the rest of the serial's colour could be recovered from off-air recordings using a process similar to that employed on The Daemons; but this segment was only available in black and white. Hand marking is used for frames in which an operator chooses tints and identifies objects as moving or stationary. Software then attempts to track objects as they move from frame to frame so that it can apply the right tint.
With the release of episodes in the past few years that employ these recolouring techniques, the Doctor Who Restoration Team has more or less completed the task of bringing the colour Doctor Who serials back to a state very close to the original viewing experience; there may even be opportunities to improve on the original remasters.
"Potentially there are sections of releases where we had previously used the colour from off-air domestic recordings which would benefit from using colour recovery instead," Roberts says, but the amount of manual effort needed by the process may rule out further processing on cost grounds.
Further information
Sign up to the E&T News e-mail to get great stories like this delivered to your inbox every day.
