Photonic interconnect is the way forward for faster computers.
Every ten years, computers get a thousand times faster thanks to Moore’s Law scaling and task-sharing over ever larger numbers of CPUs. Even so, while the number of transistors on the average circuit continues to rise, traditional wire data transfer is proving just a bit too slow for the modern processor. With increasing demands for high-speed data transmission, the spotlight is on a shift to photons to carry the data.
When IBM introduced its first petascale Roadrunner supercomputer eight years ago, it used 40,000 optical links to shuttle data around the system. Exascale machines, due in the next couple of years, are expected to use 400 million optical links between processors to achieve the targeted billion billion calculations per second.
Supercomputer designs are by nature extreme but modern data centres are filling up with optical interconnects too. Professor John Bowers, head of the optoelectronics research group at the University of California at Santa Barbara (UCSB), points out: “Typically, there’s a ten-year delay between supercomputers and what we own for personal use, whether it’s laptops or cell phones and so on.”
Bowers specialises in silicon photonics, a technology with the potential to cut the cost of optical interconnection by integrating on silicon all the circuitry of an optical transceiver including, most challengingly, the III-V semiconductor lasers. In contrast, transceivers today are complex assemblies of passive optics, III-V lasers, and silicon chips that typically cost $100 each. “Integration drove down the cost of ICs while improving their performance, and photonic integration will drive down the cost of optical transceivers and increase their capacity by a factor of a hundred,” explains Bowers.
“This is a profound change to the materials the industry works with. But large-scale production of lasers on silicon substrates is a realistic five-year objective,” says Professor Alwyn Seeds, head of photonics at University College London (UCL), whose group is working on integrating lasers into silicon chips.
According to Bowers, silicon photonics’ impact could be similar to Shuji Nakamura and colleagues’ work on blue LEDs which brought us solid-state lighting: “It’s now standard to build LEDs on other substrates to make them cheaper. We want to do the same for optical transceivers,” he says.
The copper problem
“Shoot an oscillating signal down a long copper line on a printed circuit board and it’ll lose about half its strength at 2GHz - and a staggering 98 per cent at 10GHz”. That description comes from Clint Schow, Fuad Doany and Jeffrey Kash, researchers at IBM’s TJ Watson Research Center, in their article ‘Get on the Optical Bus’ about the Roadrunner supercomputer design.
As the authors point out, the lossiness of copper interconnect stems from two effects. “First, the oscillating signal induces stray currents in the board’s conductors that suck away energy. Second, induced currents inside the wire itself push electrons to the surface of the metal, reducing the effective cross-section of the wire and thus raising resistance. The higher the frequency - that is, the clock rate - of the signal, the greater the losses will be.”When bit-rates approach 10Gbit/s, crosstalk blurs the signal, even at distances of less than a metre. When linking together the multiple multi-processing modules of a massively parallel computer - across boards or racks - the copper bandwidth bottleneck becomes particularly severe.
In contrast, as has been demonstrated for the last 30 years by the telecoms industry, light can carry high bandwidth signals efficiently over long distances. Using wavelength division multiplexing (WDM), optical signals can be encoded onto multiple different carrier wavelengths. In this way, many channels can be packed into a single strand of optical fibre enabling bandwidths of terabits per second. WDM makes it easy to upgrade a system using higher bandwidth transceivers while retaining the same fibre.
Bringing light into the data centre
For all these reasons, data centres already run most of their high bandwidth data over optical fibre, using serialiser/deserialiser (SerDes) circuits to pack data from the wide electrical buses used inside processors into a single stream for each link.
“The connection from the server to the ‘top of rack switch’ is still mostly short-reach [5 to 7m] copper direct-attached cables operating at 1Gbit/s or maybe 10Gbit/s. But as you come up and out through various aggregation layers - through leaf and larger spine switches - most of the deployments are 40Gbit/s fibre optic links,” says Adam Carter, chief commercial officer at the optical and laser components firm Oclaro.
“By June or July this year, the first data centres will be rolling out 100Gbit/s in those switches using the Broadcom Tomahawk and Jericho SerDes integrated circuits. If it were cheap enough, optical interconnect could go all the way to the server,” he says.
Carter believes standardisation efforts such as the Microsoft-driven Consortium for OnBoard Optics (COBO) is the next stage for optical interconnect in data centres. COBO is working on a set of standards to permit the development of interchangeable and interoperable optical modules that can be mounted or socketed on a network switch or adapter motherboard. Putting optical connections on the board helps switch-makers break through limits on how many optical ports can fit on a front panel.
A large data centre will have 100,000 or more servers, so the sector is a ready market for high volumes of low-cost silicon photonics transceivers. Operators such as Facebook and Google are quoting a dollar per gigabit per second as a target today for optical interconnect, according to Will Green, manager of the silicon integrated nanophotonics group at IBM’s TJ Watson Research Center. “It is very aggressive and a new paradigm for legacy long-haul telecommunications,” he says.
One way Green’s group is reducing costs is by changing how single-mode fibres are aligned to waveguides in transceivers. Used by the telecoms industry for decades, single-mode fibres are the most efficient optical wires that can guide light for very long distances with low losses. Large data centres have links as long as 2km so there is a move towards standardising on single-mode for both short and long reaches to reduce inventory and future-proof the fibre infrastructure.
Getting light out of silicon
Because silicon cannot efficiently emit light, the big challenge for silicon photonics is how to integrate efficient light-emitting III-V materials such as indium phosphide (InP), indium arsenide (InAs) and indium gallium arsenide (InGaAs) used to make semiconductor lasers. III-V materials have direct bandgap transitions from the conductance band to the valence band, which makes them highly efficient light emitters.
Growing III-V lasers on standard silicon wafers rather than on expensive and much smaller InP or GaAs wafers means overcoming the mismatch between the materials’ crystalline lattices and also accommodating the different thermal expansion coefficients. Such incompatibilities cause strain in the materials, which leads to cracks known as threading dislocations that reduce the efficiency of light emission.
“III-V materials have much larger lattice constants than silicon. If you grow them on silicon, you get around 1010 threading dislocations per cm2,” explains Huiyun Liu, professor of semiconductor photonics at UCL, whose recent work has managed to reduce these dislocations to around 105/cm2.
Bowers’ group at UCSB has approached the problem by creating a way of mechanically bonding small pieces of unprocessed InP to a silicon wafer. The lasers, detectors and optical waveguides are processed afterwards with standard semiconductor lithography and etch steps.
Using this technique, UCSB has produced lasers operating at wavelengths of 1300nm and 1550nm, as well as optical amplifiers, detectors and modulators. Aurrion, a company founded in 2008 by Bowers, is commercialising a version of the technology. Groups at Intel, Hewlett Packard Enterprises and IBM are using a similar approach to create optical transceivers.
Two years ago Aurrion demonstrated with IBM a 30Gbit/s, 10km single-mode optical link based on its photonic components. The optical transmitter and receiver assemblies consumed 75mW, significantly less than the watts of power consumed by today’s 10km-reach optics. Last year, IBM announced a fully integrated wavelength multiplexed photonics chip, which will soon enable production of 100Gbit/s optical transceivers.
For epitaxial integration, quantum-dot lasers look promising. Quantum dots are semiconductor nanocrystals with diameters in the range of 2nm to 10nm. They can emit light at different wavelengths, dependent on the size of the dots.
Most recently Liu and Seeds at UCL in conjunction with the London Centre for Nanotechnology, Cardiff University and the University of Sheffield demonstrated what they describe as the first practical electrically driven 1300nm wavelength InAs/GaAs quantum-dot lasers grown directly on a silicon wafer. With more than 3100 hours of continuous wave operating time collected, the researchers predict that the mean time to failure of the devices will be more than 100,000 hours. The lasers achieved continuous wave lasing up to 75°C with a low threshold current of 62.5A/cm2.
“Quantum dots have many advantages when it comes to making semiconductor lasers. Because they are localised in space, a dislocation is only likely to affect a few of the dots and the remaining ones keep contributing to optical gain,” explains Liu. “Quantum dots also create a strong mechanical-strain field, which can bend dislocations away from the dots.”
Alwyn Seeds says the latest quantum dot lasers on silicon are almost identical in performance to those grown directly on GaAs wafers. “We have already demonstrated powers as large as one tenth of a watt from a single laser, which is more power than you would need in optical-fibre transmission,” he says. Seeds and Liu have patented the development and UCL and the group is working with a number of companies on potential commercial development.
Optical interconnect evolution
As optical transceiver costs come down and bandwidth/performance demands increase, optical connections at ever shorter ranges will make increasing sense. “As volumes increase, I think you will see more of the silicon packaging houses getting involved and that’s where it could get interesting,” says Oclaro’s Carter. “A lot of investment needs to come and the ecosystems need to be developed.”
How far down the interconnect hierarchy can optical communication go? Power is the limiting factor for any processor, points out Seeds. “If you can’t move data optically on-chip using less energy than copper, it does not make sense. Chip area is another factor. Lasers are small but most of them are still larger than a transistor. If you use a lot of area for on-chip optical interconnects the chip becomes more expensive.”
Recent news that researchers at the US National Institute of Standards and Technology (NIST) have developed a GaAs-based ‘piezo-optomechanical’ circuit that translates between optical, acoustic and radio signals suggests that adding light to our information processing systems is only the first step to a multilingual future.