Ayar Labs Realizes Co-Packaged Silicon Photonics

Optical I/F

Sitting between the AIB interface and the optical interface is configurable crossbar glue logic that maps the AIB channels to optical channels. The crossbar allows one-to-many connections. A single electrical channel may be sent over multiple optical interfaces and vice versa. The TeraPHY chiplet integrates ten photonics macros pairs, one macro for transmit and one for receive.

Lights within the chip travel through waveguides. Thanks to the property of light, multiple wavelengths of light can travel along the same waveguide without interfering with each other. A technique wave-division multiplexing (WDM) is used to introduce multiple such wavelengths onto waveguide in order to increase the amount of data that can travel across the same fiber link. In order to make it happen, Ayar uses multiple micro-ring resonators on the same waveguide to convert data from and to optical or electrical using the different wavelengths from the waveguide. The individual low-power silicon photonic ring-resonators lock onto a specific wavelength on which they operate. Those ring resonators are driven by CMOS drivers which interface with the reset of the digital logic on the die.

Within each macro pair is a set of PLLs, TRX slices, and other logic required to make it all work. If you look at the GDSII screenshot you can make out eight TRX slices, one for each wavelength. The various PLLs in each macro is designed such that the data rate could be configurable up to 2x delta. The current TeraPHY chiplet allows for data rates 16 Gbps, 25.6 Gbps and up to 32 Gbps. Since there are eight wavelengths per waveguide you are looking at a configurable aggregated bandwidth of 128 Gbps to 256 Gbps per macro.

The current TeraPHY chiplet incorporates 10 macro pairs. This means it’s capable of up to 2.56 Tb/s aggregated bandwidth across all the optical macros. This is a fair bit more than what’s possible across the AIB link. It’s unclear why they are so unbalanced but since a single AIB channel can be routed to more than one optical channel, there could potentially be topologies where doing such communication is advantageous. For example one SoC routing traffic to two other SoCs. It’s worth adding that it uses the NRZ modulation format on the optical channels since error correction is unnecessary on the receive side.

For their Intel partnership, two TeraPHY are integrated into a Stratix 10 FPGA. This means each of those FPGAs have a combined optical bandwidth of 5.12 Tbps. Pretty impressive for two tiny sub-50 mm² chiplets!

So how do you physically get it out of the chip? The input or output waveguides end at a grating coupler, a device that has ridges and grooves on its top surface, allowing light to scatter out of the waveguide at some specific angle. Here, an optical fiber is brought just close enough that it’s able to collect the scattered light.

For the Intel Stratix 10 FPGA, the light escapes from the top. In other words, the TeraPHY assembly involves alignment and bonding of fiber through the die backside. An optical fiber connector is brought through an opening from the top of the lid directly down onto the TeraPHY chiplet.

“We’re seeing an explosion of Datacenter workloads that have an insatiable demand for bandwidth and the need to connect devices at rack-scale distances,” said Vince Hu, VP of Strategy and Innovation for Intel’s FPGA products. “The best way to do that is with optical interconnect and by using an Ayar Labs chiplet(s), we can achieve very high bandwidth at low latency and low power consumption.”

Since AIB has extremely low latency on the order of just 3 ns, there is under 10 ns of latency for a round-trip communication through the AIB to through the TeraPHY and back through the AIB and there is roughly 5 ns per meter of latency over the optical fibers (depending on the overall system configuration) with a reach of up to 2 kilometers. The total TeraPHY energy efficiency is just under 5 pJ/bit. This number encompasses the AIB interface, crossbar, and optical macros.

SuperNova Laser

One thing worth pointing out is the operating temperature of this chip. With the FPGA reaching as much as 300 Watt and the TeraPHY adding another 4.7 W, the thermals of this chip can be quite high – Ayar reporting numbers potentially in the upwards of 90 degrees Celsius. GlobalFoundries 45 RF SOI is already qualified for Grade 1 (-40ºC to +125ºC) and for Ayar’s own TeraPHY chiplet, the design has been made to tolerate those temperatures through extra control tuning logic capable of tracking and managing the thermals variations. Unfortunately, for a light source (a laser), it may suffer from these temperatures which might affect the reliability of the device. For this reason, the current TeraPHY architecture uses an external laser source that sits somewhere else in the system and provides the TeraPHY chiplets with the continuous wave that’s required.

Ayar Labs also developed an external laser called SuperNova. SuperNova currently supports eight and sixteen wavelengths. The wavelengths are multiplexed and amplified onto eight or sixteen output ports. In other words, Ayar’s current SuperNova laser supports a total of 256 channels for a total bandwidth of 8.192 Tbps. The Laser has a power efficiency of 1-2 pJ/bit.

Potential Roadmap Improvements

Ayar Labs TeraPHY chiplet shows the potential of integrated photonics. For a first of its kind, the numbers are very impressive. The first-generation AIB interface on the Stratix 10 has an energy efficiency of roughly 0.85 pJ/bit on the 55-micron micro-bump pitch on first-gen EMIB. Intel has talked a lot recently about its plan to reduce this number by as much as 0.5 pJ/bit or more. An improvement in energy efficiency coupled with higher density micro-bumps should allow for considerably higher bandwidth at similar power to the current prototype. Additionally, Ayar’s current TeraPHY chiplet takes advantage of WDM in order to send eight bits on eight wavelengths across the same waveguide. Even if we assume that they won’t be adding any more channels, they can actually double the number of wavelengths on the TeraPHY chiplet in order to double the optical I/O bandwidth. Ayar’s SuperNova laser already supports sixteen wavelengths. This change will increase the areal bandwidth density but the shoreline bandwidth density remains unchanged. Beyond this, lasers with up to 32 wavelengths already exist in a limited capacity and 64 are in deep research.

At Supercomputing 2019 Ayar Lab announced its Customer Sampling Program with select OEMs and end-users starting in early Q1 2020.