Chiplets: The First Step To Integrated Silicon Photonics For Faster Interconnects At Lower Cost

Merchant silicon applies Moore’s Law to Ethernet switching with astounding results, well documented by tech luminary Andy Bechtolsheim almost four years ago. Since then, switch chips have doubled their throughput to 25.6 Tbps, powering products with up to 64 400GbE interfaces.

The problem arises when trying to transmit electrical signals off chip because the power required is directly proportional to frequency, which as Bechtolsheim illustrates, doubles every three years.

The power required to switch a signal on a capacitive load like an output pin (such as those used to drive RJ45 ports or optical transceivers) is governed by the formula:

Furthermore, even though smaller process nodes allow reduced voltages, they also require thinner dielectric layers, increasing capacitance according to the canonical formula for parallel plate capacitors:

As Ethernet speeds increase, it becomes impossible to modulate electrical signals fast enough over copper, with the fastest 25GBASE-T and 40GBASE-T standards only feasible over short distances and requiring expensive Cat8, 4-pair shielded cable.

Higher speeds need optical transceivers to transform electrical signals to optical pulses over fiber. However, because SerDes frequencies are increasing much faster than chip voltages are decreasing, it means the power used by 400G optical transceivers is 3-4 times that of 100G modules. Indeed, a presentation at Intel Labs Day illustrated that within a few years, the power required for off-chip I/O will exceed the feasible power budget (TDP) of an entire package.

Integrated silicon photonics is a more elegant and efficient solution to bridge the worlds of switch chips and optical fibers. The process technology to integrate optical elements including light sources (laser diodes, quantum dots), light guides, optical amplifiers, photodetectors and modulators (mux/demux) on a silicon substrate with CMOS circuitry is complex.

Chiplets To The Rescue

Advances in multi-chip packaging (MCP) allow combining several smaller dies into a package. Since (assuming no redundant circuitry) the probability of a die working decreases exponentially with increasing area, carving a monolithic die like a multi-core processor or GPU into multiple chiplets improves the wafer yield and can (assuming the MCP technology is less expensive than silicon fabrication) lower costs. MCP also allows combining devices using different process nodes or fabrication processes onto a single package.

AMD is the most prominent chiplet advocate, incorporating up to 8 CPU dies built with 7nm technology and a 12nm I/O die in its Epyc and Ryzen products. The same approach can be applied to switches using switch silicon with silicon photonics modules in a multi-chip package.

Intel highlighted this approach to silicon photonics with a co-packaged optics demonstration that combined the Barefoot Tofino2 P4 programmable 12.9Tbs switch with 4 photonics engines providing 400G connectivity via 64 56G-PAM4 SerDes interfaces per module. It claims that integrated photonics provide 20 times the bandwidth density at half the cost and power consumption.

Merchant Photonics Chiplet

Intel and AMD use chiplets of their own design for products with heterogeneous components. However, the technology required to design and fabricate silicon photonics modules is sufficiently specialized such that most companies look to outside suppliers. Two large networking companies—Cisco with Luxtera and Acacia and Marvell with InPhi—have acquired photonics expertise. Presumably these vendors plan to integrate photonics into their switch silicon.

Startup Ayar Labs has taken a different approach by developing merchant optical chiplets and multi-wavelength lasers available to any company designing an SoC requiring high throughput.

Indeed, Ayar’s TeraPHY uses Intel’s Advanced Interface Bus (AIB) for die-to-die communication that supports multiple I/O standards and allows rack-scale or infra-data center communication with the speed, latency, and power usage of an in-package interconnect. Each TeraPHY chiplet supports up to 10 ports, each operating at up to 256 Gbps for a total of 2.56 Tbps. Ayar, which received funding from DARPA, currently targets fabrics for HPC and AI clusters. However, the technology could be repurposed for 400G switch interfaces.

A Bright Future Present

Intel and others in the hyperscale-HPC ecosystem have long discussed integrated optics for rack-scale communications. However, the nexus of 400G Ethernet adoption using 56G or faster SerDes, improvements in silicon photonics processes, and 2.5D-3D MCP technology make integrated photonics the only viable solution for future Ethernet switches.

The differences in process technology between CMOS logic chips and photonic devices, which often use hybrid materials like silicon (and SiO₂) indium phosphide (InP), make it impractical to fabricate both on a monolithic chip. Silicon photonic chiplets powering external optical I/Os for a 25Tbps or faster switch chip promise significant reductions in power and cost, while laying the foundation for 800G and faster interfaces.