Discover How CXL, the Enhanced PCI Express, Revolutionizes Data Centers and Workstations

If you have read the specifications of many processors and motherboards for the high-performance market, such as workstations, servers, data centers, and the like, you will often see a technology called CXL or Compute Express Link mentioned alongside PCI Express. In this article, we will explain what it is, what it is used for, and why it is not commonly seen in PCs.

One of the peculiarities of the PC and all derived systems, including some that are not for domestic use, is the fact that their memory or access to it is not unified, which necessitates the creation of new protocols to achieve this function. For a long time, different manufacturers have had to rack their brains to ensure that different components, despite having different memories, will use the same addressing. Although they could do this between devices of the same brand, things became more complicated when several companies were involved.

CXL is designed to achieve this goal through an open protocol that can be used by any manufacturer without fear of incompatibilities. Compute Express Link, or CXL, is a standard that is an expansion of PCI Express. This is because it uses the same data interface but adds a missing element: memory coherence. This means that, although physically the different devices have different physical memory pools, they have access to a common way of communicating with each other.

Here, we must differentiate between addressing and physical location. Addressing is how the CPU, GPU, or any of the key components locate data in memory and know their status, such as if a memory cell has been recently rewritten. This is usually achieved through a caching system where multiple cores share a last-level cache that allows them to have a common view of the content of the RAM.

But what happens when we have a Non-Uniform Memory Access (NUMA) type system? This is usually what we have in PCs, where we have the CPU with the system’s RAM and, on the other hand, the graphics card with its VRAM. At this point, CXL or Compute Express Link comes into play, aiming for all high-performance PC components to communicate via the PCI Express interface, hence using this as a basis for the creation of this open standard.

CXL 1.0 and 2.0 are built on the PCI Express 5.0 interface, while version 3.0 is built on PCI Express 6.0. There is no version of Compute Express Link that works with earlier generations of this high-speed parallel communication interface. However, it should be noted that for the benefits of CXL to be activated, it must be supported not only by the motherboard or the system’s PCI Express controller but also by the connected devices.

The Compute Express Link adds a set of communication standards to achieve memory coherence between different devices, which requires that the hardware be adapted for it. Currently, this adaptation can only be seen in workstations, servers, and data centers, as it is not even found in the most advanced PC motherboards of Intel and AMD.

The main benefit of using this protocol is the ability to turn non-primary RAM that comes after it in the hierarchy into primary memory for certain tasks. This prevents having to search through large-capacity memories known not to contain the data. In other words, it allows us to use the PCI Express port to access memory from a GPU or CPU external to each device.

There are three types of CXL devices:

– Type 1 devices lack local memory of their own but can benefit from using the system’s RAM.
– Type 2 devices are support chips placed as expansion cards with their own RAM. These include graphics cards and FPGA through PCI Express. CXL allows these devices and the CPU to see their respective memories as a whole and share addressing, including a memory coherence system.
– Type 3 devices are RAM that is neither part of the main RAM nor specialized for each component but can be added to the system for intercommunication or to extend memory capacity beyond physical limits.

From CXL 2.0 onwards, components compatible with this standard can use other memories besides their own for work. However, to avoid contention problems with the use of external RAM pools, memory spaces can be assigned to these accelerators, preventing conflicts in memory access between different devices. This allows for the allocation of specific memory spaces to the needs of each device outside its local memory.

CXL is crucial in systems that, due to physical limitations, cannot go beyond their assigned memory. The most obvious case is a graphics card with soldered RAM, but this situation may also arise with a CPU or any other device. The CXL technology aims to improve PCI Express for data centers and workstations, enabling better communication and access between various components and memory types in high-performance systems.