A Look at Cavium’s New High-Performance ARM Microprocessors and the Isambard Supercomputer

Earlier last month Cavium announced that ThunderX2 has hit general availability, a new family of Xeon-class ARM server processors optimized for HPC workloads. Contrary to its namesake and despite succeeding the original ThunderX line, the new processors are of an entirely different design, unlike anything Cavium has introduced in the past.

Overview

The new ThunderX2 family includes over 40 SKUs for scale-up and scale-out applications. All models are based on the same design. The die incorporates 32 custom ARM cores (detailed later) along with eight DDR4 memory channels, two USB ports, two SATA ports, x56 lanes of PCIe Gen 3, and 2nd-generation CCPI (also detailed later).

Chip Overview

With eight memory channels, each supporting DDR4 at rates of up to 2666 MT/s for both 2 DPC and 1 DPC, those processors have an aggregated bandwidth of up to 170.7 GB/s – same as AMD’s EPYC. RDIMM, LRDIMM, and NVDIMM-P in single, dual and quad rank modules are supported. Cavium has many of the enhanced RAS features one should expect from a modern server chip including failed DIMM identification, demand and patrol scrubbing, address parity protection, corrupt data containment (data poisoning), SECDED ECC, and SDDC.

As far as I/O go, ThunderX2 incorporates 56 lanes of PCIe Gen 3. That’s x8 lanes more than Skylake SP.

I/O Capabilities (Cavium)

With 14 PCIe controllers, those lanes can be bifurcated as 3×16, 7×8, and various other combinations. Other I/O include 2 SATA3 ports, 2 USB 3 ports, and general-purpose I/O.

Models

Cavium is launching over forty SKUs which have been sorted out by how they compare to Intel’s Skylake offerings. In paticular, the models were compared against Xeon Gold and Platinum based on their SPECIntRate and TDP scores (tested using gcc –o3). To simplify things, we can actually compress the stack down to just six models:

ThunderX2 SKU Stack
Model Cores / Threads Frequency PCIe Lanes Memory
CN9980 32 / 128 2.0 – 2.5 GHz x56 8 × DDR4-2666
CN9978 30 / 120 1.8 – 2.3 GHz x56 4/8 × DDR4-2666
CN9975 28 / 112 1.8 – 2.4 GHz x56 6/8 × DDR4-2666
CN9970 24 / 96 1.8 – 2.5 GHz x56 4/6/8 × DDR4-2666
CN9965 20 / 80 1.8 – 2.5 GHz x56 4/6 × DDR4-2666
CN9960 16 / 64 1.6 – 2.2 GHz x48 4 × DDR4-2666

For the most part, the stack segmentation is straightforward – cores, frequency, and memory channels. SKUs range from $800-$3000. The top-binned CN9980 part is a 32-core chip running at 2.5, topping out at 200 Watts. The 2.2 GHz variant of the CN9980 with a TDP of 180 W is priced at $1,795 in quantities of a 1,000 units. At the other end of the scale is the 16-core CN9960 part operating at 1.6 GHz with a TDP of 75 W which costs $800.

A few things are notably missing from the table. The technical specs Cavium has given us mentions that the top binned part operates at 2.5 GHz with a turbo mode of up to 3 GHz. Turbo modes for all other parts supposedly exist. Unfortunately, we were unable to obtain this information. For now, Cavium is not sharing the full TDP specs for the other models either. The full SKU stack disclosed is shown below.

ThunderX2 SKU Stack (Cavium)

The table above, provided by Cavium, has what they consider a direct mapping between their SKUs and Intel’s Skylake. We have expanded on their table to show the full spec from both models for a better comparison.

ThunderX2 SKU Stack
Cavium Intel
Model Price Cores Clock PCIe Ch Model Price Cores Clock
CN9980-2500LG4077-Y21-G $3,000 32 2.5 56 8 Platinum 8180 $10,009 28 2.5/2.1//1.7
CN9980-2400LG4077-Y21-G 32 2.4 56 8 Platinum 8176 $8,719 28 2.1/1.7/1.3
CN9980-2300LG4077-Y21-G 32 2.3 56 8 Platinum 8170 $7,405 26 2.1/1.7/1.3
CN9980-2200LG4077-Y21-G $1,795 32 2.2 56 8 Platinum 8164 $6,114 26 2.0/1.6/1.2
CN9980-2100LG4077-Y21-G 32 2.1 56 8 Platinum 8164 $6,114 26 2.0/1.6/1.2
CN9980-2000LG4077-Y21-G 32 2.0 56 8 Platinum 8160 $4,702 24 2.1/1.8/1.4
CN9978-2300LG4077-Y21-G 30 2.3 56 8 Platinum 8160 $4,702 24 2.1/1.8/1.4
CN9978-2200LG4077-Y21-G 30 2.2 56 8 Gold 6150 $3,358 18 2.7/2.3/1.9
CN9978-2100LG4077-Y21-G 30 2.1 56 8 Gold 6140 $2,445 18 2.3/1.9/1.5
CN9978-2000LG4077-Y21-G 30 2.0 56 8 Gold 6138 $2,612 20 2.0/1.6/1.3
CN9978-1800LG4077-Y21-G 30 1.8 56 8 Gold 6132 $2,111 14 2.6/2.2/1.7
CN9978-1800LG4077-Y21-G 30 1.8 56 4 Gold 6132 $2,111 14 2.6/2.2/1.7
CN9975-2400LG4077-Y21-G 28 2.4 56 8 Gold 6140 $2,445 18 2.3/1.9/1.5
CN9975-2300LG4077-Y21-G 28 2.3 56 8 Gold 6142 $2,946 16 2.6/2.2/1.6
CN9975-2200LG4077-Y21-G 28 2.2 56 8 Gold 6142 $2,946 16 2.6/2.2/1.6
CN9975-2100LG4077-Y21-G 28 2.1 56 8 Gold 6138 $2,612 20 2.0/1.6/1.3
CN9975-2000LG4077-Y21-G 28 2.0 56 8 Gold 6132 $2,111 14 2.6/2.2/1.7
CN9975-1800LG4077-Y21-G 28 1.8 56 8 Gold 6130 $1,894 16 2.1/1.6/1.3
CN9975-1800LG4077-Y21-G 28 1.8 56 6 Gold 6130 $1,894 16 2.1/1.6/1.3
CN9970-2500LG4077-Y21-G 24 2.5 56 8 Gold 6138 $2,612 20 2.0/1.6/1.3
CN9970-2400LG4077-Y21-G 24 2.4 56 8 Gold 6132 $2,111 14 2.6/2.2/1.7
CN9970-2300LG4077-Y21-G 24 2.3 56 8 Gold 6132 $2,111 14 2.6/2.2/1.7
CN9970-2200LG4077-Y21-G 24 2.2 56 8 Gold 6136 $2,460 12 3.0/2.6/2.1
CN9970-2200LG4077-Y21-G 24 2.2 56 6 Gold 6136 $2,460 12 3.0/2.6/2.1
CN9970-2100LG4077-Y21-G 24 2.1 56 8 Gold 6132 $2,111 14 2.6/2.2/1.7
CN9970-2000LG4077-Y21-G 24 2.0 56 8 Gold 6130 $1,894 16 2.1/1.6/1.3
CN9970-1800LG4077-Y21-G 24 1.8 56 8 Gold 6126 $1,776 12 2.6/2.2/1.7
CN9970-1800LG4077-Y21-G 24 1.8 56 4 Gold 6126 $1,776 12 2.6/2.2/1.7
CN9965-2500LG4077-Y21-G 20 2.5 56 6 Gold 6130 $1,894 16 2.1/1.6/1.3
CN9965-2400LG4077-Y21-G 20 2.4 56 6 Gold 6126 $1,776 12 2.6/2.2/1.7
CN9965-2300LG4077-Y21-G 20 2.3 56 6 Gold 5120 $1,555 14 2.2/1.8/1.2
CN9965-2200LG4077-Y21-G 20 2.2 56 6 Gold 5120 $1,555 14 2.2/1.8/1.2
CN9965-2100LG4077-Y21-G 20 2.1 56 6 Gold 5118 $1,273 12 2.3/1.9/1.2
CN9965-2000LG4077-Y21-G 20 2.0 56 6 Gold 5118 $1,273 12 2.3/1.9/1.2
CN9965-1800LG4077-Y21-G 20 1.8 56 6 Silver 4116 $1,002 12 2.1/1.7/1.1
CN9965-1800LG4077-Y21-G 20 1.8 56 4 Silver 4116 $1,002 12 2.1/1.7/1.1
CN9960-2200LG4077-Y21-G 16 2.2 48 4 Silver 4116 $1,002 12 2.1/1.7/1.1
CN9960-2000LG4077-Y21-G 16 2.0 48 4 Silver 4114 $694 10 2.2/1.8/1.1
CN9960-1800LG4077-Y21-G 16 1.8 48 4 Silver 4110 $501 8 2.1/1.7/1.0
CN9960-1600LG4077-Y21-G $800 16 1.6 48 4 Silver 4108 $417 8 1.8/1.4/0.9

Note that we have only listed the base clock frequencies for Intel (non-AVX, AVX2, and AVX-512). Since Cavium did not provide turbo frequencies and TDPs, a direct comparison is quite difficult. It’s also worth pointing out that Intel models have three turbo frequencies, making the comparison highly workload-specific. Regardless of the model, all Intel models in the table have six memory channels, x48 PCIe lanes, and all features enabled. It’s also worth noting that the Xeon Gold 6000 series, as well as all Xeon Platinum models, come with a second AVX-512 FMA unit.

Spotted an error? Help us fix it! Simply select the problematic text and press Ctrl+Enter to notify us.

Leave a Reply

avatar
  Subscribe  
Notify of