A Look at Cavium’s New High-Performance ARM Microprocessors and the Isambard Supercomputer

Earlier last month Cavium announced that ThunderX2 has hit general availability, a new family of Xeon-class ARM server processors optimized for HPC workloads. Contrary to its namesake and despite succeeding the original ThunderX line, the new processors are of an entirely different design, unlike anything Cavium has introduced in the past.

Overview

The new ThunderX2 family includes over 40 SKUs for scale-up and scale-out applications. All models are based on the same design. The die incorporates 32 custom ARM cores (detailed later) along with eight DDR4 memory channels, two USB ports, two SATA ports, x56 lanes of PCIe Gen 3, and 2nd-generation CCPI (also detailed later).

With eight memory channels, each supporting DDR4 at rates of up to 2666 MT/s for both 2 DPC and 1 DPC, those processors have an aggregated bandwidth of up to 170.7 GB/s – same as AMD’s EPYC. RDIMM, LRDIMM, and NVDIMM-P in single, dual and quad rank modules are supported. Cavium has many of the enhanced RAS features one should expect from a modern server chip including failed DIMM identification, demand and patrol scrubbing, address parity protection, corrupt data containment (data poisoning), SECDED ECC, and SDDC.

As far as I/O go, ThunderX2 incorporates 56 lanes of PCIe Gen 3. That’s x8 lanes more than Skylake SP.

With 14 PCIe controllers, those lanes can be bifurcated as 3×16, 7×8, and various other combinations. Other I/O include 2 SATA3 ports, 2 USB 3 ports, and general-purpose I/O.

Models

Cavium is launching over forty SKUs which have been sorted out by how they compare to Intel’s Skylake offerings. In paticular, the models were compared against Xeon Gold and Platinum based on their SPECIntRate and TDP scores (tested using gcc –o3). To simplify things, we can actually compress the stack down to just six models:

ThunderX2 SKU Stack
Model	Cores / Threads	Frequency	PCIe Lanes	Memory
CN9980	32 / 128	2.0 – 2.5 GHz	x56	8 × DDR4-2666
CN9978	30 / 120	1.8 – 2.3 GHz	x56	4/8 × DDR4-2666
CN9975	28 / 112	1.8 – 2.4 GHz	x56	6/8 × DDR4-2666
CN9970	24 / 96	1.8 – 2.5 GHz	x56	4/6/8 × DDR4-2666
CN9965	20 / 80	1.8 – 2.5 GHz	x56	4/6 × DDR4-2666
CN9960	16 / 64	1.6 – 2.2 GHz	x48	4 × DDR4-2666

For the most part, the stack segmentation is straightforward – cores, frequency, and memory channels. SKUs range from $800-$3000. The top-binned CN9980 part is a 32-core chip running at 2.5, topping out at 200 Watts. The 2.2 GHz variant of the CN9980 with a TDP of 180 W is priced at $1,795 in quantities of a 1,000 units. At the other end of the scale is the 16-core CN9960 part operating at 1.6 GHz with a TDP of 75 W which costs $800.

A few things are notably missing from the table. The technical specs Cavium has given us mentions that the top binned part operates at 2.5 GHz with a turbo mode of up to 3 GHz. Turbo modes for all other parts supposedly exist. Unfortunately, we were unable to obtain this information. For now, Cavium is not sharing the full TDP specs for the other models either. The full SKU stack disclosed is shown below.

The table above, provided by Cavium, has what they consider a direct mapping between their SKUs and Intel’s Skylake. We have expanded on their table to show the full spec from both models for a better comparison.

ThunderX2 SKU Stack
Cavium						Intel
Model	Price	Cores	Clock	PCIe	Ch	Model	Price	Cores	Clock
CN9980-2500LG4077-Y21-G	$3,000	32	2.5	56	8	Platinum 8180	$10,009	28	2.5/2.1//1.7
CN9980-2400LG4077-Y21-G		32	2.4	56	8	Platinum 8176	$8,719	28	2.1/1.7/1.3
CN9980-2300LG4077-Y21-G		32	2.3	56	8	Platinum 8170	$7,405	26	2.1/1.7/1.3
CN9980-2200LG4077-Y21-G	$1,795	32	2.2	56	8	Platinum 8164	$6,114	26	2.0/1.6/1.2
CN9980-2100LG4077-Y21-G		32	2.1	56	8	Platinum 8164	$6,114	26	2.0/1.6/1.2
CN9980-2000LG4077-Y21-G		32	2.0	56	8	Platinum 8160	$4,702	24	2.1/1.8/1.4
CN9978-2300LG4077-Y21-G		30	2.3	56	8	Platinum 8160	$4,702	24	2.1/1.8/1.4
CN9978-2200LG4077-Y21-G		30	2.2	56	8	Gold 6150	$3,358	18	2.7/2.3/1.9
CN9978-2100LG4077-Y21-G		30	2.1	56	8	Gold 6140	$2,445	18	2.3/1.9/1.5
CN9978-2000LG4077-Y21-G		30	2.0	56	8	Gold 6138	$2,612	20	2.0/1.6/1.3
CN9978-1800LG4077-Y21-G		30	1.8	56	8	Gold 6132	$2,111	14	2.6/2.2/1.7
CN9978-1800LG4077-Y21-G		30	1.8	56	4	Gold 6132	$2,111	14	2.6/2.2/1.7
CN9975-2400LG4077-Y21-G		28	2.4	56	8	Gold 6140	$2,445	18	2.3/1.9/1.5
CN9975-2300LG4077-Y21-G		28	2.3	56	8	Gold 6142	$2,946	16	2.6/2.2/1.6
CN9975-2200LG4077-Y21-G		28	2.2	56	8	Gold 6142	$2,946	16	2.6/2.2/1.6
CN9975-2100LG4077-Y21-G		28	2.1	56	8	Gold 6138	$2,612	20	2.0/1.6/1.3
CN9975-2000LG4077-Y21-G		28	2.0	56	8	Gold 6132	$2,111	14	2.6/2.2/1.7
CN9975-1800LG4077-Y21-G		28	1.8	56	8	Gold 6130	$1,894	16	2.1/1.6/1.3
CN9975-1800LG4077-Y21-G		28	1.8	56	6	Gold 6130	$1,894	16	2.1/1.6/1.3
CN9970-2500LG4077-Y21-G		24	2.5	56	8	Gold 6138	$2,612	20	2.0/1.6/1.3
CN9970-2400LG4077-Y21-G		24	2.4	56	8	Gold 6132	$2,111	14	2.6/2.2/1.7
CN9970-2300LG4077-Y21-G		24	2.3	56	8	Gold 6132	$2,111	14	2.6/2.2/1.7
CN9970-2200LG4077-Y21-G		24	2.2	56	8	Gold 6136	$2,460	12	3.0/2.6/2.1
CN9970-2200LG4077-Y21-G		24	2.2	56	6	Gold 6136	$2,460	12	3.0/2.6/2.1
CN9970-2100LG4077-Y21-G		24	2.1	56	8	Gold 6132	$2,111	14	2.6/2.2/1.7
CN9970-2000LG4077-Y21-G		24	2.0	56	8	Gold 6130	$1,894	16	2.1/1.6/1.3
CN9970-1800LG4077-Y21-G		24	1.8	56	8	Gold 6126	$1,776	12	2.6/2.2/1.7
CN9970-1800LG4077-Y21-G		24	1.8	56	4	Gold 6126	$1,776	12	2.6/2.2/1.7
CN9965-2500LG4077-Y21-G		20	2.5	56	6	Gold 6130	$1,894	16	2.1/1.6/1.3
CN9965-2400LG4077-Y21-G		20	2.4	56	6	Gold 6126	$1,776	12	2.6/2.2/1.7
CN9965-2300LG4077-Y21-G		20	2.3	56	6	Gold 5120	$1,555	14	2.2/1.8/1.2
CN9965-2200LG4077-Y21-G		20	2.2	56	6	Gold 5120	$1,555	14	2.2/1.8/1.2
CN9965-2100LG4077-Y21-G		20	2.1	56	6	Gold 5118	$1,273	12	2.3/1.9/1.2
CN9965-2000LG4077-Y21-G		20	2.0	56	6	Gold 5118	$1,273	12	2.3/1.9/1.2
CN9965-1800LG4077-Y21-G		20	1.8	56	6	Silver 4116	$1,002	12	2.1/1.7/1.1
CN9965-1800LG4077-Y21-G		20	1.8	56	4	Silver 4116	$1,002	12	2.1/1.7/1.1
CN9960-2200LG4077-Y21-G		16	2.2	48	4	Silver 4116	$1,002	12	2.1/1.7/1.1
CN9960-2000LG4077-Y21-G		16	2.0	48	4	Silver 4114	$694	10	2.2/1.8/1.1
CN9960-1800LG4077-Y21-G		16	1.8	48	4	Silver 4110	$501	8	2.1/1.7/1.0
CN9960-1600LG4077-Y21-G	$800	16	1.6	48	4	Silver 4108	$417	8	1.8/1.4/0.9

Note that we have only listed the base clock frequencies for Intel (non-AVX, AVX2, and AVX-512). Since Cavium did not provide turbo frequencies and TDPs, a direct comparison is quite difficult. It’s also worth pointing out that Intel models have three turbo frequencies, making the comparison highly workload-specific. Regardless of the model, all Intel models in the table have six memory channels, x48 PCIe lanes, and all features enabled. It’s also worth noting that the Xeon Gold 6000 series, as well as all Xeon Platinum models, come with a second AVX-512 FMA unit.