AMD Smashes Expectations with AI Servers 28.3 Times More Efficient Than 2020 Models

Performance efficiency is essential for the ongoing advancements in AI and HPC processors, prompting fierce competition among companies like AMD with each new generation of products. In 2021, AMD set an ambitious target to enhance the energy efficiency of its EPYC processors and Instinct accelerators by 30 times by the year 2025, compared to their 2020 benchmarks. It seems that AMD has nearly reached this milestone with the introduction of its EPYC 9005-series ‘Turin’ CPUs and Instinct MI300X GPUs, achieving this goal a year ahead of schedule.

To demonstrate this achievement, AMD deployed a system outfitted with two 64-core EPYC 9575F CPUs, eight Instinct MI300X accelerators, and 2,304 GB of DDR5 memory to test its inference capabilities using the Llama3.1-70B (vLLM 0.6.1.post2, TP8 Parallel, FP8, continuous batching) model. Through a series of complex calculations, AMD assessed the energy efficiency of this setup and found it to be 28.3 times more energy efficient than a comparable 2020 system, the specifics of which were not disclosed.

While AMD did not reveal the details of the 2020 system, it likely involved the company’s earlier EPYC 7002-series processors, which utilized the Zen 2 microarchitecture and could support up to 64 cores per CPU, along with Instinct MI100 accelerators based on the CDNA 1 architecture. [EDIT: AMD has provided details here. We have also included a table below to illustrate the configurations used.] 

The Instinct MI100 accelerators, unlike the newer MI300X, do not support FP8 operations (though they do support INT8 at the same rate as MI300X). Comparing INT8 performance between MI100 (184.6 TOPS) and MI300X (2615 TOPS/5230 TOPS with sparsity), shows a 14 – 28-fold increase. A similar scale of improvement is visible with FP16 performance, validating the comparison. Given the significantly enhanced memory subsystems (from 32 GB HBM2 at 1.20 GB/s to 192 GB HBM3 at 5.30 GB/s) and improved CPUs, it’s unsurprising that AMD’s latest machines significantly outperform and are more energy-efficient than those from 2020.

AMD attributes the leap in performance efficiency not only to hardware enhancements but also to architectural innovations and software improvements, which are standard expectations in such advancements.

Recently, AMD introduced its Instinct MI325X accelerators, which employ the CDNA 3 architecture and include a 288 GB HBM3E memory subsystem. Looking ahead to next year, AMD plans to launch its Instinct MI355X processors, based on the CDNA 4 architecture, which will enhance compute FP8 and FP16 performance by approximately 80% over the MI325X. Additionally, the MI325X will support new FP4 and FP6 formats for AI, boosting its peak performance to 9.2 PetaFLOPS (FP4), a critical advancement for handling extensive language models. With these developments, AMD is well on its way to surpassing its 30×25 energy efficiency target.

Sam Naffziger, Senior Vice President, AMD Corporate Fellow, and Product Technology Architect, expressed confidence in the company’s strategy, noting, “With our strategic approach to hardware and software integration, we are optimistic about our roadmap exceeding the 30×25 goal and excited about future opportunities for substantial energy efficiency gains in the coming years.”

AMD continues to identify opportunities for further enhancements and envisions a potential to increase energy efficiency by up to 100X by 2027.