High Bandwidth Memory is one of the central specifications of an accelerator for AI and machine learning workloads, because the model and its working data have to fit into it. JEDEC published the HBM4 standard in April 2025, and the first commercial HBM4 shipments were announced in February 2026. This article explains what HBM4 changes compared with HBM3E, which accelerators use it and what it means when infrastructure teams plan or buy AI and machine learning servers.
What HBM Is and Why Accelerators Use It
High Bandwidth Memory is stacked DRAM placed directly on the accelerator package beside the GPU die. A very wide interface connects the memory dies to the processor: HBM3E uses a 1024-bit interface, and HBM4 doubles that to 2048 bits. The short distance and the wide interface deliver the bandwidth AI workloads need.
HBM is integrated on the accelerator package. It is not a separate module that can be added or replaced later, so memory capacity and bandwidth are fixed when the accelerator is chosen. An AMD Instinct MI455X carries 432 GB HBM4 across twelve stacks according to AMD; that figure cannot be extended by adding more memory to the server. System memory, meaning the host’s DIMMs or LPDDR, is a separate layer: the accelerator’s HBM holds the model and its active working data, while host memory supports data staging, preprocessing and orchestration.
HBM3E vs. HBM4: What the New Standard Changes
JEDEC published JESD270-4, the HBM4 standard, in 2025. Compared with HBM3, it doubles both the interface width and the number of channels per stack. The table summarises the key figures of the standard.
| HBM4 per JEDEC JESD270-4 | Value |
|---|---|
| Interface width | 2048 bits (HBM3E: 1024 bits) |
| Channels per stack | 32 (HBM3: 16), each with two pseudo-channels |
| Transfer rate per pin | up to 8 Gb/s |
| Bandwidth per stack | up to 2 TB/s |
| Stack heights | 4, 8, 12 or 16 dies of 24 Gb or 32 Gb |
| Maximum capacity per stack | 64 GB (16-high, 32 Gb dies) |
The doubled interface raises total bandwidth at the same per-pin speed: 2048 bits at 8 Gb/s give the up to 2 TB/s per stack defined by the standard. The HBM4 interface definition is backwards compatible with HBM3 controllers, so a single controller can support both standards if needed. Stack height and die density determine capacity per stack: a 12-high stack of 24 Gb dies provides 36 GB, a 16-high stack of 32 Gb dies 64 GB.
Standard Versus Product: Why Data Sheets Show Higher Figures
The JEDEC standard defines the interface and a transfer rate of up to 8 Gb/s for HBM4. Manufacturers ship products that exceed that figure: Samsung announced on 12 February 2026 that its commercial HBM4 runs consistently at 11.7 Gbps, with capability up to 13 Gbps. The 2 TB/s per stack from the standard is therefore not the bandwidth of a shipped accelerator.
Product data sheets state the bandwidth an accelerator actually has. NVIDIA specifies up to 22 TB/s aggregate HBM4 bandwidth for Rubin; AMD specifies 23.3 TB/s for the Instinct MI455X. Both are manufacturer figures for the complete accelerator, not the theoretical value from the JEDEC document. When planning server configurations, use the bandwidth the accelerator manufacturer published, not the standard’s figure.
The same applies to capacity. The standard allows up to 64 GB per stack; the capacity of a product depends on how many stacks the manufacturer places on the package and which stack configuration it uses. Rubin integrates up to 288 GB across eight stacks, the MI455X 432 GB across twelve stacks, which corresponds to 36 GB per stack in both cases. Neither uses the 64 GB per stack the standard permits.
HBM4 GPUs: Which Accelerators Use HBM4
Two examples of accelerators with HBM4 are NVIDIA Rubin and the AMD Instinct MI400 series.
NVIDIA Rubin integrates up to 288 GB HBM4 per GPU with aggregate bandwidth up to 22 TB/s according to NVIDIA. NVIDIA states that HBM4 doubles the interface width compared with the HBM3E of the preceding Blackwell generation, which NVIDIA lists at 8 TB/s. Rubin is paired with the Vera CPU, which carries its own 1.5 TB LPDDR5X memory, coherently connected to the GPU via NVLink-C2C. CPU memory and GPU memory remain two separate layers.
AMD Instinct MI455X, launched on 23 July 2026, integrates 432 GB HBM4 across twelve stacks with a peak memory bandwidth of 23.3 TB/s according to AMD. AMD specifies it as an Enhanced Accelerator Module (EAM) with direct liquid cooling, designed for the AMD Helios rack-scale solution.
Many current data center GPUs, including the AMD Instinct MI355X and NVIDIA Blackwell, use HBM3E. Because HBM sits on the accelerator package, moving from HBM3E to HBM4 means moving to a new accelerator generation and its platform; the memory cannot be swapped on its own.
What HBM4 Means for Planning AI and GPU Servers
HBM capacity per accelerator is the first constraint when matching hardware to a model. Parameters alone take two bytes each in BF16, so a model with 70 billion parameters needs about 140 GB before activations, the key-value cache and framework overhead are added. If the total exceeds the accelerator’s HBM, the model has to be quantised, offloaded or distributed across several accelerators, and each of these options has a cost in accuracy or throughput.
The estimate starts with parameter count and precision: one billion parameters take 4 GB in FP32, 2 GB in BF16 and 1 GB in INT8. Activations grow with batch size and sequence length, the key-value cache of transformer models grows with context length, and the framework adds its own overhead. The sum needs headroom, because longer inputs or larger batches raise memory use at runtime.
The number of accelerators in the GPU server follows from the memory requirement and the HBM capacity per accelerator. Where one accelerator is not enough, the model is distributed across several GPUs within the server or, for larger configurations, across several servers. How quickly the accelerators exchange data then depends substantially on the interconnect deployed, not on HBM.
System memory is separate from HBM. Host DIMMs or the CPU’s LPDDR hold data before it is transferred to the accelerators, support the orchestration layer and handle preprocessing. When configuring the server, both layers must be sized: accelerator HBM for the model and its working data, host memory for staging and concurrent processes. Adding more DIMMs to the host does not extend the accelerator’s memory; if the model does not fit in HBM, it has to be quantised, offloaded or distributed across several accelerators, for example.
Cooling belongs in the same planning step. AMD specifies the MI455X with direct liquid cooling and designs it for its Helios rack-scale solution, so the data center must provide liquid cooling at rack level before such systems can be installed.
In our experience with systems shipped so far with HBM3E accelerators, HBM capacity should always be sized in the context of model size, precision and workload. This applies regardless of whether an NVIDIA GPU server, Supermicro AI server or Gigabyte GPU server is deployed: these factors substantially determine which accelerators and how many GPUs the respective system needs.
HBM4 vs. HBM4E and HBM3E: The Three Generations Compared
The table puts HBM3E, HBM4 and HBM4E side by side. For HBM4 the values are from the JEDEC standard; for HBM3E and HBM4E it gives manufacturer figures as examples, from Micron for HBM3E and from Samsung for HBM4E.
| HBM3E | HBM4 | HBM4E | |
|---|---|---|---|
| Status | in use in current accelerators (e.g. AMD Instinct MI355X, NVIDIA Blackwell) | JEDEC standard JESD270-4 (April 2025); Samsung mass production since February 2026 | Samsung samples since May 2026; mass production aligned with customer schedules |
| Interface | 1024 bits (Micron) | 2048 bits (JEDEC) | not stated in Samsung’s announcement |
| Transfer rate per pin | more than 9.2 Gb/s (Micron) | up to 8 Gb/s (JEDEC); 11.7 Gb/s (Samsung) | 14 Gb/s stable, up to 16 Gb/s (Samsung) |
| Bandwidth per stack | more than 1.2 TB/s (Micron) | up to 2 TB/s (JEDEC) | up to 3.6 TB/s (Samsung) |
| Capacity per stack | 24 GB (8-high), 36 GB (12-high) (Micron) | up to 64 GB (JEDEC) | 48 GB (12 layers); planned 32 GB (8 layers) and 64 GB (16 layers) (Samsung) |
| Accelerators (examples) | AMD Instinct MI355X, NVIDIA Blackwell | NVIDIA Rubin, AMD Instinct MI455X | none announced in the sources used |
HBM4E is the next step after HBM4. Samsung announced on 29 May 2026 that it has begun shipping the first 12-layer HBM4E samples to customers, with a stable pin speed of 14 Gb/s, scalable up to 16 Gb/s, and up to 3.6 TB/s per stack. According to Samsung, energy efficiency improves by 16 percent compared with the previous generation. Samsung plans mass production aligned with customer schedules. Samples are not products: Samsung’s announcement gives no dates for accelerators with HBM4E.
For server planning, what counts is the HBM4 memory built into current accelerators such as NVIDIA Rubin and AMD Instinct MI455X; HBM4E concerns a coming accelerator generation. As with the step from HBM3E to HBM4, HBM4E cannot be retrofitted, because the memory sits on the accelerator package.
HBM Capacity Is Decided with the Accelerator
HBM4 doubles the interface width and channel count compared with HBM3. Shipped products exceed the standard’s 8 Gb/s transfer rate: Samsung’s commercial HBM4 runs at 11.7 Gbps, and the accelerator manufacturers specify up to 22 TB/s (NVIDIA Rubin) and 23.3 TB/s (AMD Instinct MI455X).
For planning AI servers, HBM capacity is fixed when the accelerator is chosen. The memory sits on the accelerator package and cannot be extended later; a model that needs more memory than the accelerator offers has to be quantised, offloaded or distributed. System memory on the host is a separate layer, and adding DIMMs does not extend HBM. When choosing a data center GPU, infrastructure teams therefore size HBM capacity and bandwidth to match the workload and at the same time consider cooling and rack integration.
For AI GPUs for machine learning and generative AI, alongside compute performance, model size, precision, HBM capacity and memory bandwidth all play a role. From our experience with AI and GPU servers we know that for system design, GPU compute performance alone is not decisive. HBM capacity and bandwidth must match the workload just as host memory, interconnect, power supply and cooling must. We advise you on selecting the appropriate server GPUs and support you with compatibility questions, platform choice and configuration. Together with our engineers we configure the appropriate hardware for your AI and machine learning workloads – contact us.
Online Marketing Manager
