AI inference refers to the execution of already trained AI models to generate predictions, classifications, or content. Unlike training, model weights are not continuously updated during inference. Instead, AI inference systems must process requests at defined throughput levels and within specified latency limits. For IT architects deploying LLM serving platforms, real-time recommendation engines, or computer vision applications, maximum compute performance is therefore not the only priority. What matters is which server architecture delivers the required performance at an economically viable level of utilization.
This article explains what hardware is required for AI inference, when CPU- or GPU-based inference makes sense, and the roles of memory capacity, memory bandwidth, interconnects, and serving software. It also examines which accelerator classes are suitable for different inference workloads.
What Is the Difference Between AI Inference and Training?
During AI training, model parameters are adjusted through repeated calculations and the processing of large datasets. AI inference, by contrast, uses already trained model weights to process new inputs and generate results.
| Criterion | AI Inference | AI Training |
|---|---|---|
| Objective | Apply a trained model to new inputs | Optimize model parameters using training data |
| Computations | Forward pass to calculate predictions or outputs | Forward and backward passes, gradient calculations, and parameter updates |
| Compute performance | Depends on the model, batch size, and request profile | High compute performance required for training and gradient calculations |
| Memory requirements | Model weights, activations, and KV cache for autoregressive Transformers | Model weights, gradients, activations, and optimizer states |
| Memory bandwidth | Particularly important for LLM decoding with small batch sizes | Relevant for data movement during training |
| Key performance metrics | Latency, token throughput, request capacity, and cost per request | Training throughput, convergence time, and model quality |
| Hardware sizing | Model size, precision, context length, concurrency, and latency targets | Model size, training method, precision, and training duration |
For large language model inference, token generation at small batch sizes is often limited by memory bandwidth. Processing longer inputs, also known as prefill, and using larger batch sizes can be more constrained by available compute performance. A 70-billion-parameter model requires approximately 140 GB for its model weights alone at FP16 precision. During autoregressive token generation, these weights must be repeatedly read from GPU memory for successive decoding steps. The speed of this process depends on memory bandwidth, model architecture, and the efficiency of the inference software.
Reducing numerical precision also plays an important role in inference. While mixed-precision techniques are frequently used during training, inference models can often run using INT8, FP8, or 4-bit quantization. This can reduce memory requirements. However, whether sufficient model quality is maintained must be evaluated for the specific model and use case.
Request processing adds another consideration: inference systems often need to handle fluctuating arrival rates, varying input lengths, and multiple concurrent users. Dynamic batching can improve GPU utilization by processing several requests together. However, larger batches do not automatically improve the performance of individual requests and may introduce additional waiting times.
AI Inference on CPUs, GPUs, or Specialized Accelerators
CPUs are a practical option for inference workloads involving smaller models, moderate request volumes, or relatively relaxed latency requirements. For example, an image classification model that processes requests only occasionally can be operated economically on an Intel Xeon processor or AMD EPYC processor without requiring a dedicated GPU. Modern server processors also support vector and matrix operations that can accelerate certain inference computations. Whether CPU inference is sufficient depends on the model architecture, the software used, and the desired throughput. For larger models, high request volumes, and compute-intensive applications, AI GPUs are increasingly used. They provide specialized compute units and high memory bandwidth for parallel processing of model operations.
Different accelerator classes are available for GPU-accelerated inference. The NVIDIA RTX PRO 6000 Blackwell Server Edition with 96 GB of GDDR7, for example, is suitable as a platform for professional inference applications, multimodal models, and production AI servers. For larger language models and memory-intensive workloads, accelerators with high HBM capacity, such as the NVIDIA H200 or AMD Instinct MI350X and MI355X, may be suitable.
For demanding aggregate throughput requirements and the parallel processing of many requests, multi-GPU platforms featuring NVIDIA B200 or B300 accelerators can be useful. HBM4-based accelerators such as NVIDIA Rubin and AMD Instinct MI455X expand the possibilities for particularly demanding inference infrastructures. However, selection should not be based solely on the GPU generation. Model size, precision, memory requirements, concurrency, and the latency and throughput values achievable in practice are the decisive factors.
GPU Memory Capacity and Bandwidth for AI Inference
GPU memory capacity is a major factor in determining which models can run on an accelerator. A 7-billion-parameter model theoretically requires approximately 14 GB for its model weights at FP16 precision. Additional memory is needed for the KV cache, activations, runtime environment, and other computations. A server GPU with 24 GB may be sufficient for certain 7B inference configurations, but this does not represent a universally applicable minimum requirement.
A 70B model requires approximately 140 GB for its weights alone at FP16 precision. A single NVIDIA H100 with 80 GB cannot hold these weights entirely in its local GPU memory. An AMD Instinct MI300X with 192 GB of HBM3, by contrast, provides sufficient capacity for the model weights and additional memory space. How much memory remains available for context and concurrent requests depends on the specific implementation. In addition to capacity, memory bandwidth affects performance. The NVIDIA H100 SXM, for example, offers up to 3.35 TB/s of HBM bandwidth, while AMD specifies up to 5.3 TB/s for the MI300X. These figures are technical specifications of the respective accelerators and do not constitute a direct comparison of achievable inference throughput.
Higher memory bandwidth can improve token throughput at small batch sizes, provided that token generation is actually memory-bandwidth-bound. At larger batch sizes, compute performance, cache capacity, and scheduling can become more important.
How Much System RAM Does an Inference Server Need?
The server’s system RAM supports preprocessing and postprocessing, data buffering, model deployment, and other tasks performed by the serving software. Computer vision applications, for example, require resources for image decoding and scaling. For LLMs, additional tasks include tokenization, request management, and processing generated outputs.
The required system memory capacity depends on the number of GPUs, model deployment, request volume, and preprocessing and postprocessing tasks. General recommendations such as 512 GB or 1 TB of RAM are therefore not reliable without a specific workload description.
LLM Inference: KV Cache, Quantization, and Multi-GPU Serving
Large language models place particular demands on GPU memory. In addition to model weights, further data must be available to process active requests. In autoregressive Transformer models, text generation proceeds step by step. A further forward pass is executed for each new token. Particularly at small batch sizes, repeatedly processing the model weights can place heavy demands on memory bandwidth.
Why Is the KV Cache Important?
The KV cache stores the key and value tensors of previously processed tokens so that this information does not have to be fully recomputed during subsequent decoding steps.
The required memory depends on several factors:
- Model architecture and number of Transformer layers
- Number and dimensions of key-value heads
- Context length and KV cache precision
- Number of concurrently processed requests
With longer context windows and high concurrency, the KV cache can consume a substantial portion of available GPU memory. The parameter count of a model alone is therefore insufficient to determine total VRAM requirements. Two models with comparable parameter counts can have significantly different cache requirements due to their architectures.
How Does Quantization Affect Memory Requirements?
Quantization reduces the numerical precision of model weights and, where applicable, other data structures. A 70B model theoretically requires approximately the following amount of memory for its weights:
| Precision | Model Weight Memory Requirements |
|---|---|
| FP16 / BF16 | 140 GB |
| INT8 | 70 GB |
| 4-bit | 35 GB |
Simplified calculation excluding quantization metadata, additional model structures, KV cache, and runtime overhead.
Quantizing model weights can reduce the amount of data that must be transferred during token generation. For memory-bandwidth-bound workloads, this can improve throughput. Quantization does not change the GPU’s physical memory bandwidth. Furthermore, the actual performance gain depends on whether the hardware and serving framework efficiently support the selected precision formats. Model quality must also be evaluated after quantization using representative tasks.
Tensor and Pipeline Parallelism
When a model is distributed across multiple GPUs, different parallelization methods are available. With tensor parallelism, computations within individual model layers are distributed across multiple accelerators. This results in regular communication between GPUs. Available interconnect bandwidth and communication latency therefore influence scaling.
With pipeline parallelism, different model layers are distributed across separate GPUs. This can change the communication requirements between accelerators but introduces dependencies between processing stages. Which method is appropriate depends on the model architecture, GPU topology, batch size, and latency targets. Tensor parallelism is not inherently faster, and pipeline parallelism is not automatically the better choice for PCIe systems.
In large inference deployments, both methods can be combined. For high request volumes, deploying several independent model instances may also be more appropriate than distributing a single model across all GPUs.
Serving Software for AI Inference: Triton, TensorRT, and ONNX Runtime
Serving software influences which server hardware can be used and how efficiently available resources are utilized.
NVIDIA Triton Inference Server
NVIDIA Triton Inference Server (the manager) supports various backends, including TensorRT, ONNX Runtime, PyTorch, and TensorFlow. Its features include dynamic batching, management of multiple models, and HTTP/gRPC interfaces. Hardware requirements depend on the backend used. TensorRT requires NVIDIA GPUs, while ONNX Runtime also enables CPU-based inference depending on the execution provider.
When sizing a Triton server, request processing, preprocessing and postprocessing, and the parallel execution of multiple models must be considered alongside GPU performance.
NVIDIA TensorRT and TensorRT-LLM
NVIDIA TensorRT (the tuner) optimizes neural networks for execution on supported NVIDIA GPUs. This includes selecting suitable compute kernels, optimizing operators, and utilizing supported precision formats.
TensorRT-LLM extends these capabilities for large language models and supports optimized decoding methods, quantization, and inflight batching, among other features. Available functionality depends on the TensorRT-LLM version, GPU architecture, and specific model. A particular performance improvement over PyTorch cannot be assumed in general. For reliable comparisons, the model, precision, hardware, and test conditions must be identical.
ONNX Runtime
ONNX Runtime (the all-rounder) enables supported models to run through different execution providers. These include CPU-based execution environments and GPU backends for supported hardware platforms. ONNX Runtime can therefore be of interest for inference environments where portability and support for different accelerators are important.
However, achievable performance depends on the model, execution provider, and available optimizations. It cannot be generally assumed that ONNX Runtime delivers lower performance than TensorRT across all models and configurations.
Framework Selection Influences Server Architecture
A deployment designed around TensorRT is tied to supported NVIDIA GPUs. For AMD Instinct accelerators, compatibility with ROCm and the intended inference frameworks must be verified. Serving platforms supporting multiple backends can offer advantages for heterogeneous model environments. At the same time, the required runtime environments, model instances, and memory resources must be considered. The serving software should therefore be selected before the final server configuration is determined.
AI Inference Sizing: Model, Hardware, and Workload
Sizing an inference server begins with the characteristics of the model and the requirements of the application. These include parameter count, model architecture, target precision, the expected number of concurrent requests, and desired response times. For LLM serving in particular, input and output lengths must be considered separately. Processing long inputs places different demands on hardware than subsequent token generation.
NVIDIA also addresses the relationship between workload requirements and total cost of ownership in its technical article on GPU sizing for AI inference. Planning should therefore consider not only model size but also Time to First Token (TTFT), token throughput, concurrency, and GPU utilization. Only on this basis can it be determined whether additional GPU capacity improves cost efficiency or merely creates unused resources.
Typical Inference Requirements and Suitable GPU Platforms
Based on our experience in consulting and configuring professional AI servers, we have matched typical inference requirements with suitable GPU and platform classes. The overview is intended as guidance for hardware selection and does not represent a direct performance ranking.
| Inference Requirement | Suitable GPUs / Platforms | Key Criteria |
|---|---|---|
| Professional AI inference | NVIDIA RTX PRO 6000 Blackwell Server Edition, NVIDIA L40S | Model size, GPU memory, latency, power requirements |
| Memory-intensive LLM inference | NVIDIA H200, AMD Instinct MI350X / MI355X | HBM capacity, memory bandwidth, KV cache, model precision |
| Scaled multi-GPU inference | NVIDIA HGX B200 / B300, AMD Instinct MI350-series platforms | Concurrent requests, aggregate throughput, GPU interconnect, serving framework |
| Next-generation inference infrastructure | NVIDIA Vera Rubin platforms, AMD Instinct MI455X-based systems | HBM4, scale-up/scale-out architecture, networking, platform availability |
The actual suitability of a platform depends on the specific model and serving configuration. A GPU with high memory capacity is not automatically the most cost-effective solution for every inference application.
Likewise, a platform with multiple independent model instances may be more suitable for certain request patterns than a large multi-GPU configuration, provided that each instance has sufficient GPU memory.
How Can Hardware Requirements Be Determined Systematically?
Reliable capacity planning should include the following steps:
- Determine model requirements: Architecture, parameter count, precision, and memory requirements for model weights.
- Assess the request profile: Concurrent users, input and output tokens, context lengths, and expected peak loads.
- Define performance targets: TTFT, token generation rate, overall latency, and required throughput.
- Size GPU memory: Model weights, KV cache, runtime overhead, and required reserves.
- Select the platform architecture: Single GPU, multiple independent GPU instances, or distributed model serving.
- Test real workloads: Measure latency, throughput, memory usage, and utilization under representative load.
- Evaluate cost efficiency: Consider hardware costs, power requirements, infrastructure overhead, and cost per processed token.
Only after these steps should the final CPU configuration, system memory, network connectivity, and storage configuration be determined.
A server optimized for training is not necessarily the best solution for inference. In particular, the balance between GPU resources, CPU performance, memory, and interconnect must match the actual serving workload.
Practical Example: AI Inference with a 70B Model in an Enterprise
A company wants to deploy a 70B language model for an internal knowledge platform. The application must serve multiple concurrent users and generate responses with low waiting times. At FP16 precision, the model requires approximately 140 GB for its weights alone. Additional memory is needed for the KV cache, runtime overhead, and other requirements. Quantization can reduce the memory requirements of the model weights. Whether the model can subsequently run on a single GPU depends on the available capacity and additional memory requirements. When multiple GPUs are used, a decision must be made between distributing the model across several accelerators and deploying multiple independent instances on GPUs with sufficient memory capacity. This decision affects both communication requirements and achievable concurrency. For a well-founded hardware selection, different configurations must therefore be tested using representative input and output lengths. The highest theoretical compute performance does not necessarily deliver the lowest cost per request.
For AI Inference, the Workload Determines the Hardware
A real-time computer vision application places different demands on server architecture than an internal LLM platform or a highly scaled inference service. The decisive factors in hardware selection are the interaction between model architecture, GPU memory, interconnect, and serving software, as well as the required latency and throughput values.
Planning GPU servers for AI inference? server-hardware.com supports companies, system integrators, and technical buyers in selecting and configuring suitable server systems. Send us your requirements regarding model, framework, latency, and throughput — our server specialists will advise you on the appropriate system architecture.
Technical Sources:
Online Marketing Manager
