AI Workloads Are Changing What "Server" Means: A Practical Guide to Specifying GPU Servers

Specifying a server for AI workloads is a different exercise from specifying one for databases or virtualization. The bottlenecks are different, the power and cooling requirements are different, and buying based on traditional server specs (core count, clock speed) will lead to a system that's mismatched to the actual job. Here's what to check before you buy.

The bottleneck usually isn't the CPU

For traditional workloads, CPU core count and clock speed are often the headline spec. For AI training and inference, the CPU is frequently just an orchestrator — the real work happens on GPUs, and the questions that matter shift accordingly:

• GPU memory (VRAM), not just GPU count — a model that doesn't fit in a single GPU's memory either needs to be split across multiple GPUs or run on cards with more VRAM. This is usually the first wall projects hit, not raw compute.
• Interconnect between GPUs — for multi-GPU training, how the GPUs communicate with each other (and whether that link is a bottleneck) matters more than any single GPU's raw spec sheet number.
• Storage throughput — training workloads read large datasets repeatedly; a storage subsystem that can't keep GPUs fed leaves expensive compute sitting idle.

Training vs. inference: different systems, different priorities

These are often treated as the same buying decision, but they have different sweet spots:

• Training workloads are compute- and memory-bandwidth heavy, run for extended periods, and benefit most from higher-end GPUs with more VRAM and faster interconnects — the upfront cost is higher but idle time is the expensive failure mode.
• Inference workloads (running an already-trained model to serve predictions) are typically less demanding per request but need to handle concurrent requests efficiently — here, GPU count and efficient batching often matter more than having the single most powerful card available.

Buying training-class hardware for a pure inference deployment is a common and expensive over-spec.

Power and cooling: the part that gets underestimated

High-end GPU servers draw significantly more power per rack unit than traditional compute nodes, and this has two knock-on effects that are easy to miss during initial budgeting:

• Power delivery to the rack — existing data center or server room power circuits sized for traditional servers may not support a dense GPU deployment without electrical upgrades.
• Cooling capacity — air cooling that was adequate for a traditional rack may not keep pace with GPU-dense racks; liquid cooling is increasingly common at scale, which is a facility-level decision, not just a server-level one.

Confirming rack power and cooling capacity before ordering hardware avoids the expensive scenario of equipment arriving that the room can't actually support.

What to actually ask before specifying

• What's the largest model (by parameter count or memory footprint) this system needs to run, today and over the next 1–2 years?
• Is the workload training, inference, or both — and does that change which GPU tier actually makes sense?
• Does the server room's existing power and cooling infrastructure support the target rack density, or does that need to be assessed first?
• What's the realistic utilization pattern — a system for occasional experimentation has very different economics than one running production inference continuously?

Summary

The traditional server-buying questions (core count, clock speed) are secondary for AI workloads. The real specification work starts with the model's memory footprint, whether the job is training or inference, and whether the physical room can actually support the power and cooling density the hardware needs.

For general server sourcing considerations before you get to the GPU-specific details, see our enterprise server sourcing guide, or browse NAD's current IT and server hardware range.