What a cluster is. What it isn't. And the honest difference between real datacenter clustering and a pile of desktop cards.
A model too large to fit in one GPU's memory can be split across multiple GPUs — or multiple machines — so that each piece holds a slice. The GPUs communicate constantly as they process each token. From the outside, it looks like one system. From the inside, it's a coordinated handoff happening thousands of times per second.
This is called model parallelism. It's how you run a 671-billion-parameter model when no single piece of hardware holds 805 GB of GPU memory.
GPU communication speed determines how fast the handoffs happen. Slow communication creates a bottleneck — the model waits for data instead of computing. This is why NVLink and InfiniBand exist.
| Connection | PCIe lanes / CPU bridge |
| GPU-to-GPU bandwidth | ~64 GB/s (shared) |
| Memory pooling | No — each GPU is isolated |
| Run a 156 GB model? | No — 96 GB total, fragmented |
| Failover | None — one card dies, job dies |
| Scheduling | Manual / ad hoc |
| Power per GPU | 450 W × 4 = 1,800 W |
| Total VRAM | 96 GB (4 × 24 GB, not pooled) |
| Price | ~$10,000 |
Desktop cards share a PCIe bus through the CPU. They can cooperate on some tasks (like training a small model), but they can't pool their memory. A model that needs 100 GB cannot run across 4 RTX 4090s — each card still only sees its own 24 GB.
| Connection | NVSwitch + InfiniBand |
| GPU-to-GPU bandwidth | 900 GB/s (NVLink) + 400 Gb/s (IB) |
| Memory pooling | Yes — all GPUs are one logical pool |
| Run a 156 GB model? | Yes — 2,256 GB available |
| Failover | Hardware HA, Kubernetes scheduling |
| Scheduling | Slurm / K8s / NGC containers |
| Power per node | 10,200 W × 2 = 20,400 W |
| Total VRAM | 2,256 GB (16 × 141 GB, fully pooled) |
| Price | ~$930,000 |
DGX nodes use NVSwitch to give all 8 internal GPUs full memory visibility. InfiniBand links the nodes. The result: a 671B-parameter model that needs 805 GB of VRAM fits comfortably, with room to spare, and the system can failover automatically if a node goes down.
Every build in our catalog fits on this line. Desktop cards are fine for development. DGX systems are where real production begins. Clusters are where scale lives.