Distributed training notes
Fundamentals of distributed training
Compute, memory, and communication across training workers.
Background & research notes · AGIACC
Technical note. This series introduces system boundaries rather than an AGIACC training product.
Why distribute work?#
Training can exceed one device’s memory or available compute time. Multiple workers can share data, model state, and computation, with communication and synchronization overhead. Model size alone is not a capacity estimate: gradients, optimizer state, activations, and temporary buffers also consume memory.
The PyTorch distributed overview distinguishes replicated data parallelism, state sharding, tensor parallelism, and pipeline parallelism. These techniques can be combined, subject to model and platform constraints.
Map the trust boundary too#
The same worker groups that exchange tensors also depend on launch services, credentials, checkpoints, communication libraries, and administrative access. Efficient coordination does not establish who is authorized to join a job or access its data.
An evaluation should therefore state both the performance topology and the trust topology, including what happens when a worker fails or an artifact is rejected.
Next: Data parallelism.