Distributed training notes
Hybrid parallelism
Combining data, tensor, and pipeline dimensions with explicit assumptions.
Background & research notes · AGIACC
Technical note. A hybrid topology must be evaluated as a complete system.
Compose the dimensions#
A training job can use data parallelism between groups, tensor parallelism within operations, and pipeline parallelism between layer groups. The available memory, network topology, model structure, and schedule constrain the useful combinations. Megatron-LM is a primary example of software for training transformer models at scale.
Increasing the number of parallel dimensions also increases configuration and recovery complexity. A performance result should record the full topology, precision, batch and sequence sizes, model, hardware, and software revision.
Keep the trust map explicit#
Performance groups and isolation domains may differ. Identify the services that distribute configuration, admit workers, load checkpoints, and retain credentials. A failure or compromise in orchestration may affect every group even when their computation is partitioned.
Next: Frameworks and tools.