Distributed training notes
Model parallelism
Splitting computation within model operations and layers.
Background & research notes · AGIACC
Technical note. Tensor partitioning is a performance mechanism with additional coordination requirements.
Split the operation#
Tensor parallelism partitions selected tensor operations across workers. The partitioning and collectives depend on the model and implementation. PyTorch’s tensor parallelism documentation describes APIs and supported layouts.
This can help when individual computation units are too large or when a suitable interconnect makes cooperation effective. Communication, memory layout, and kernel support determine the practical tradeoff.
Review the worker group#
A tensor-parallel group shares intermediate values and depends on coordinated execution. Define admission controls, communication assumptions, artifact integrity, and failure handling for the whole group. Logical tensor partitions alone do not isolate mutually distrusting tenants.
Evaluate end-to-end time and memory on the target topology rather than transferring a speedup from a different model or device count.
Next: Pipeline parallelism.