Distributed training notes
Data parallelism
Replicated computation and sharded state across training workers.
Background & research notes · AGIACC
Technical note. Efficiency depends on model state, communication, and the workload configuration.
Replication and sharding#
In replicated data parallelism, workers process different data with model replicas and synchronize gradients. PyTorch documents its DistributedDataParallel design, including gradient communication.
State sharding reduces per-worker replication. DeepSpeed ZeRO partitions optimizer state, then gradients, then parameters across its stages. PyTorch FSDP2 is another implementation path for sharded training.
Questions beyond peak throughput#
Sharding changes when and where parameters are materialized. A security review should identify which workers, checkpoint readers, storage systems, and administrators can access that state. A shard is not automatically a confidentiality boundary.
Benchmark peak memory, communication volume, checkpoint/restart behavior, and end-to-end time on a stated workload. There is no universal scaling-efficiency percentage.
Next: Model parallelism.