Distributed training notes
Frameworks and tools
Primary implementation references for distributed AI workloads.
Background & research notes · AGIACC
Technical note. These are third-party projects, not AGIACC partners or supported integrations.
| Project | Useful starting point |
|---|---|
| PyTorch Distributed | Worker groups and collective communication |
| PyTorch FSDP2 | State-sharded training |
| DeepSpeed | ZeRO memory optimization |
| Megatron-LM | Large-scale transformer training |
| Ray Train | Distributed training orchestration |
Versions and capabilities evolve. Use the documentation matching a pinned release and record its configuration in benchmark artifacts. Selection should account for workload fit, reproducibility, failure recovery, and the privileged services that operate the job.
AGIACC’s research question is how isolation, memory protection, and evidence can compose with real AI systems. Listing a framework does not establish a completed integration.
Next: Security challenges.