Pipeline parallelism

Technical note. Pipeline stages are useful places to examine both scheduling and data handoffs.

Partition a sequence

#

Pipeline parallelism places sequential groups of model operations on different workers. Splitting a batch into microbatches can keep more stages busy. The original GPipe paper presents this approach; PyTorch’s pipeline API documents implementation choices and schedules.

Idle periods, stage imbalance, activation memory, and communication affect utilization. Different schedules trade these costs differently, so a stage count does not determine a speedup by itself.

Treat a handoff as an interface

#

A pipeline boundary has assumptions about tensor shape, type, ownership, ordering, and peer identity. Error handling should define how invalid input, a lost worker, or a timeout affects the remaining job and its checkpoints.

These are useful systems questions for trustworthy training infrastructure. They do not imply that partitioning a model provides a security sandbox.

Next: Hybrid parallelism.