Skip to content

Commit

Permalink
Fail when using tracer made without seed checkpoint (#522)
Browse files Browse the repository at this point in the history
Stack from [ghstack](https://github.com/ezyang/ghstack) (oldest at
bottom):
* #473
* __->__ #522
  • Loading branch information
H-Huang authored Aug 16, 2024
1 parent 6c16807 commit f339363
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions train.py
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,11 @@ def loss_fn(pred, labels):
checkpoint_loaded = checkpoint.load()

if parallel_dims.pp_enabled and not checkpoint_loaded:
if job_config.experimental.pipeline_parallel_split_mode == "tracer":
raise RuntimeError(
"Pipeline parallelism with tracer mode is not supported without a seed checkpoint."
)

# TODO: fix this by allowing each rank to set their own seed
logger.warning(
"Pipeline Parallelism is being used without a seed checkpoint. "
Expand Down

0 comments on commit f339363

Please sign in to comment.