Add support for remote checkpoints and train data files #237

epwalsh · 2023-07-31T21:53:44Z

Adds support for restarting training from a remote checkpoint by setting --load_path to a folder on S3 or GCS (e.g. gs://ai2-olmo/4245903).
Adds support for uploading checkpoints to S3 or GCS during training by setting --remote_save_folder to an S3/GCS "folder" (e.g. gs://ai2-olmo/ai2-llm/olmo-medium/tlui2x1q/step70000).
Adds support for specifying remote training data files (on S3 or CGS) in training configs (globs not supported).

dirkgr · 2023-08-02T00:05:45Z

olmo/util.py

+
+
+def is_remote_file(path: PathOrStr) -> bool:
+    return re.match(r"[a-z0-9]+://.*", str(path)) is not None


I'm OK with this, but technically you can say "file://foo/bar.txt".

Add support for remote checkpoints

9f75dd6

epwalsh requested a review from dirkgr July 31, 2023 21:54

epwalsh added 3 commits July 31, 2023 14:54

update train.py

517d4b0

Add support for remote training data files

5a11cd1

fix merge conflicts

8b21459

epwalsh changed the title ~~Add support for remote checkpoints~~ Add support for remote checkpoints and train data files Aug 1, 2023

fix

ee34096

dirkgr approved these changes Aug 2, 2023

View reviewed changes

handle 'file://' scheme

3af656c

epwalsh merged commit 642d0fa into main Aug 2, 2023
10 checks passed

epwalsh deleted the remote-checkpoints branch August 2, 2023 20:22

Provide feedback



		def is_remote_file(path: PathOrStr) -> bool:
		return re.match(r"[a-z0-9]+://.*", str(path)) is not None