Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev-infctx][batch 4] Torch.compile + deepspeed_3 support + typo/notebook/readme changes #45

Open
wants to merge 63 commits into
base: dev-infctx
Choose a base branch
from

Conversation

PicoCreator
Copy link

  • tweaks in the model to support

    • both Torch.compile (disabled unless we are on torch 2.1.0 by default)
      • Torch.compile is currently only on nightly branch for some repos, which is unstable !!!
    • deepspeed_3 (it technically still has a warning for trace issues, but according to the validation run, it works)
  • bump versions in readme for

    • lightning==2.0.4 deepspeed==0.9.5
    • pytorch==2.0.1 pytorch-cuda=11.8
  • change default ctx_len_cutoff to blank array (disable)

  • fixed a typo of multi_column_separator_encodings (was multi_column_seperator_encodings) for data processing

  • reorganised notebooks for better reproducibility

PS: let me know if you want this to be split up, the bulk of the core changes however is only in model.py (require batch 3 merge), with the rest being typo/readme/notebook changes

PicoCreator and others added 30 commits June 25, 2023 10:12
Merge pull request #7 from PicoCreator/dev-infctx-lr-final
last line whitespace change (force GH merge conflict to revaluate)
[dev-infctx][batch 3] Trainer validation notebooks (#41)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants