Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Changes for 65B and 70B runs #414

Merged
merged 130 commits into from
Mar 19, 2024
Merged

Changes for 65B and 70B runs #414

merged 130 commits into from
Mar 19, 2024

Commits on Jan 8, 2024

  1. Stops training after a specified number of steps

    @dirkgr
    dirkgr committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    e7a73ce View commit details
    Browse the repository at this point in the history

  2. Get ready to run 16h at a time

    @dirkgr
    dirkgr committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    ead441e View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2024

  1. Switch to cosine LR

    @dirkgr
    dirkgr committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    9152ea5 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2024

  1. Merge remote-tracking branch 'origin/main' into mitchish65-2

    @dirkgr
    dirkgr committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    9b2de63 View commit details
    Browse the repository at this point in the history

  2. Batch size warmup config, doubling once

    @dirkgr
    dirkgr committed Jan 19, 2024
    Configuration menu
    Copy the full SHA
    f860ad1 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2024

  1. 128 nodes

    @dirkgr
    dirkgr committed Jan 22, 2024
    Configuration menu
    Copy the full SHA
    672e4b4 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. More ephemeral checkpoints

    @dirkgr
    dirkgr committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    41cfd01 View commit details
    Browse the repository at this point in the history

  2. Run roughly 1000 steps per slurm job

    @dirkgr
    dirkgr committed Jan 23, 2024
    Configuration menu
    Copy the full SHA
    ea81867 View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2024

  1. Merge branch 'main' into mitchish65-2

    @dirkgr
    dirkgr authored Jan 24, 2024
    Configuration menu
    Copy the full SHA
    1690d8b View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2024

  1. Eval every 1000 steps regardless of saving

    @dirkgr
    dirkgr committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    98e93e4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'mitchish65-2' of https://github.com/allenai/LLM into mi… 

    …tchish65-2
    @dirkgr
    dirkgr committed Jan 25, 2024
    Configuration menu
    Copy the full SHA
    0348696 View commit details
    Browse the repository at this point in the history

Commits on Feb 6, 2024

  1. Go back to 48h jobs

    @dirkgr
    dirkgr committed Feb 6, 2024
    Configuration menu
    Copy the full SHA
    f1bab7a View commit details
    Browse the repository at this point in the history

Commits on Feb 7, 2024

  1. Implements grouped query attention

    @dirkgr
    dirkgr committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    c5c5e3a View commit details
    Browse the repository at this point in the history

  2. 70B config using GQA

    @dirkgr
    dirkgr committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    09beb0c View commit details
    Browse the repository at this point in the history

  3. Removes OLMo Parallel Block

    @dirkgr
    dirkgr committed Feb 7, 2024
    Configuration menu
    Copy the full SHA
    d0500ab View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2024

  1. Switch to flash attention proper.

    @dirkgr
    dirkgr committed Feb 8, 2024
    Configuration menu
    Copy the full SHA
    1acac5c View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2024

  1. LUMI Dockerfile that builds Flash Attention

    @dirkgr
    dirkgr committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    9c6a144 View commit details
    Browse the repository at this point in the history

  2. Adds another mitchish65 script that lets us specify a random seed

    @dirkgr
    dirkgr committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    53f9ba6 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'mitchish65-2' of https://github.com/allenai/LLM into mi… 

    …tchish65-2
    @dirkgr
    dirkgr committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    d4e5ef7 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'mitchish65-2' into mitchish65-2-gqa

    @dirkgr
    dirkgr committed Feb 9, 2024
    Configuration menu
    Copy the full SHA
    d4b14f9 View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2024

  1. Makes the config actually work

    @dirkgr
    dirkgr committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    bc41306 View commit details
    Browse the repository at this point in the history

  2. Run script

    @dirkgr
    dirkgr committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    fbc8cba View commit details
    Browse the repository at this point in the history

  3. Remove obsolete docstring

    @dirkgr
    dirkgr committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    95dd681 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'mitchish65-2-gqa' of https://github.com/allenai/LLM int… 

    …o mitchish65-2-gqa
    @dirkgr
    dirkgr committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    b9ed52c View commit details
    Browse the repository at this point in the history

  5. We do batch size warmup.

    @dirkgr
    dirkgr committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    5bf7211 View commit details
    Browse the repository at this point in the history

  6. Offline HF Datasets

    @dirkgr
    dirkgr committed Feb 10, 2024
    Configuration menu
    Copy the full SHA
    559ab28 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2024

  1. Adds a config that runs against data in S3

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    96abd69 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'mitchish65-2-gqa' of https://github.com/allenai/LLM int… 

    …o mitchish65-2-gqa
    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    6980f07 View commit details
    Browse the repository at this point in the history

  3. Gantry script

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    e04ee47 View commit details
    Browse the repository at this point in the history

  4. Nail down torch dependency

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    c91ecb4 View commit details
    Browse the repository at this point in the history

  5. Add flash to requirements

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    d8c1ef9 View commit details
    Browse the repository at this point in the history

  6. New Docker image for gantry

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    013bd44 View commit details
    Browse the repository at this point in the history

  7. Trying for a standalone Docker image

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    c9c928d View commit details
    Browse the repository at this point in the history

  8. Why is everything so abstracted away? What's wrong with just writing … 

    …down what you fucking mean?
    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    e286076 View commit details
    Browse the repository at this point in the history

  9. Tighter pin for torch

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    275463a View commit details
    Browse the repository at this point in the history

  10. Lazy import of flash to make this work on Mac

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    cb1343b View commit details
    Browse the repository at this point in the history

  11. Optional dependency on flash

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    ee49920 View commit details
    Browse the repository at this point in the history

  12. Script for re-grouping wandb runs

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    010ef38 View commit details
    Browse the repository at this point in the history

  13. This script is obsolete.

    @dirkgr
    dirkgr committed Feb 15, 2024
    Configuration menu
    Copy the full SHA
    4ffa5ea View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2024

  1. Mitchish70b in S3

    @dirkgr
    dirkgr committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    e6fd8f3 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2024

  1. Hack to force a certain wrapping

    @dirkgr
    dirkgr committed Feb 18, 2024
    Configuration menu
    Copy the full SHA
    5369c4c View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2024

  1. Fix wrap policy without hack

    @epwalsh
    epwalsh committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    e4894bb View commit details
    Browse the repository at this point in the history

  2. use the right wrapping strategy

    @epwalsh
    epwalsh committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    859eb9c View commit details
    Browse the repository at this point in the history

  3. Fix

    @epwalsh
    epwalsh committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    ea847d0 View commit details
    Browse the repository at this point in the history

  4. revert and try debug

    @epwalsh
    epwalsh committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    4f586ba View commit details
    Browse the repository at this point in the history

  5. I'm sick of PyTorch

    @epwalsh
    epwalsh committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    f66c248 View commit details
    Browse the repository at this point in the history

  6. Fix dockerfile

    @dirkgr
    dirkgr committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    527f3ed View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2024

  1. This is how you set base images?

    @dirkgr
    dirkgr committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    deb353b View commit details
    Browse the repository at this point in the history

  2. Fix ninja install

    @dirkgr
    dirkgr committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    241761f View commit details
    Browse the repository at this point in the history

  3. Forgot to install torch

    @dirkgr
    dirkgr committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    9d4c9fe View commit details
    Browse the repository at this point in the history

  4. Match torch version on LUMI

    @dirkgr
    dirkgr committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    05003ef View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. add mcli config

    @epwalsh
    epwalsh committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    f84d7e4 View commit details
    Browse the repository at this point in the history

  2. Fall back to torch when flash-attn not available

    @epwalsh
    epwalsh committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    95fd624 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. Fix the way we call flash attention

    @dirkgr
    dirkgr committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    2f70619 View commit details
    Browse the repository at this point in the history

  2. Test for the fix for how we call flash attention

    @dirkgr
    dirkgr committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    7925438 View commit details
    Browse the repository at this point in the history

  3. Uses flash-attn's "fused" CE loss function, when available (#465)

    @epwalsh
    epwalsh authored Feb 23, 2024
    Configuration menu
    Copy the full SHA
    b82a51f View commit details
    Browse the repository at this point in the history

  4. Remove tests for parallel block

    @dirkgr
    dirkgr committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    20d6a4c View commit details
    Browse the repository at this point in the history

  5. Fix test for flash attention

    @dirkgr
    dirkgr committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    523b23b View commit details
    Browse the repository at this point in the history

  6. Flash Attention does not support Alibi yet

    @dirkgr
    dirkgr committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    4ab7aa4 View commit details
    Browse the repository at this point in the history

  7. beaker configs

    @epwalsh
    epwalsh committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    9e6f251 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2024

  1. Workaround so we can use GQA on LUMI

    @dirkgr
    dirkgr committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    666ffbb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'mitchish65-2-gqa' of https://github.com/allenai/LLM int… 

    …o mitchish65-2-gqa
    @dirkgr
    dirkgr committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    dd59f98 View commit details
    Browse the repository at this point in the history

  3. The flash attention flag is now about Tri Dao's flash attention.

    @dirkgr
    dirkgr committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    37eaf9c View commit details
    Browse the repository at this point in the history

  4. Flash attention is unsafe in general, so it's off by default, and on … 

    …where we need it
    @dirkgr
    dirkgr committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    4cb93c4 View commit details
    Browse the repository at this point in the history

  5. Turn on FA for MosaicML hardware

    @dirkgr
    dirkgr committed Feb 24, 2024
    Configuration menu
    Copy the full SHA
    c3084c5 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. update gantry image build

    @epwalsh
    epwalsh committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    4fccc21 View commit details
    Browse the repository at this point in the history

  2. Make fused CE loss opt-in

    @epwalsh
    epwalsh committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    54919e0 View commit details
    Browse the repository at this point in the history

  3. Allow setting different data order seed

    @epwalsh
    epwalsh committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    b66a599 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. Skip saving final checkpoint if already exists

    @epwalsh
    epwalsh committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    f2331a8 View commit details
    Browse the repository at this point in the history

  2. update config

    @epwalsh
    epwalsh committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    2292afd View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2024

  1. update config

    @epwalsh
    epwalsh committed Feb 29, 2024
    Configuration menu
    Copy the full SHA
    21e288a View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2024

  1. Merge branch 'main' into mitchish65-2

    @epwalsh
    epwalsh committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    ea13729 View commit details
    Browse the repository at this point in the history

  2. fix merge conflicts

    @epwalsh
    epwalsh committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    394bb22 View commit details
    Browse the repository at this point in the history

  3. Fix?

    @epwalsh
    epwalsh committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    f537966 View commit details
    Browse the repository at this point in the history

  4. Add type annotation

    @epwalsh
    epwalsh committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    951b7ca View commit details
    Browse the repository at this point in the history

  5. Update install in mcli config

    @epwalsh
    epwalsh committed Mar 1, 2024
    Configuration menu
    Copy the full SHA
    ddf8467 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2024

  1. Fix

    @epwalsh
    epwalsh committed Mar 2, 2024
    Configuration menu
    Copy the full SHA
    05d29b8 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. try GQA directly with SDPA

    @epwalsh
    epwalsh committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    e80587a View commit details
    Browse the repository at this point in the history

  2. revert

    @epwalsh
    epwalsh committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    e9c6f20 View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2024

  1. Prepare for sequence length ablation

    @epwalsh
    epwalsh committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    66a99ff View commit details
    Browse the repository at this point in the history

  2. Pick and choose eval sets

    @epwalsh
    epwalsh committed Mar 7, 2024
    Configuration menu
    Copy the full SHA
    f84bb62 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2024

  1. fix merge conflicts

    @epwalsh
    epwalsh committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    c7962be View commit details
    Browse the repository at this point in the history

  2. Fix edge case in update_legacy_settings

    @epwalsh
    epwalsh committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    b58fef4 View commit details
    Browse the repository at this point in the history

  3. CHANGELOG

    @epwalsh
    epwalsh committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    cfc362c View commit details
    Browse the repository at this point in the history

  4. fix merge conflicts

    @epwalsh
    epwalsh committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    1ae86c5 View commit details
    Browse the repository at this point in the history

  5. clean up

    @epwalsh
    epwalsh committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    9556dd9 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'mitchish65-2-gqa' into epwalsh/mitchish65-2-gqa

    @epwalsh
    epwalsh committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    1e2729c View commit details
    Browse the repository at this point in the history

  7. Fix config

    @epwalsh
    epwalsh committed Mar 8, 2024
    Configuration menu
    Copy the full SHA
    f60ae3a View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2024

  1. don't set start method, this causes a crash sometimes

    @epwalsh
    epwalsh committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    40cf61c View commit details
    Browse the repository at this point in the history

  2. try setting start method again

    @epwalsh
    epwalsh committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    5da1c7e View commit details
    Browse the repository at this point in the history

  3. update config

    @epwalsh
    epwalsh committed Mar 9, 2024
    Configuration menu
    Copy the full SHA
    1e60b73 View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. Ensure we raise pickle-able errors with new-style checkpointing

    @epwalsh
    epwalsh committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    fb15986 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2024

  1. Fix canceling run via W&B tags

    @epwalsh
    epwalsh committed Mar 12, 2024
    Configuration menu
    Copy the full SHA
    f2fdf99 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2024

  1. Force setting start method

    @epwalsh
    epwalsh committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    1394236 View commit details
    Browse the repository at this point in the history

  2. Always print the start method

    @epwalsh
    epwalsh committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    3897360 View commit details
    Browse the repository at this point in the history

  3. Create S3 clients before launching threads

    @epwalsh
    epwalsh committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    d625ccb View commit details
    Browse the repository at this point in the history

  4. Default to 1 thread with RemoteFileSystemWriter

    @epwalsh
    epwalsh committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    bc73854 View commit details
    Browse the repository at this point in the history

  5. Log start method

    @epwalsh
    epwalsh committed Mar 13, 2024
    Configuration menu
    Copy the full SHA
    17b3316 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2024

  1. Update configs for longer seq length

    @epwalsh
    epwalsh committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    058e20e View commit details
    Browse the repository at this point in the history

  2. update default microbatch size

    @epwalsh
    epwalsh committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    7ddb124 View commit details
    Browse the repository at this point in the history

  3. clean up how we apply activation checkpointing to blocks

    @epwalsh
    epwalsh committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    671c852 View commit details
    Browse the repository at this point in the history

  4. Add "two_in_three" and "three_in_four" strategies

    @epwalsh
    epwalsh committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    439e18f View commit details
    Browse the repository at this point in the history

  5. update configs with improved settings for throughput

    @epwalsh
    epwalsh committed Mar 14, 2024
    Configuration menu
    Copy the full SHA
    fba3dee View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. update how we set env vars

    @epwalsh
    epwalsh committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    b99b93e View commit details
    Browse the repository at this point in the history

  2. welp, back to our own checkpointing

    @epwalsh
    epwalsh committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    6103643 View commit details
    Browse the repository at this point in the history

  3. Make find_latest_checkpoint more robust

    @epwalsh
    epwalsh committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    9ed5965 View commit details
    Browse the repository at this point in the history

  4. clean up config

    @epwalsh
    epwalsh committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    8133f58 View commit details
    Browse the repository at this point in the history

  5. Merge pull request #492 from allenai/epwalsh/mitchish65-2-gqa 

    Updates to 70B config and checkpointing
    @epwalsh
    epwalsh authored Mar 15, 2024
    Configuration menu
    Copy the full SHA
    026c26c View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. update evals

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    ca0fe2b View commit details
    Browse the repository at this point in the history

  2. Revert metric name change

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    856860d View commit details
    Browse the repository at this point in the history

  3. update name and branch

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    b1e3855 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #443 from allenai/mitchish65-2-gqa 

    GQA into Mitchich65
    @epwalsh
    epwalsh authored Mar 18, 2024
    Configuration menu
    Copy the full SHA
    fa8ec33 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into mitchish65-2

    @epwalsh
    epwalsh authored Mar 18, 2024
    Configuration menu
    Copy the full SHA
    286675e View commit details
    Browse the repository at this point in the history

  6. Skip flash-attn tests when CUDA not available

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    12755a4 View commit details
    Browse the repository at this point in the history

  7. Add new evals

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    d3f0e41 View commit details
    Browse the repository at this point in the history

  8. Add QKV clipping

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    bc9fe29 View commit details
    Browse the repository at this point in the history

  9. Don't warm up cache

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    832bd31 View commit details
    Browse the repository at this point in the history

  10. Allow nightly

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    fc64b77 View commit details
    Browse the repository at this point in the history

  11. revert torch version

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    74d0527 View commit details
    Browse the repository at this point in the history

  12. Add --safe-tensors option to unshard.py

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    6a97372 View commit details
    Browse the repository at this point in the history

  13. update evals

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    d31f14c View commit details
    Browse the repository at this point in the history

  14. Fix?

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    888fc44 View commit details
    Browse the repository at this point in the history

  15. fix filename

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    169b7b8 View commit details
    Browse the repository at this point in the history

  16. fix

    @epwalsh
    epwalsh committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    28719d8 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2024

  1. Update configs, add tokenizer

    @epwalsh
    epwalsh committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    824c985 View commit details
    Browse the repository at this point in the history

  2. update eos token ID

    @epwalsh
    epwalsh committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    8e80f23 View commit details
    Browse the repository at this point in the history

  3. Revert debug

    @epwalsh
    epwalsh committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    f8580eb View commit details
    Browse the repository at this point in the history

  4. Comment

    @epwalsh
    epwalsh committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    c6c114d View commit details
    Browse the repository at this point in the history

  5. update beaker launch script

    @epwalsh
    epwalsh committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    92d2a08 View commit details
    Browse the repository at this point in the history

  6. final touches 

    - Update cluster
- Up-sample wikipedia
    @epwalsh
    epwalsh committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    ed0c5b1 View commit details
    Browse the repository at this point in the history