Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Finetuning #258

Open
wants to merge 36 commits into
base: master
Choose a base branch
from
Open

Finetuning #258

wants to merge 36 commits into from

Commits on Mar 3, 2019

  1. Add finetuning code.

    Neil Shepperd committed Mar 3, 2019
    Configuration menu
    Copy the full SHA
    d1fc873 View commit details
    Browse the repository at this point in the history

  2. chmod +x

    Neil Shepperd committed Mar 3, 2019
    Configuration menu
    Copy the full SHA
    1fba31f View commit details
    Browse the repository at this point in the history

  3. Add finetuning instructions

    Neil Shepperd committed Mar 3, 2019
    Configuration menu
    Copy the full SHA
    dfca3cf View commit details
    Browse the repository at this point in the history

  4. Fix sample generation with batch_size greater than 1.

    Neil Shepperd committed Mar 3, 2019
    Configuration menu
    Copy the full SHA
    9423776 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2019

  1. Add training script with Horovod support 

    This enables multi-GPU or distributed training using Horovod
    @tlkh
    tlkh authored Mar 18, 2019
    Configuration menu
    Copy the full SHA
    3e18729 View commit details
    Browse the repository at this point in the history

  2. Fix typo in train command in README

    @tlkh
    tlkh authored Mar 18, 2019
    Configuration menu
    Copy the full SHA
    ec16bad View commit details
    Browse the repository at this point in the history

  3. Added instructions for training using Horovod

    @tlkh
    tlkh authored Mar 18, 2019
    Configuration menu
    Copy the full SHA
    0bad9e4 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2019

  1. Merge pull request #2 from tlkh/finetuning 

    Added instructions and script for distributed training with Horovod
    @nshepperd
    nshepperd authored Mar 19, 2019
    Configuration menu
    Copy the full SHA
    ef62678 View commit details
    Browse the repository at this point in the history

  2. autoformat

    Neil Shepperd committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    c465071 View commit details
    Browse the repository at this point in the history

  3. Combine input text files with <|endoftext|> delimiter to ensure there… 

    … are enough contiguous tokens to sample from.

This can be disabled with --combine 0.
    Neil Shepperd committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    1e32b10 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2019

  1. Write losses to summary file for tensorboard.

    Neil Shepperd committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    3a3ce65 View commit details
    Browse the repository at this point in the history

  2. Add learning rate as command line flag.

    Neil Shepperd committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    d5b387b View commit details
    Browse the repository at this point in the history

  3. Use argparse instead of fire in train.py.

    Neil Shepperd committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    b106d0a View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2019

  1. Fix encode.py

    Neil Shepperd committed Mar 21, 2019
    Configuration menu
    Copy the full SHA
    2044d13 View commit details
    Browse the repository at this point in the history

  2. Add gradient accumulation with default of 5 minibatches

    Neil Shepperd committed Mar 21, 2019
    Configuration menu
    Copy the full SHA
    a359a34 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2019

  1. Merge remote-tracking branch 'origin/master' into finetuning

    Neil Shepperd committed Mar 25, 2019
    Configuration menu
    Copy the full SHA
    8738950 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2019

  1. Turn off gradient accumulation by default, it shouldn't be needed.

    Neil Shepperd committed May 2, 2019
    Configuration menu
    Copy the full SHA
    eda8777 View commit details
    Browse the repository at this point in the history

Commits on May 4, 2019

  1. Add gradient checkpointing and another optimization necessary to allo… 

    …w training the 345M model.
    Neil Shepperd committed May 4, 2019
    Configuration menu
    Copy the full SHA
    47df6da View commit details
    Browse the repository at this point in the history

  2. Add "validation" loss calculation.

    Neil Shepperd committed May 4, 2019
    Configuration menu
    Copy the full SHA
    c46ed99 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2019

  1. Add toposort to requirements

    @Tenoke
    Tenoke committed May 5, 2019
    Configuration menu
    Copy the full SHA
    941a762 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2019

  1. Merge pull request #3 from Tenoke/finetuning 

    Add toposort to requirements
    Neil Shepperd authored May 6, 2019
    Configuration menu
    Copy the full SHA
    13c5412 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2019

  1. Add option to use SGD for optimizer

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    3985cc7 View commit details
    Browse the repository at this point in the history

  2. Record learning rate in tensorboard logs

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    7fc2a44 View commit details
    Browse the repository at this point in the history

  3. Add text in README for --optimizer flag

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    a464925 View commit details
    Browse the repository at this point in the history

  4. Reduce default learning rate of train.py.

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    ae535b6 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into finetuning

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    2d4fd0c View commit details
    Browse the repository at this point in the history

Commits on May 15, 2019

  1. New feature: add noise to network inputs to regularize against overre… 

    …acting to typos.
    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    6a77a7b View commit details
    Browse the repository at this point in the history

  2. Add top-p sampling

    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    87fe3d7 View commit details
    Browse the repository at this point in the history

  3. Add top_p to interactive_conditional_samples.py and generate_uncondit… 

    …ional_samples.py.
    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    e99ee37 View commit details
    Browse the repository at this point in the history

  4. fix typo in top_p

    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    2b24145 View commit details
    Browse the repository at this point in the history

  5. Fix top_p sampling for batch_size>1

    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    6c1f21d View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2019

  1. Updated README.md 

    Added the medium blog link "Beginner’s Guide to Retrain GPT-2 (117M) to Generate Custom Text Content"
    @biranchi2018
    biranchi2018 authored Aug 15, 2019
    Configuration menu
    Copy the full SHA
    cca7144 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2019

  1. Merge pull request #22 from biranchi2018/biranchi2018-patch-1 

    Updated README.md
    N Shepperd authored Aug 27, 2019
    Configuration menu
    Copy the full SHA
    a070f38 View commit details
    Browse the repository at this point in the history

  2. Add note to install cudnn, re nshepperd#8

    Neil Shepperd committed Aug 27, 2019
    Configuration menu
    Copy the full SHA
    50fa3b6 View commit details
    Browse the repository at this point in the history

  3. Add flag to set encoding for text reading and writing, defaulting to … 

    …utf-8.
    Neil Shepperd committed Aug 27, 2019
    Configuration menu
    Copy the full SHA
    b7cda3f View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2020

  1. Replace Dockerfile with a fixed one

    @neo2478
    neo2478 committed Aug 13, 2020
    Configuration menu
    Copy the full SHA
    b8cd943 View commit details
    Browse the repository at this point in the history