Add nanogpt_train to torchbench, and merge with nanogpt_generate into joint folder

[xmfan]

Currently, we have nanogpt_generate, which is the inference only version of https://github.com/karpathy/nanoGPT/blob/master/sample.py. This PR adds the training component, simplified from https://github.com/karpathy/nanoGPT/blob/master/train.py.

Tests

unit tests:
> python test.py -k "test_nanogpt_"
pass

for accuracy check:
> python benchmarks/dynamo/torchbench.py --float16 -dcuda --training --backend=inductor --ddp --multiprocess --accuracy --collect-outputs --only nanogpt 
pass

for perf measurement:
> python benchmarks/dynamo/torchbench.py --float16 -dcuda --training --backend=inductor --ddp --multiprocess --performance --only nanogpt 
1.547x

[xuzhao9]

Can we merge nanogpt_train with nanogpt_generate? Their model.py looks similiar and it is better to have a model with both inference and train tests.

[xmfan]

@xuzhao9 by merging, should we combine them into a new nanogpt model folder, or just reuse the model.py? Also what should be the task associated to the model? I think nanogpt_generate is NLP.GENERATION whereas for training we would want to use NLP.LANGUAGE_MODELING

[xuzhao9]

@xuzhao9 by merging, should we combine them into a new nanogpt model folder, or just reuse the model.py? Also what should be the task associated to the model? I think nanogpt_generate is NLP.GENERATION whereas for training we would want to use NLP.LANGUAGE_MODELING

Yes, if they have the same origin, we should combine them into a new nanogpt model folder. In this case, let's follow the training test and use NLP.LANGUAGE_MODELING.

[xmfan]

nanogpt_generate was included in some CI skips, which need to be renamed too so that we don't break CI

benchmark: #1916 rename done in this PR
pytorch: pytorch/pytorch#109746, will update with this torchbench commit pin

[xuzhao9]

LGTM, thanks for merging nanogpt!

[drisspg]

@msaroufim these changes shouldn't mess up the dashboard right?

[xmfan]

@drisspg Since the dashboard only aggregates (logs) this change should be safe. Once we land this PR, we will update the torchbench commit hash in pytorch repo. The dashboard's job (logs) will checkout the new torchbench version, run all models under torchbenchmark/models/ and aggregate the benchmark results

[facebook-github-bot]

@xmfan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[xmfan]

@pytorchbot merge

[facebook-github-bot]

@xmfan merged this pull request in d00b942.