Add GPT-2 Model and its Variants

[abheesht17]

Partially resolves #337

[mattdangerw]

This looks great! Will take a deeper look, but you can do two things for now

1. Comment with a pointer to wherever you are grabbing the hyperparameters from for the custom sizes. Does openai provide this anywhere directly?
2. Update your colab from the issue to actually use a gpt-2 symbols from this branch? That would be a nice validation that we have fully recreated the correct graph here.

[abheesht17]

@mattdangerw,

This looks great! Will take a deeper look, but you can do two things for now

1. Comment with a pointer to wherever you are grabbing the hyperparameters from for the custom sizes. Does openai provide this anywhere directly?
2. Update your colab from the issue to actually use a gpt-2 symbols from this branch? That would be a nice validation that we have fully recreated the correct graph here.

1. They have a config file in their checkpoint file named hparams.json. This is the format of the links:
   https://openaipublic.blob.core.windows.net/gpt-2/models/{num_params}/hparams.json.

   If you want to double check once, please visit these links:
   GPT-2 Small: https://openaipublic.blob.core.windows.net/gpt-2/models/124M/hparams.json
   GPT-2 Medium: https://openaipublic.blob.core.windows.net/gpt-2/models/355M/hparams.json
   GPT-2 Large: https://openaipublic.blob.core.windows.net/gpt-2/models/774M/hparams.json
   GPT-2 ExtraLarge: https://openaipublic.blob.core.windows.net/gpt-2/models/1558M/hparams.json

   intermediate_dim is always 4 * hidden_dim.

2. Sure, will do this now!

Edit: Here is the updated notebook: https://colab.research.google.com/drive/1vI3dDmFEFOQM0GbDBfVf6sanATjkwV9X?usp=sharing.

[mattdangerw]

Overall lgtm, just a few minor comments.

[mattdangerw]

I think we should get out of the habit of listing the full model name in a first line. That should be short, scannable, and informative.

GPT-2 core network with custom hyperparmeters. or something like that

[abheesht17]

Cool! I have shifted the full name below

[jbischof]

Great work, thank you!

One request: please move your style changes of other files to a separate PR.

[mattdangerw]

Also just a heads up I will be pushing a name change cleanup that will probably require updating any checkpoints conversion scripts using TransformerDecoder. But it's just reworking the layer names.

#353

[abheesht17]

Also just a heads up I will be pushing a name change cleanup that will probably require updating any checkpoints conversion scripts using TransformerDecoder. But it's just reworking the layer names.

@mattdangerw, updated: https://colab.research.google.com/drive/1vI3dDmFEFOQM0GbDBfVf6sanATjkwV9X?usp=sharing!

[abheesht17]

Please hold off from merging this; I'll validate ckpt conversion for all 4 sizes

[abheesht17]

Please hold off from merging this; I'll validate ckpt conversion for all 4 sizes

Good to go! :)