How to train the three model

[tiancity-NJU]

when I try to train key_gen model, it may needs 1-billion.txt， but I don't find the txt anywhere. Can you tell me how to train my model and reproduce you result? thanks

[J-zin]

FileNotFoundError: [Errno 2] No such file or directory: '../data/1-billion/1-billion.txt'

[NingMiao]

Since I'm not authorized to release 1-billion dataset in my code, please download it from its official website.
By the way, you can use any dataset to train the language model, or even replace the current model with a pre-trained GPT-2.

[Milozms]

Since I'm not authorized to release 1-billion dataset in my code, please download it from its official website.
By the way, you can use any dataset to train the language model, or even replace the current model with a pre-trained GPT-2.

@NingMiao I'm also interested in replacing current model with a pre-trained GPT-2. But the released GPT-2 is only uni-directional, while your method requires a bidirectional (both forward and backward) pre-trained language model. Are there any possible solutions?