Instrusion pretrain

[quoc-khanh]

About the pretrain part, was your work '.fit' two time (on Instrusion then the train dataset) or following any other techniques?

[zhao-zilong]

Hi @quoc-khanh for the pretrain, yes, I pretrain with the dataset sequentially. First say intrusion, then the train dataset. I actually also tried to pretrain with all the datasets together then on the train dataset. I forgot to update the arxiv. So I just updated the arxiv pdf, it may update in next few days, then you can see the new result that I used several datasets together to pretrain the LLM and then on the train dataset.