training scooby

[sergeyvilov]

Hi,

First, I'm trying to reproduce your results for the RNA-only model on NeurIPS (using hg38 as reference). Could you please tell me which metrics are to expect in lora? The validation metrics evolve as on the attached screenshot and I don't think that's the expected behaviour.

Second, do you think that it's ok to use lora to fine-tune to a different genome (e.g. hg19). Or would you expect that complete re-training would be necessary?

I'm training on 2 NVIDIA A100 with --gradient_accumulation_steps 4.

Thanks in advance for your responses

[johahi]

Hi Sergey,

the weights and biases numbers are not super representative of model performance (PR coming soon), which is why we run the model evaluation notebook per epoch.

We trained with batch size 8, and doing 4x gradient accumulation is not equivalent (as Borzoi has batch norms), so I am not sure if you maybe have to lower the learning rate or change other hparams. I am also unsure if you need to warm up the learning rate 4x longer then. We also observed the transform_borzoi_emb to make training less stable, so maybe try to set that to false - it does not drastically change results. Let me know if these changes help :)

Regarding your second question, I guess as long as you lift train-val-test regions over to hg19 to maintain the same data split, it will still work, but we have not tried that.

Cheers,
Johannes

[sergeyvilov]

Hi Johannes,

Thanks a lot for the prompt reply!
I'll test the tricks you advise and let you know if it works.

To evaluate the rna-only model using your evaluation notebook designed for the multiome model, I only change model_type to 'rna' when calling get_pseudobulk_count_pred and fix_rev_comp_multiome to fix_rev_comp_rna in the predict function. Are there other changes to make?

Also, would you mind making the rna-only model available on the hub?

Best,
Sergey