confusing contact_head in RnaFmPreTrainingHeads

[xzdlj]

Thank you very much for providing such an excellent library!

As stated in the README and the original paper, the only pretraining dataset for the RnaFM model consists of sequences from RNAcentral, which do not include labels for contact prediction. However, I noticed that the RnaFmPreTrainingHeads class includes a contact_head that appears to require labels to compute its loss. This is quite confusing to me.

If I have misunderstood or missed anything, please feel free to point it out.

[ZhiyuanChen]

Thank you for raising the very first issue regarding MultiMolecule.

Unfortunately, I don’t have a definitive answer at the moment. We’ve made every effort to ensure that our converted checkpoint matches the official one provided by the authors. From what I know, the official checkpoint does include a concat_head.

If I had to take an educated guess, I would say the authors simply copied the structure from ESM and may have forgotten to remove the concat_head.

[xzdlj]

Many thanks for the response!

If I had to take an educated guess, I would say the authors simply copied the structure from ESM and may have forgotten to remove the concat_head.

I believe the guess makes a lot of sense.
I’ve also raised the issue in the official repository, so hopefully, we might receive a conclusive answer soon.

[ZhiyuanChen]

No worries! I hope you’ve found MultiMolecule useful and helpful.

And don’t forget to let me know if you think anything can be added—new models, datasets, tools, or features for training and fine-tuning neural networks.
Anything you find interesting is welcome! 🤔

[ZhiyuanChen]

Hey, just wanted to let you know that we have introduced a new pipeline for RNA secondary structure prediction.

You can use a pre-trained model for predicting RNA secondary structure on your own data in just a few lines!

import multimolecule  # you must import multimolecule to register models
from transformers import pipeline


predictor = pipeline("rna-secondary-structure", model="multimolecule/ernierna-ss")

output = predictor("ggucucucugguuagaccagaucugagccu")