Attention can be None in ModernBertForSequenceClassification

[ashmikuz]

In the ModernBertForSequenceClassification class, the attention is never computed outside of the self.model (which is a ModernBertModel). Therefore when the attention is not used as input for the model the .unsqueeze() here fails.
I solved this issue by assagning torch.ones(batch_size,seq_len) to the attention_mask, but I am not sure whether this is correct.

[Rocketknight1]

Hi @ashmikuz, when no attention mask is passed then we can't really work out which positions are masked! Although we could add code to estimate this (like attention_mask = input_ids != self.config.pad_token_id), this is error-prone and I think a better solution is just to raise a clear error in this case, telling the user they have to pass an attention mask if they want to use self.classifier_pooling == "mean".

Would you be interested in making a PR for that?

[tom13878]

Hi @Rocketknight1 , @ashmikuz,

I had the same issue. Is anyone working on this? Otherwise I will raise a PR and add this RuntimeError:

if self.config.classifier_pooling == "mean" and attention_mask is None: raise RuntimeError("Mean pooling requires an attention mask to properly compute the pooled output. Please provide an attention mask to indicate which tokens should be considered in the mean pooling calculation.")

[ashmikuz]

Sorry I was quite busy in the last few days. Shouldn't it match how other models behave? As far as I understand, other models just print a warning and then create an attention mask from torch.ones, right?

[tom13878]

Yes, you are right, I see the torch.ones for e.g. deberta here

I don't see the warning but I may have missed it ...

[ashmikuz]

I'm working on a quick PR, just a moment and i'll send it. Hopefully it fixes the issue and is in line with other models.