Embeddings of [CLS] are different from Notebook 3.03. Generating BERT embedding .ipynb

[pancodia]

I am following the 3.03. Generating BERT embedding .ipynb notebook to learn how to get the embeddings from a BERT model.

I have question about the result of hidden_rep, cls_head = model(token_ids, attention_mask = attention_mask). When I compare the following two values, they are different.

print(hidden_rep[0][0].detach().numpy()[:10])
print(cls_head[0].detach().numpy()[:10])

output:

[-0.0719209   0.2163076   0.00471864 -0.08153436 -0.30399242 -0.26997408
  0.36993372  0.43028143  0.01193172 -0.20673896]
[-0.90660435 -0.3418912  -0.33728653  0.7713965   0.0609756  -0.10524714
  0.90143335  0.2582197  -0.278811   -0.9999693 ]

My understanding is that they are both the embeddings for [CLS]. So I expect them to be the same. Is my understanding not correct?

BTW, I am using transformers == 4.12.5 if that matters.

[pancodia]

I notice that the output type of the model is transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions. Does it mean that the value of cls_head is actually after pooling of individual tokens' embeddings?

[RAravindDS]

I notice that the output type of the model is transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions. Does it mean that the value of cls_head is actually after pooling of individual tokens' embeddings?

I think the version changed. I also see this!