[Help] Getting embeddings of ara-gpt2

[kwcckw]

Hi,

Good day.

i'm trying to get the embedding output of ara-gpt2 but looks like facing error：

And here's the code:

    import torch
    
    from transformers import GPT2TokenizerFast, pipeline
    #for base and medium
    from transformers import GPT2LMHeadModel
    #for large and mega
    from aragpt2.grover.modeling_gpt2 import GPT2LMHeadModel
    
    from preprocess import ArabertPreprocessor
    
    device = 0 if torch.cuda.is_available() else -1
    
    MODEL_NAME='aragpt2-base'
    arabert_prep = ArabertPreprocessor(model_name=MODEL_NAME)
    
    text1  = 'ذهب نوح مظفر ضرغام بصحبة رؤوف بن لؤي ظافر عطعوط وهلال  ';
    
    text_clean = arabert_prep.preprocess(text1)
    
    model = GPT2LMHeadModel.from_pretrained(MODEL_NAME)
    tokenizer = GPT2TokenizerFast.from_pretrained(MODEL_NAME)
    generation_pipeline = pipeline("text-generation",model=model,tokenizer=tokenizer, device=device)
    
    input_size = len(tokenizer.tokenize(text_clean))
    
    gen_text = generation_pipeline(text_clean,
                pad_token_id=0, # 0 for AraGPT2
                do_sample = False,
                num_beams=5,
                max_length=input_size + 5,
                top_k=10,
                top_p=0.95,
                repetition_penalty = 3.0,
                no_repeat_ngram_size = 3,
                num_return_sequences = 5)
    
    inputs = tokenizer.encode_plus(text_clean, return_tensors='pt')
    
    print(tokenizer.convert_ids_to_tokens(inputs['input_ids'][0]))
    
    outputs = model(**inputs.to(device))
    
    embeddings = outputs['last_hidden_state']

i checked and doesn't see any 'last_hidden_state' in the outputs too:

Could you advise on how to get the embedding?

Thank you.

[WissamAntoun]

check the following notebook https://colab.research.google.com/drive/17B3bY38CW5VeWkfJsLOaxq6pzHtHKc5W?usp=sharing for how to extract embeddings from GPT2 models. Also please be careful from where are you import the GPT2Model class (use the huggingface library for aragpt2-base and medium, and from arabert.grover... for large and mega)

[kwcckw]

Thanks, looks like i need to enable the hidden state to do allow it:

model = GPT2LMHeadModel.from_pretrained(MODEL_NAME,output_hidden_states=True)