Fill mask for arabertv2, error in preprocess

[kwcckw]

Hi,

Good day.

I was following the code here:
https://github.com/aub-mind/arabert/blob/master/examples/old/AraBERT_Fill_Mask.ipynb

And i'm using the arabertv2:
https://huggingface.co/aubmindlab/bert-base-arabertv2

but i'm facing error with this line of code:
preprocess(text, do_farasa_tokenization=True , farasa = farasa_segmenter, use_farasapy = True)

while i'm able to preprocess it with just preprocess, but i'm not sure on the correctness:
textv2 = arabert_prep.preprocess(text)

This is the code:

    from transformers import AutoModelWithLMHead, AutoTokenizer
    from transformers import pipeline
    from farasa.segmenter import FarasaSegmenter
    from preprocess import ArabertPreprocessor
    
    model_name = "bert-base-arabertv2"
    
    modelv02 = AutoModelWithLMHead.from_pretrained(model_name)
    tokenizerv02 = AutoTokenizer.from_pretrained(model_name)
    arabert_prep = ArabertPreprocessor(model_name=model_name)
    
    pipeline_fill_mask_arabertv2 = pipeline("fill-mask",model=modelv02, tokenizer=tokenizerv02)
    
    
    farasa_segmenter = FarasaSegmenter(interactive=True)
    
    text = " عاصمة لبنان هي [MASK]"
    
    # ok
    textv2 = arabert_prep.preprocess(text)
    
    # error
    textv2 = arabert_prep.preprocess(text, do_farasa_tokenization=True , farasa = farasa_segmenter, use_farasapy = True)
    
    textv2 = textv2.replace("MASK","[MASK]")
    textv2 = textv2.replace(":[",":")
    
    pipeline_fill_mask_arabertv2(textv2)

Could you advise further?

Thank you.

[WissamAntoun]

what is the output of textv2 = arabert_prep.preprocess(text) ?

[kwcckw]

i'm trying with this text:

text = "هناك 24 ساعة لكل [MASK]. "

while i'm getting this:

Looks correct?

[WissamAntoun]

it seems that the preprocess function is removing standalone brackets when it tries to do farasa segmentation. What you can do is textv2 = textv2.replace("MASK","[MASK]") which should fix your issue

[kwcckw]

Okay, and if i didn't specify using farasa in the preprocess, now it is using it by default?

[WissamAntoun]

the preprocessor will automatically apply farasa segmentation when the model passed into it requires segmentation.

[kwcckw]

alright, thanks again for the information.