Added mamba model support and test CI script

[zzhang37]

What does this PR do?

Added Mamba model support using custom op, and added the test on Mamba, the test command is
GAUDI2_CI=1 RUN_SLOW=1 python -m pytest tests/test_text_generation_example.py -s -v -k "mamba"
It passed on 1.18.0release and 1.19.0, build 497 with corresponding so files.

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[zzhang37]

Just run the command
GAUDI2_CI=1 RUN_SLOW=1 python -m pytest tests/test_text_generation_example.py -s -v -k "mamba"
It will pass on 1.18 and 1.19 version

[regisss]

This should be done automatically in the script if the model type is Mamba

[zzhang37]

The test did the automatically download, but for users want to run the mamba, they can just download the so files form the Habana/mamba card.

[zzhang37]

I also remove the wget and use hf_hub_download in the test script.

[regisss]

I think this will discourage many users to use this model. Can't we put the code you added to the test file to donwload the .so in a new utils file in the Mamba modeling folder? And then we could call it here and in the test.

[zzhang37]

This is just readme file.

[regisss]

I'm not sure what you mean. I think automatically downloading these .so files in the script if the model type is mamba would give a better user experience. Why not doing it there?

[jiminha]

@zzhang37 it seems the mamba run code fails if we don't set this environment variable at all. Is this intended? Can customer run with/without custom code? If the intention is always run mamba with custom kernel, we should put all these download/setup code in modeling_mamba file at the beginning before you load the library.

[zzhang37]

@regisss Removed the all wget and added util to do the job.
@jiminha Just checked with latest build 538, works fine.

[github-actions]

The code quality check failed, please run make style.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zzhang37]

Fixed the make style issue.

[regisss]

@zzhang37 There are some new issues now but these should be quite straightforward to fix:

• Calling adapt_transformers_to_gaudi will return an error if GC_KERNEL_PATH has not been specified before. I suggest to run this piece of code only if GC_KERNEL_PATH is in os.environ. If I understood correctly, GC_KERNEL_PATH should be specified if the user intends to use Mamba right?
• The test doesn't work because GC_KERNEL_PATH is not specified. You should specify it similarly to how it is done here if the tested model is Mamba.

[zzhang37]

@regisss GC_KERNEL_PATH is used by all models. There is a default path for GC_KERNEL_PATH, but mamba required an additional path to add .The problem is GC_KERNEL_PATH needs to be set before the start of the script since Synapse needs to use this path to load the TPC kernels in the beginning, if we add additional path to GC_KERNEL_PATH in the middle of the script, it will miss the loading of the additional kernels we added in the path, the mamba will not able to use that kernel. So that is the reason if we run mamba model, we need to add additional path to GC_KERNEL_PATH beforehand rather than in the middle of the script. Just like command we should use is
C_KERNEL_PATH=/root/.cache/huggingface/hub/models--Habana--mamba/blobs/libcustom_tpc_perf_lib.so:$GC_KERNEL_PATH python run_generation.py .....