Can I add a composition Block of different adapter types?

[MRawhani]

Can i stack prefix tuning with bottleneck adapter? somehting like config unioin but as composition block.

Here is my scenario:

Let us suppose I have a trained prefix tuning or lora adapter that i want to load to my model and activate it, and i want to use the output of this adapter as an input to a stacked new bottlneck adapter that will be trained to a different task. So both the loaded lora adapter and the new adapter are activated, but only the new one is trained. Can i do that?

here is my code:

lora_mlm_target = model.load_adapter(f"{config.Config.ADAPTER_SAVE_PATH}/lora-mlm-target",with_head=False)
adapter_name= "adapter_test"

model.add_adapter(adapter_name,config="seq_bn")
model.train_adapter([adapter_name])

model.add_classification_head("mnli", num_labels=3)
model.active_adapters = ac.Stack(lora_mlm_target, adapter_name)

• The adapter summery is like i want:

[{'name': 'lora-mlm-target',
'architecture': 'lora',
'active': True,
'#param': 147456,
'train': False,
'%param': 0.22219650503413957},
{'name': 'adapter-test',
'architecture': 'bottleneck',
'active': True,
'#param': 447264,
'train': True,
'%param': 0.6739671334336303},
{'name': 'Full model', '#param': 66362880, '%param': 100.0, 'train': False}]

But when I run evlauate or train it gives me this error:

Invalid adapter setup: str is not a valid adapter name or composition block.

What is the problem? is it not supported?

[lenglaender]

Hi @MRawhani
Are you using the current library version? AFAIK we fixed something related to Stack some time ago. I couldn't replicate your problems with this:

import adapters

model = adapters.AutoAdapterModel.from_pretrained("google-bert/bert-base-uncased")
model.add_adapter("a",config="seq_bn")
model.add_adapter("b",config="lora")
model.add_classification_head("b", num_labels=3)
model.active_adapters = adapters.composition.Stack("a", "b")

This throws no error, and the adapter summary looks like this:

>>> print(model.adapter_summary())
================================================================================
Name                     Architecture         #Param      %Param  Active   Train
--------------------------------------------------------------------------------
a                        bottleneck          894,528       0.817       1       1
b                        lora                294,912       0.269       1       1
--------------------------------------------------------------------------------
Full model                               109,482,240     100.000               1
================================================================================

[MRawhani]

Hi @lenglaender ,
thanks for the reply, i think the key was to update the adapters package. It works now, Thank you.
However, i still have one more question please:
how is the flow of the forward pass now in your example? as i understood from the stack archtecture, the output of "a" will be used as an input of "b", but regarding your example and similar structures, you are making it as the output of the adapter (a) will be used as an input to lora (b) ? but from what i know, lora is before inserted bottleneck adapters, how will that be?

Thanks