DPMultiheadedAttention is not a full drop in replacement for nn.MultiheadAttention

[jfb54]

🐛 Bug

Not only is the API missing the batch_first parameter (#512), it is missing the in_proj_weight parameter. Thus it makes it impossible to use Opacus with a transformer.

Expected behavior

Ability to access the in_proj_weight parameter so that it may be initialized.

Environment

Opacus 1.4
PyTorch 2.01

[HuanyuZhang]

Thanks for reporting bugs. Will fix it and #512 to make it work.

[tranvansang]

I am facing the same issue. In addition to what described in the issue description, forward() call also lacks is_causal parameter.

[HuanyuZhang]

Closed this issue, since we launched fixes in PR(#598). Btw, it is also feasible to use (https://github.com/lxuechen/private-transformers) for transformers in hugging face.