[Bug]: ditto

[dz12123]

Issue Type

Build/Install

Modules Involved

Others

Have you reproduced the bug with SPU HEAD?

Yes

Have you searched existing issues?

Yes

SPU Version

ditto

OS Platform and Distribution

22.04

Python Version

3.8

Compiler Version

11.2.1

Current Behavior?

我在复现ditto这篇文章时，遇见了两个问题，第一个问题是RTE的量化模型准确率只有0.47， 第二个问题是我在运行运行实例代码时出现如下错误

Standalone code to reproduce the issue

print("A bug")

Relevant log output

Continuing because --nocheck_visibility is active
INFO: Analyzed target //examples/python/ml/experimental_mp:bert_bench (286 packages loaded, 20960 targets configured).
INFO: Found 1 target...
Target //examples/python/ml/experimental_mp:bert_bench up-to-date:
  bazel-bin/examples/python/ml/experimental_mp/bert_bench
INFO: Elapsed time: 28.555s, Critical Path: 0.12s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/examples/python/ml/experimental_mp/bert_bench --gelu quad --config examples/python/ml/experimental_mp/3pc_nExp.json
No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/1fb23739f2f05359848549583528e814/execroot/spulib/bazel-out/k8-opt/bin/examples/python/ml/experimental_mp/bert_bench.runfiles/spulib/examples/python/ml/experimental_mp/bert_bench.py", line 67, in <module>
    pretrained_model = FlaxBertForSequenceClassification.from_pretrained(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 903, in from_pretrained
    model = cls(config, *model_args, _do_init=_do_init, **model_kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 786, in __init__
    super().__init__(config, module, input_shape=input_shape, seed=seed, dtype=dtype, _do_init=_do_init)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 220, in __init__
    random_params = self.init_weights(self.key, input_shape)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 821, in init_weights
    module_init_outputs = self.module.init(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 166, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 1846, in init
    _, v_out = self.init_with_output(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/jax/_src/traceback_util.py", line 166, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 1747, in init_with_output
    return init_with_output(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/core/scope.py", line 1034, in wrapper
    return apply(fn, mutable=mutable, flags=init_flags)(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/core/scope.py", line 998, in wrapper
    y = fn(root, *args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 2371, in scope_fn
    return fn(module.clone(parent=scope, _deep_clone=True), *args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 467, in wrapped_module_method
    return self._call_wrapped_method(fun, args, kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 967, in _call_wrapped_method
    y = fun(self, *args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 1330, in __call__
    outputs = self.bert(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 467, in wrapped_module_method
    return self._call_wrapped_method(fun, args, kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 967, in _call_wrapped_method
    y = fun(self, *args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 996, in __call__
    outputs = self.encoder(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 467, in wrapped_module_method
    return self._call_wrapped_method(fun, args, kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 967, in _call_wrapped_method
    y = fun(self, *args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 655, in __call__
    return self.layer(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 467, in wrapped_module_method
    return self._call_wrapped_method(fun, args, kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 967, in _call_wrapped_method
    y = fun(self, *args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 595, in __call__
    layer_outputs = layer(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 467, in wrapped_module_method
    return self._call_wrapped_method(fun, args, kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 967, in _call_wrapped_method
    y = fun(self, *args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 537, in __call__
    hidden_states = self.intermediate(attention_output)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 467, in wrapped_module_method
    return self._call_wrapped_method(fun, args, kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 949, in _call_wrapped_method
    self._try_setup()
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 1231, in _try_setup
    self.setup()
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 467, in wrapped_module_method
    return self._call_wrapped_method(fun, args, kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/flax/linen/module.py", line 967, in _call_wrapped_method
    y = fun(self, *args, **kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 464, in setup
    self.activation = ACT2FN[self.config.hidden_act]
jax._src.traceback_util.UnfilteredStackTrace: KeyError: 'quad'

The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.

--------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/1fb23739f2f05359848549583528e814/execroot/spulib/bazel-out/k8-opt/bin/examples/python/ml/experimental_mp/bert_bench.runfiles/spulib/examples/python/ml/experimental_mp/bert_bench.py", line 67, in <module>
    pretrained_model = FlaxBertForSequenceClassification.from_pretrained(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 903, in from_pretrained
    model = cls(config, *model_args, _do_init=_do_init, **model_kwargs)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 786, in __init__
    super().__init__(config, module, input_shape=input_shape, seed=seed, dtype=dtype, _do_init=_do_init)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/modeling_flax_utils.py", line 220, in __init__
    random_params = self.init_weights(self.key, input_shape)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 821, in init_weights
    module_init_outputs = self.module.init(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 1330, in __call__
    outputs = self.bert(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 996, in __call__
    outputs = self.encoder(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 655, in __call__
    return self.layer(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 595, in __call__
    layer_outputs = layer(
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 537, in __call__
    hidden_states = self.intermediate(attention_output)
  File "/root/miniconda3/envs/ditto/lib/python3.8/site-packages/transformers/models/bert/modeling_flax_bert.py", line 464, in setup
    self.activation = ACT2FN[self.config.hidden_act]
KeyError: 'quad'

[zimu-yuxi]

方便提供下参考的文章吗？

[dz12123]

方便提供下参考的文章吗？

方便提供下参考的文章吗？

Ditto: Quantization-aware Secure Inference of Transformers upon MPC

[w-gc]

PTAL @llCurious