InvalidArgumentError: attack from SpsaWithRandomSpatialAttack

[tengerye]

I got an error when running the attacks from SpsaWithRandomSpatialAttack:

tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [] [Condition x <= y did not hold element-wise:x (while/Exit_1:0) = ] [[[[0.0285807811 0.0468903147 0.0507806689]]]...] [y (assert_less_equal/y:0) = ] [0.0690196082]
	 [[{{node assert_less_equal/Assert/AssertGuard/Assert}} = Assert[T=[DT_STRING, DT_STRING, DT_FLOAT, DT_STRING, DT_FLOAT], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](assert_less_equal/Assert/AssertGuard/Assert/Switch/_39, assert_less_equal/Assert/AssertGuard/Assert/data_0, assert_less_equal/Assert/AssertGuard/Assert/data_1, assert_less_equal/Assert/AssertGuard/Assert/Switch_1/_41, assert_less_equal/Assert/AssertGuard/Assert/data_3, assert_less_equal/Assert/AssertGuard/Assert/Switch_2/_43)]]
	 [[{{node assert_less_equal/Assert/AssertGuard/Assert/_46}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_179_assert_less_equal/Assert/AssertGuard/Assert", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

[tengerye]

Everything is fine with running the demo. But the error rises when it runs my own algorithm. The following is the traceback of error:

Traceback (most recent call last):
  File "/unrestricted-adversarial-examples/examples/undefended_pytorch_resnet/toy_defense.py", line 82, in <module>
    main(parse_arguments(sys.argv[1:]))
  File "/unrestricted-adversarial-examples/examples/undefended_pytorch_resnet/toy_defense.py", line 77, in main
    eval_kit.evaluate_bird_or_bicycle_model(my_very_robust_model)
  File "/unrestricted-adversarial-examples/unrestricted-advex/unrestricted_advex/eval_kit.py", line 301, in evaluate_bird_or_bicycle_model
    attack_list=attack_list)
  File "/unrestricted-adversarial-examples/unrestricted-advex/unrestricted_advex/eval_kit.py", line 107, in evaluate_two_class_unambiguous_model
    logits, labels, correct, x_adv, image_ids = run_attack(model_fn, attack_data_iter, attack)
  File "/unrestricted-adversarial-examples/unrestricted-advex/unrestricted_advex/eval_kit.py", line 53, in run_attack
    x_adv = attack_fn(model, x_np, y_np)
  File "/unrestricted-adversarial-examples/unrestricted-advex/unrestricted_advex/attacks.py", line 596, in __call__
    x_adv = self.spsa_attack(model, x_after_spatial_np, y_np)
  File "/unrestricted-adversarial-examples/unrestricted-advex/unrestricted_advex/attacks.py", line 195, in __call__
    self.y_label: np.expand_dims(y_np[i], axis=0),
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: assertion failed: [] [Condition x <= y did not hold element-wise:x (while/Exit_1:0) = ] [[[[0.0203881785 0.0502899662 0.0203380361]]]...] [y (assert_less_equal/y:0) = ] [0.0690196082]
	 [[node assert_less_equal/Assert/AssertGuard/Assert (defined at /anaconda3/envs/tf/lib/python3.6/site-packages/cleverhans/attacks_tf.py:1765)  = Assert[T=[DT_STRING, DT_STRING, DT_FLOAT, DT_STRING, DT_FLOAT], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](assert_less_equal/Assert/AssertGuard/Assert/Switch/_39, assert_less_equal/Assert/AssertGuard/Assert/data_0, assert_less_equal/Assert/AssertGuard/Assert/data_1, assert_less_equal/Assert/AssertGuard/Assert/Switch_1/_41, assert_less_equal/Assert/AssertGuard/Assert/data_3, assert_less_equal/Assert/AssertGuard/Assert/Switch_2/_43)]]
	 [[{{node assert_less_equal/Assert/AssertGuard/Assert/_46}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_179_assert_less_equal/Assert/AssertGuard/Assert", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op 'assert_less_equal/Assert/AssertGuard/Assert', defined at:
  File "/unrestricted-adversarial-examples/examples/undefended_pytorch_resnet/toy_defense.py", line 82, in <module>
    main(parse_arguments(sys.argv[1:]))
  File "/unrestricted-adversarial-examples/examples/undefended_pytorch_resnet/toy_defense.py", line 77, in main
    eval_kit.evaluate_bird_or_bicycle_model(my_very_robust_model)
  File "/unrestricted-adversarial-examples/unrestricted-advex/unrestricted_advex/eval_kit.py", line 281, in evaluate_bird_or_bicycle_model
    num_steps=200,
  File "/unrestricted-adversarial-examples/unrestricted-advex/unrestricted_advex/attacks.py", line 592, in __init__
    is_debug=is_debug)
  File "/unrestricted-adversarial-examples/unrestricted-advex/unrestricted_advex/attacks.py", line 181, in __init__
    is_debug=is_debug)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/cleverhans/attacks.py", line 1604, in generate
    is_debug=is_debug,
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/cleverhans/attacks_tf.py", line 1765, in pgd_attack
    check_diff = tf.assert_less_equal(final_perturbation, epsilon * 1.1)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/ops/check_ops.py", line 618, in assert_less_equal
    return control_flow_ops.Assert(condition, data, summarize=summarize)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
    return _add_should_use_warning(fn(*args, **kwargs))
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 167, in Assert
    guarded_assert = cond(condition, no_op, true_assert, name="AssertGuard")
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2097, in cond
    orig_res_f, res_f = context_f.BuildCondBranch(false_fn)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 1930, in BuildCondBranch
    original_result = fn()
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 165, in true_assert
    condition, data, summarize, name="Assert")
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 52, in _assert
    name=name)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): assertion failed: [] [Condition x <= y did not hold element-wise:x (while/Exit_1:0) = ] [[[[0.0203881785 0.0502899662 0.0203380361]]]...] [y (assert_less_equal/y:0) = ] [0.0690196082]
	 [[node assert_less_equal/Assert/AssertGuard/Assert (defined at /anaconda3/envs/tf/lib/python3.6/site-packages/cleverhans/attacks_tf.py:1765)  = Assert[T=[DT_STRING, DT_STRING, DT_FLOAT, DT_STRING, DT_FLOAT], summarize=3, _device="/job:localhost/replica:0/task:0/device:CPU:0"](assert_less_equal/Assert/AssertGuard/Assert/Switch/_39, assert_less_equal/Assert/AssertGuard/Assert/data_0, assert_less_equal/Assert/AssertGuard/Assert/data_1, assert_less_equal/Assert/AssertGuard/Assert/Switch_1/_41, assert_less_equal/Assert/AssertGuard/Assert/data_3, assert_less_equal/Assert/AssertGuard/Assert/Switch_2/_43)]]
	 [[{{node assert_less_equal/Assert/AssertGuard/Assert/_46}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_179_assert_less_equal/Assert/AssertGuard/Assert", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Thank you.

[carlini]

The error looks like it's happening from within CleverHans. Can you try running SPSA against your model from within cleverhans to see if you get the same error?

[tengerye]

@carlini Thank you so much for your kind reply. The default defense logits_np = np.array([[-5.0, 5.0]] * batch_size) runs great. How shall I run SPSA within CleverHans please?

[carlini]

You can look here to see how we do it

https://github.com/google/unrestricted-adversarial-examples/blob/master/unrestricted-advex/unrestricted_advex/attacks.py#L159

Basically: create the SPSA class, and call generate_np.

[tengerye]

@carlini Hi, sorry for the late reply. I try to follow your instructions, and change the codes to the following:

def main(args):
    # Evaluate the model (this will take ~10 hours on a GPU)

    # eval_kit.evaluate_bird_or_bicycle_model(my_very_robust_model)

    graph = tf.Graph()

    with graph.as_default():
      sess = tf.Session(graph=graph)

      x_input = tf.placeholder(tf.float32, shape=(1,) + (224, 224, 3))
      y_label = tf.placeholder(tf.int32, shape=(1,))

      attack = SPSA(CleverhansPyfuncModelWrapper(my_very_robust_model), sess=sess)
      x_adv = attack.generate(
        x_input,
        y=y_label,
        epsilon=(16. / 255),
        num_steps=200,
        early_stop_loss_threshold=-1.,
        batch_size=32,
        is_debug=False)

    graph.finalize()

Everything seems fine when running with it. Do you think is it because of the implementation of my model using PyTorch?

[carlini]

Calling generate will only create the graph, not actually run it -- can you try with generate_np?

[tengerye]

@carlini Dear Mr, Carlini, I am still struggling with the code. I tried the following:

x_batch_adv = attack.generate_np(x_input, y=y_label, eps=.3,
                                 nb_iter=40, eps_iter=.01,
                                 rand_init=True,
                                 clip_min=0, clip_max=1)

But it raises another error:

Traceback (most recent call last):
  File "/unrestricted-adversarial-examples/examples/undefended_pytorch_resnet/toy_defense.py", line 131, in <module>
    main(parse_arguments(sys.argv[1:]))
  File "/unrestricted-adversarial-examples/examples/undefended_pytorch_resnet/toy_defense.py", line 122, in main
    clip_min=0, clip_max=1)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/cleverhans/attacks.py", line 149, in generate_np
    self.construct_graph(fixed, feedable, x_val, hash_key)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/cleverhans/attacks.py", line 123, in construct_graph
    x_adv = self.generate(x, **new_kwargs)
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/cleverhans/attacks.py", line 1604, in generate
    is_debug=is_debug,
  File "/anaconda3/envs/tf/lib/python3.6/site-packages/cleverhans/attacks_tf.py", line 1723, in pgd_attack
    minval=-epsilon, maxval=epsilon,
TypeError: bad operand type for unary -: 'NoneType'

It is really hard for me after so much effort. If it is possible, would you mind providing some snippets that I can work from? Thank you so much.