tensorflow.python.framework.errors_impl.InvalidArgumentError

[hieule88]

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[176] = 2530 is not in [0, 2520)
[[{{node InceptionV3/bilinear_attention_pooling/logits/Gather/GatherV2}}]]

I got this Error when trying to run train_sample_aircraft.sh . I made some effort to fix it and found a lot of answers that indicate that the error is in the dataset (which I downloaded from the Drive in readme). Can anyone please check it out?

[hieule88]

This is a part of the error log:

Traceback (most recent call last):
File "/content/drive/MyDrive/ColabNotebooks/Ro-SAC/train.py", line 719, in
tf.app.run()
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/content/drive/MyDrive/ColabNotebooks/Ro-SAC/train.py", line 714, in main
session_config=config
File "/usr/local/lib/python3.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 790, in train
ignore_live_threads=ignore_live_threads)
File "/usr/lib/python3.7/contextlib.py", line 130, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/supervisor.py", line 1014, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/supervisor.py", line 839, in stop
ignore_live_threads=ignore_live_threads)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/lib/python3.7/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/coordinator.py", line 495, in run
self.run_loop()
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/training/supervisor.py", line 1045, in run_loop
[self._sv.summary_op, self._sv.global_step])
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[176] = 2530 is not in [0, 2520)
[[node InceptionV3/bilinear_attention_pooling/logits/Gather/GatherV2 (defined at content/drive/MyDrive/ColabNotebooks/Ro-SAC/nets/inception_v3_topk.py:657) ]]

Original stack trace for 'InceptionV3/bilinear_attention_pooling/logits/Gather/GatherV2':
File "content/drive/MyDrive/ColabNotebooks/Ro-SAC/train.py", line 719, in
tf.app.run()
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/platform/app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "usr/local/lib/python3.7/dist-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "usr/local/lib/python3.7/dist-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "content/drive/MyDrive/ColabNotebooks/Ro-SAC/train.py", line 602, in main
clones = model_deploy.create_clones(deploy_config, clone_fn, [batch_queue])
File "content/drive/MyDrive/ColabNotebooks/Ro-SAC/deployment/model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, **kwargs)
File "content/drive/MyDrive/ColabNotebooks/Ro-SAC/train.py", line 527, in clone_fn
logits_1, end_points = network_fn(images, indices)
File "content/drive/MyDrive/ColabNotebooks/Ro-SAC/nets/nets_factory.py", line 139, in network_fn
return func(images, indices, dataset_name, num_classes, is_training=is_training, **kwargs)
File "content/drive/MyDrive/ColabNotebooks/Ro-SAC/nets/inception_v3_topk.py", line 657, in inception_v3_topk
topk_cls = tf.gather(params, ids, batch_dims=-1)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/array_ops.py", line 3462, in gather
return _batch_gather(params, indices, batch_dims, axis)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/array_ops.py", line 3626, in _batch_gather
flat_result = gather(flat_params, flat_indices)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/array_ops.py", line 3475, in gather
return gen_array_ops.gather_v2(params, indices, axis, name=name)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 4097, in gather_v2
batch_dims=batch_dims, name=name)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()