Invalid argument: indices[24,899,5] = [24, 899, 900] does not index into param shape [25,900,900]

[kmanojkkmr]

Hello,

When i am training the model with the data which was mentioned in this repo(i.e. google drive link) I am getting below error.

I have tried this on Windows 8.1 CPU version and Linux 64 bit CPU.
Tensorflow Version: 2.2.0 also tried on TF version: 1.15.3

I have modified the code according to 2.* version. But I couldn't find the fix for this issue.
Could you please help me what changes to be done to overcome this.

Here is the traceback of the issue.

2020-09-01 10:26:56.662955: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[24,899,5] = [24, 899, 900] does not index into param shape [25,900,900]
2020-09-01 10:26:56.711510: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[24,899,5] = [24, 899, 900] does not index into param shape [25,900,900]
2020-09-01 10:26:56.783215: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[21,788,3] = [21, 788, 900] does not index into param shape [25,900,900]
2020-09-01 10:26:58.902602: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[12,450,3] = [12, 900] does not index into param shape [25,900,128]
2020-09-01 10:26:58.905858: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[23,394,5] = [23, 900] does not index into param shape [25,900,128]
2020-09-01 10:26:58.912144: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at gather_nd_op.cc:47 : Invalid argument: indices[24,899,5] = [24, 900] does not index into param shape [25,900,128]
Traceback (most recent call last):
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[24,899,5] = [24, 899, 900] does not index into param shape [25,900,900]
[[{{node conv_graph_dgcnn_fast_conv_1/GatherNd_10}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "bin/iterate/table_adjacency_parsing.py", line 35, in
trainer.train()
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../iterators/table_adjacency_parsing_iterator.py", line 84, in train
model.run_training_iteration(sess, summary_writer, iteration_number)
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 481, in run_training_iteration
ops_result = sess.run(ops_to_run, feed_dict = feed_dict)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 958, in run
run_metadata_ptr)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1181, in _run
feed_dict_tensor, options, run_metadata)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[24,899,5] = [24, 899, 900] does not index into param shape [25,900,900]
[[node conv_graph_dgcnn_fast_conv_1/GatherNd_10 (defined at /home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py:259) ]]

Errors may have originated from an input operation.
Input Source operations connected to node conv_graph_dgcnn_fast_conv_1/GatherNd_10:
conv_graph_dgcnn_fast_conv_1/Placeholder_5 (defined at /home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py:163)
conv_graph_dgcnn_fast_conv_1/concat_23 (defined at /home/sdpu
ser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py:244)

Original stack trace for 'conv_graph_dgcnn_fast_conv_1/GatherNd_10':
File "bin/iterate/table_adjacency_parsing.py", line 35, in
trainer.train()
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../iterators/table_adjacency_parsing_iterator.py", line 48, in train
model.initialize(training=True)
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 96, in initialize
self.build_computation_graphs()
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 445, in build_computation_graphs
self.build_classification_segments(graph_features, placeholders)
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 337, in build_classification_segments
sampled_indices, computation_graph, gt_matrix = self.do_monte_carlo_sampling(graph_features, gt_sampled_adj_matrix)
File "/home/sdpuser/Desktop/Table_Detection/TIES/TIES-2.0/python_v2/bin/iterate/../../models/basic_model.py", line 259, in do_monte_carlo_sampling
return samples, x, tf.gather_nd(gt_matrix, indexing_tensor_for_adj_matrices)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 4852, in gather_nd_v2
return gather_nd(params, indices, name=name, batch_dims=batch_dims)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 4844, in gather_nd
return gen_array_ops.gather_nd(params, indices, name=name)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3666, in gather_nd
"GatherNd", params=params, indices=indices, name=name)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3327, in _create_op_internal
op_def=op_def)
File "/home/sdpuser/.conda/envs/table_detection/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1791, in init
self._traceback = tf_stack.extract_stack()

[levanpon98]

Try to run on GPU, this error happen when you use tf.gather_nd and the indices is out of bound index. When run on CPU, it will return an error, but when run on GPU, it will replace 0 in the corresponding output value

[kmanojkkmr]

Thanks for the response @levanpon98.
I have changed "is_sampling_balanced" value from 1 to 0 in config.ini file. After the change training started and ran for 19 iterations and then got the same error.
Currently we don't have an option to work on GPU systems. :( So, if there is any changes to be made like by replacing tf.gather_nd with other method(if any) please suggest.

Also do you have any idea how to inference on a single image and extract the information ?
I have completed the table detection part. I should give the table's BBox coordinates for structure recognition and then extract the data from it.

Thanks a lot in advance.

[Ankan1998]

@kmanojkkmr is this issue solved for you. If yes can you please guide me about the solution. I am stuck with same error

[ichko]

@Ankan1998 @kmanojkkmr
I had the same problem, it looks like this is a known issue in tf, and there is no solution:

This is a known issue in TF we cannot change without breaking backward compatibility; gather with invalid indices fails on the CPU but does not on the GPU :-/

tensorflow/tensorflow#34788 (comment)