Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not found: Key tower/fully_connected/biases not found in checkpoint #19

Open
estathop opened this issue May 4, 2018 · 2 comments
Open

Comments

@estathop
Copy link

estathop commented May 4, 2018

I downloaded your pretrained model and tried to evaluate it with 10 videos from the training sample, but as I press this command, an error follows. And I can't find why this is happening. Everything else is installed normally

python eval.py --eval_data_pattern=/home/estathop/features/train*.tfrecord --train_dir=/home/estathop/Documents/pretrained/public/ --run_once=True
/home/estathop/tensorflow/local/lib/python2.7/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
tensorflow version: 1.9.0-dev20180502
INFO:tensorflow:Using batch size of 1024 for evaluation.
INFO:tensorflow:number of evaluation files: 43
INFO:tensorflow:built evaluation graph
2018-05-04 15:33:19.482097: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-05-04 15:33:19.573605: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-05-04 15:33:19.573963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1349] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.7845
pciBusID: 0000:01:00.0
totalMemory: 7.93GiB freeMemory: 7.20GiB
2018-05-04 15:33:19.573978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1428] Adding visible gpu devices: 0
2018-05-04 15:33:19.760748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-05-04 15:33:19.760790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:922] 0
2018-05-04 15:33:19.760795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:935] 0: N
2018-05-04 15:33:19.760999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1046] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6955 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Loading checkpoint for eval: /home/estathop/Documents/pretrained/public/model.ckpt-310001
INFO:tensorflow:Restoring parameters from /home/estathop/Documents/pretrained/public/model.ckpt-310001
2018-05-04 15:33:19.884136: W tensorflow/core/framework/op_kernel.cc:1290] CtxFailure at save_restore_v2_ops.cc:184: Not found: Key tower/fully_connected/biases not found in checkpoint
Traceback (most recent call last):
File "eval.py", line 332, in
app.run()
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "eval.py", line 328, in main
evaluate()
File "eval.py", line 320, in evaluate
last_global_step_val)
File "eval.py", line 197, in evaluation_loop
saver.restore(sess, latest_checkpoint)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1746, in restore
six.reraise(exception_type, exception_value, exception_traceback)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1734, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key tower/fully_connected/biases not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT32, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op u'save/RestoreV2', defined at:
File "eval.py", line 332, in
app.run()
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "eval.py", line 328, in main
evaluate()
File "eval.py", line 309, in evaluate
saver = tf.train.Saver(tf.global_variables())
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1266, in init
self.build()
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1278, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1315, in _build
build_save=build_save, build_restore=build_restore)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 763, in _build_internal
restore_sequentially, reshape)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 400, in _AddRestoreOps
restore_sequentially)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 814, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1463, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3408, in create_op
op_def=op_def)
File "/home/estathop/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1734, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

NotFoundError (see above for traceback): Key tower/fully_connected/biases not found in checkpoint
[[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT32, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

@SunnyThakur2000
Copy link

I am also facing the same issue.
Are You using Windows or Linux?

@estathop
Copy link
Author

I am using Ubuntu 16.04 LTS but I traced the problem to be that I tried to evaluate and inference tfrecords of different level features than the model. I wanted video level features and this is an image level feature model

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants