ValueError: Tensor adanet/iteration_0/ensemble_t0_3_self_built_cnn/weighted_subnetwork_0/subnetwork/batch_normalization_2/moving_mean is not found in /data4/data/adanet/modelZoo/2018_12_10_11_23/model.ckpt-7500 checkpoint

[Npccc]

Hello,
I am using tf 1.12.0,Python 2.7.15，I run my own built subnetwork under the fashion mnist dataset，but I am getting the warning:

Traceback (most recent call last):
File "class_builtCNN.py", line 39, in
steps=None))
File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate
return executor.run()
File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 610, in run
return self.run_local()
File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/estimator/training.py", line 711, in run_local
saving_listeners=saving_listeners)
File "/home/guost/Adanet/Newversion/1206/adanet-master/adanet/core/estimator.py", line 499, in train
self._prepare_next_iteration(input_fn)
File "/home/guost/Adanet/Newversion/1206/adanet-master/adanet/core/estimator.py", line 618, in _prepare_next_iteration
params)
File "/home/guost/Adanet/Newversion/1206/adanet-master/adanet/core/estimator.py", line 578, in _call_adanet_model_fn
self._adanet_model_fn(features, labels, mode, params)
File "/home/guost/Adanet/Newversion/1206/adanet-master/adanet/core/estimator.py", line 1093, in _adanet_model_fn
tf.train.warm_start(latest_checkpoint, vars_to_warm_start=[".*"])
File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/warm_starting_util.py", line 463, in warm_start
_warm_start_var(variable, ckpt_to_initialize_from, prev_var_name)
File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/warm_starting_util.py", line 181, in _warm_start_var
checkpoint_utils.init_from_checkpoint(prev_ckpt, {prev_tensor_name: var})
File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/checkpoint_utils.py", line 187, in init_from_checkpoint
_init_from_checkpoint, ckpt_dir_or_file, assignment_map)
File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/distribute.py", line 1053, in merge_call
return self._merge_call(merge_fn, *args, **kwargs)
File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/distribute.py", line 1061, in _merge_call
return merge_fn(self._distribution_strategy, *args, **kwargs)
File "/home/guost/anaconda2/lib/python2.7/site-packages/tensorflow/python/training/checkpoint_utils.py", line 216, in _init_from_checkpoint
tensor_name_in_ckpt, ckpt_dir_or_file, variable_map
ValueError: Tensor adanet/iteration_0/ensemble_t0_3_self_built_cnn/weighted_subnetwork_0/subnetwork/batch_normalization_2/moving_mean is not found in /data4/data/adanet/modelZoo/2018_12_10_11_23/model.ckpt-7500 checkpoint

Any suggestions on how to solve the problem or avoid it in the first place?
Thank you so much
@cweill

[cweill]

Interesting. Are you using tf.keras.layers or tf.layers? I think I ran into this issue with tf.keras.layers because they handle variable scoping differently.

[Npccc]

You are right，thank you @cweill

[cweill]

Thank you, for confirming @Npccc. We'll look into this.