Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] register_checkpoint_saver API was not stable under TF version 2.11. #473

Merged
merged 1 commit into from
Oct 17, 2024

Conversation

MoFHeka
Copy link
Collaborator

@MoFHeka MoFHeka commented Oct 16, 2024

Description

[Fix] register_checkpoint_saver API was not stable under TF version 2.11. Therefore, for compatibility considerations, before version 2.14, SingleDeviceSaver was used to save.

Here is the error looks like when using TF 2.9

AssertionError: Tried to export a function which references 'untracked' resource Tensor("movie_id_DELayer-parameter/DynamicEmbedding/movie_id_DELayer-shadow/m/cond/movie_id_DELayer-parameter_DynamicEmbedding_movie_id_DELayer-shadow_m_mht_1of1:0", shape=(), dtype=resource). TensorFlow objects (e.g. tf.Variable) captured by functions must be 'tracked' by assigning them to an attribute of a tracked object or assigned to an attribute of the main object directly. See the information below:
        Function name = b'__inference_restore_fn_5544'
        Captured Tensor = Tensor("movie_id_DELayer-parameter_DynamicEmbedding_movie_id_DELayer-shadow_m_mht_1of1:0", shape=(), dtype=resource, device=/job:localhost/replica:0/task:0/device:CPU:0)
        Trackable referencing this tensor = <tensorflow_recommenders_addons.dynamic_embedding.python.ops.cuckoo_hashtable_ops.CuckooHashTable object at 0x7f4630380a30>

Type of change

  • Bug fix
  • New Tutorial
  • Updated or additional documentation
  • Additional Testing
  • New Feature

Checklist:

  • I've properly formatted my code according to the guidelines
    • By running yapf
    • By running clang-format
  • This PR addresses an already submitted issue for TensorFlow Recommenders-Addons
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works

How Has This Been Tested?

This bug causes the code of the latest main branch to fail to save any DE variable and raise error under TF2.9:

model.fit()
model.save(export_dir, overwrite=True, options=saved_options)

@MoFHeka MoFHeka requested a review from rhdong as a code owner October 16, 2024 13:22
jq
jq previously approved these changes Oct 16, 2024
….11.

Therefore, for compatibility considerations, before version 2.11,
SingleDeviceSaver was used to save.
@MoFHeka MoFHeka changed the title [Fix] register_checkpoint_saver API was not stable under TF version 2.14. [Fix] register_checkpoint_saver API was not stable under TF version 2.11. Oct 16, 2024
@rhdong rhdong merged commit 4799949 into tensorflow:master Oct 17, 2024
39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants