Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug in TFrecord index filename generation #256

Open
russfellows opened this issue Feb 21, 2025 · 0 comments
Open

Bug in TFrecord index filename generation #256

russfellows opened this issue Feb 21, 2025 · 0 comments

Comments

@russfellows
Copy link

The following line creates the second argument incorrectly. It drops the beginning part of the full path that exists on the source file argument. This comes from not using the value of "storage_root" consistently. See following lines for an example of the incorrect output:

Currently, dlio will prepend the "storage_root" value in the first argument, but not in the second argument, which is the path to where the index file should be located. As a result, of this error, index files are not placed into the correct path and will not be found.

If the following two values are passed to this program, shown below as "Orig Path 1" and "Orig Path 2", they will be updated with the values shown in "New Path 1" and "New Path 2".

Orig Arg 1: /test/data/resnet50/train/img_0000_of_1278.tfrecord
Orig Arg 2: ./data/resnet50/index/train/img_0000_of_1278.tfrecord.idx

Correct Arg 1: /test/data/resnet50/train/img_0000_of_1278.tfrecord
Correct Arg 2: /test/data/resnet50/index/train/img_0000_of_1278.tfrecord.idx

Notice that the value of "/test" is missing from "Orig Path 2" above, but is added to the path seen in "New Path 2".

[https://github.com/argonne-lcf/dlio_benchmark/blob/7055ff007fd1bacdf5df2d51c383279e05ba9b50/dlio_benchmark/data_generator/tf_generator.py#L79]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant