You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The following line creates the second argument incorrectly. It drops the beginning part of the full path that exists on the source file argument. This comes from not using the value of "storage_root" consistently. See following lines for an example of the incorrect output:
Currently, dlio will prepend the "storage_root" value in the first argument, but not in the second argument, which is the path to where the index file should be located. As a result, of this error, index files are not placed into the correct path and will not be found.
If the following two values are passed to this program, shown below as "Orig Path 1" and "Orig Path 2", they will be updated with the values shown in "New Path 1" and "New Path 2".
The following line creates the second argument incorrectly. It drops the beginning part of the full path that exists on the source file argument. This comes from not using the value of "storage_root" consistently. See following lines for an example of the incorrect output:
Currently, dlio will prepend the "storage_root" value in the first argument, but not in the second argument, which is the path to where the index file should be located. As a result, of this error, index files are not placed into the correct path and will not be found.
If the following two values are passed to this program, shown below as "Orig Path 1" and "Orig Path 2", they will be updated with the values shown in "New Path 1" and "New Path 2".
Orig Arg 1: /test/data/resnet50/train/img_0000_of_1278.tfrecord
Orig Arg 2: ./data/resnet50/index/train/img_0000_of_1278.tfrecord.idx
Correct Arg 1: /test/data/resnet50/train/img_0000_of_1278.tfrecord
Correct Arg 2: /test/data/resnet50/index/train/img_0000_of_1278.tfrecord.idx
Notice that the value of "/test" is missing from "Orig Path 2" above, but is added to the path seen in "New Path 2".
[https://github.com/argonne-lcf/dlio_benchmark/blob/7055ff007fd1bacdf5df2d51c383279e05ba9b50/dlio_benchmark/data_generator/tf_generator.py#L79]
The text was updated successfully, but these errors were encountered: