OOM Fix #217
Merged
OOM Fix #217
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Bring v1.0 to the most recent commit (#202) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * sync up (#205) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Bring v1.0 to the most recent commit (#202) (#203) * Request changes from MLPerf Storage (#199) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * Fixed potential insufficient samples due to num_files is not divisible by comm.size (#200) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * recovered back dlio_profiler * fixed potential not enough samples * Update tf_reader.py * Mlperf requests (#201) * added au metric to the configuration file; set shuffling and shuffle buffer size to be 2 for cosmoflow * removed dependencies on dlioprofiler * fixed bugs * fixed issue with dlio_profiler * bring back dlio_profiler_py * Fix requirements file (#204) Signed-off-by: Johnu George <[email protected]> --------- Signed-off-by: Johnu George <[email protected]> Co-authored-by: Johnu George <[email protected]> * barrier in the beginning * fixed bugs * fixed MPI initilization issue --------- Signed-off-by: Johnu George <[email protected]> Co-authored-by: Johnu George <[email protected]>
* Switch DLIO Profiler to DFTracer. * Github workflow fixed. * DFTracer pip package address fixed. * switch mpi to openmpi * added build debug so that symbols are shown. * added DEBUG logging for DFTRACER * removed explicit install of DFTracer as it is done during pip. * added clanup code. * ci cleanup * Updated for DFTracer changes * switched to 1.0.1 Due to a simple pypi bug * Switch to release 1.0.2 * Package info updated. (#210) * Update dali_tfrecord_reader.py to import PerfTrace and Profile from utils.utility * Update dali_npy_reader.py * Update dali_image_reader.py * Update custom_npz_reader.py * Update custom_torch_data_loader.py [ci-skip] * Update pytorch_checkpointing.py [ci-skip] * Update native_dali_data_loader.py [ci-skip] * Publish on PyPI (#211) * PyPI publish workflow added. * `requirements.txt` cleaned up. * Action names refactored. * Missing requirements fixed. * `setup.py` requirement versions refactored. * CI support for running via `requirements.txt`. * `VENV_PATH` fixed in CI script. * CI script conditional executable fix. * Cleared redundant CI matrix options. --------- Co-authored-by: Izzet Yildirim <[email protected]> Co-authored-by: Izzet Yildirim <[email protected]> Co-authored-by: Huihuo Zheng <[email protected]>
The sample building and native data loader case is needed only for DLIO created data loaders. For native data loaders which provide their own API;s they provide their own indexing and there this sampling can be ignored.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixed the OOM issue. It is related to PR #215, but not including the shuffle fix