Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use tf datasets instead of keras sequence #22

Open
zaccharieramzi opened this issue Nov 29, 2019 · 3 comments
Open

Use tf datasets instead of keras sequence #22

zaccharieramzi opened this issue Nov 29, 2019 · 3 comments
Labels
data Something data related enhancement New feature or request

Comments

@zaccharieramzi
Copy link
Owner

This can be done in the same PR as #15 .

Basically the idea is to have a way cleaner data pipeline, and gain x3 time on training.

@zaccharieramzi zaccharieramzi added the enhancement New feature or request label Nov 29, 2019
@zaccharieramzi
Copy link
Owner Author

zaccharieramzi commented Jan 29, 2020

This is dependent on tensorflow/io#745

@zaccharieramzi
Copy link
Owner Author

Actually, just re-tried with tf datasets using py_function and still using h5py, this results in a 1.5x acceleration compared to sequences for the pdnet training (see pdnet_af4_1568384763 vs pdnet_af8_1581355529). This also resolves #40 so will submit a PR solving both problems by replacing all sequences by datasets at least for training (will remove them from validation some time after).

@zaccharieramzi
Copy link
Owner Author

Partly adressed in #43 .
The validation should now use tf datasets as well to get rid completely of keras Sequences.

@zaccharieramzi zaccharieramzi added the data Something data related label Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data Something data related enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant