Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dataloader2 is forced to copy data #12

Open
coreyjadams opened this issue Jun 4, 2019 · 1 comment
Open

dataloader2 is forced to copy data #12

coreyjadams opened this issue Jun 4, 2019 · 1 comment

Comments

@coreyjadams
Copy link
Member

7% of execution time to read batches of data with dataloader2 is spent copying data. When trying to not copy data, an error occurs:

Traceback (most recent call last):
  File "bin/resnet3d.py", line 79, in <module>
    main()
  File "bin/resnet3d.py", line 53, in main
    trainer.initialize(io_only=True)
  File "/home/cadams/DLP3/NEXT_SparseEventID/src/utils/trainercore.py", line 155, in initialize
    self._initialize_io()
  File "/home/cadams/DLP3/NEXT_SparseEventID/src/utils/trainercore.py", line 82, in _initialize_io
    self._larcv_interface.prepare_manager('primary', io_config, FLAGS.MINIBATCH_SIZE, data_keys)
  File "/home/cadams/DLP3/dlp/lib/python2.7/site-packages/larcv-3.0a1-py2.7-linux-x86_64.egg/larcv/larcv_interface.py", line 90, in prepare_manager
    self.next(mode)
  File "/home/cadams/DLP3/dlp/lib/python2.7/site-packages/larcv-3.0a1-py2.7-linux-x86_64.egg/larcv/larcv_interface.py", line 116, in next
    self._dataloaders[mode].next(store_event_ids=True, store_entries=True)
  File "/home/cadams/DLP3/dlp/lib/python2.7/site-packages/larcv-3.0a1-py2.7-linux-x86_64.egg/larcv/dataloader2.py", line 257, in next
    storage.set_data(next_storage_id, batch_data)
  File "/home/cadams/DLP3/dlp/lib/python2.7/site-packages/larcv-3.0a1-py2.7-linux-x86_64.egg/larcv/dataloader2.py", line 66, in set_data
    self._npy_data = larcv.as_ndarray(larcv_batchdata.data())
NotImplementedError: Wrong number or type of arguments for overloaded function 'as_ndarray'.
  Possible C/C++ prototypes are:
    larcv3::as_ndarray(std::vector< short,std::allocator< short > > const &)
    larcv3::as_ndarray(std::vector< unsigned short,std::allocator< unsigned short > > const &)
    larcv3::as_ndarray(std::vector< long long,std::allocator< long long > > const &)
    larcv3::as_ndarray(std::vector< unsigned long long,std::allocator< unsigned long long > > const &)
    larcv3::as_ndarray(larcv3::Image2D const &)

It looks like there is some work to do to fix this, but it would give a moderate boost to io performance and data pipelining.

@marcodeltutto
Copy link
Member

#14 should resolve this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants