Skip to content

Testing data packaging method from CVPR 2023 paper FFCV: Accelerating Training by Removing Data Bottlenecks

Notifications You must be signed in to change notification settings

v1lanboy/ffcv_training

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 

Repository files navigation

FFCV_training

Create FFCV dataset

python FFCV_datasets.py create --dataset-path <path_to_directory> --dataset-name <DatasetClass> --dataset-splits <train/val> --ffcv-directory <path_to_directory> 

Issues:

  • PAGE_SIZE being too low will give malloc error halt the process in memory allocator
  • Default workers in the DatasetWriter is equal to num of cpu cores. For me this crashes the whole system time to time.

About

Testing data packaging method from CVPR 2023 paper FFCV: Accelerating Training by Removing Data Bottlenecks

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages