Data Prep Kit

Data Prep Kit is a community-driven project that simplifies unstructured data preparation for LLM application development. It addresses the growing challenge of preparing diverse data (language, code, vision, multimodal) for fine-tuning, instruction-tuning, and RAG applications. The modules in the kit have been tested in producing pre-training datasets for the Granite open source LLM models.

Features

• The kit provides a growing set of modules/transforms targeting laptop-scale to datacenter-scale processing.
• The data modalities supported today are: Natural Language and Code.
• The modules are built on common frameworks for Python, Ray and Spark runtimes for scaling up data processing.
• The kit provides a framework for developing custom transforms for processing parquet files.
• The kit uses Kubeflow Pipelines-based workflow automation.

Installation

The latest version of the Data Prep Kit is available on PyPi for Python 3.10, 3.11 or 3.12. It can be installed using:

pip install  'data-prep-toolkit-transforms[all]'

This will install all available transforms.

For guidance on creating the virtual environment for installing the data prep kit, click here.

🚀 Getting Started

Fastest way to experience Data Prep Kit

With no setup necessary, let's use a Google Colab friendly notebook to try Data Prep Kit. This is a simple transform to extract content from PDF files: examples/notebooks/Run_your_first_transform_colab.ipynb | . (Here are some tips for running Data Prep Kit transforms on Google Colab. For this simple example, these tips are either already taken care of, or are not needed.) The same notebook can be downloaded and run on the local machine, without cloning the repo or any other setup.

Examples

Now that you have run a single transform, the next step is to explore how to put these transforms together to run a data prep pipeline for end to end real enterprise use cases like fine-tuning a model or building a RAG application.

We have a complete set of data processing recipes for such use cases.

We also have a developer tutorial for contributing a new transform to the kit.

For advanced users, here is more information for adding your own transform, its scaling and automation. Also,repository structure and use are discussed here.

Windows users

Please click here for guidance on how to run transforms in Windows.

Using HuggingFace data files

All the transforms in the kit include small sample data files for testing, but advanced users who want to download real data files from HuggingFace and use them in testing, can refer to this.

Current list of transforms

The matrix below shows the the combination of modules and supported runtimes. All the modules can be accessed here and can be combined to form data processing pipelines, as shown in the examples folder.

Modules	Python-only	Ray	Spark	KFP on Ray
Data Ingestion
Code (from zip) to Parquet	✅	✅		✅
PDF to Parquet	✅	✅		✅
HTML to Parquet	✅	✅		✅
Web to Parquet	✅
Universal (Code & Language)
Exact dedup filter	✅	✅		✅
Fuzzy dedup filter	✅	✅	✅	✅
Unique ID annotation	✅	✅	✅	✅
Filter on annotations	✅	✅	✅	✅
Profiler	✅	✅	✅	✅
Resize	✅	✅	✅	✅
Hate, Abuse, Profanity (HAP)	✅	✅		✅
Tokenizer	✅	✅		✅
Language-only
Language identification	✅	✅		✅
Document quality	✅	✅		✅
Document chunking for RAG	✅	✅		✅
Text encoder	✅	✅		✅
PII Annotator/Redactor	✅	✅		✅
Similarity	✅
Code-only
Programming language annotation	✅	✅		✅
Code quality annotation	✅	✅		✅
Malware annotation	✅	✅		✅
Header cleanser	✅	✅		✅
Semantic file ordering		✅
License Select Annotation	✅	✅		✅
Code profiler	✅	✅

Contributing

Contributors are welcome to add new modules to expand to other data modalities as well as add runtime support for existing modules! Please read this for details.

Get help and support

Please feel free to connect with us using the discussion section.

Resources

Papers, talks, presentations and tutorials.

Citation

If you use Data Prep Kit in your research, please cite our paper:

@misc{wood2024dataprepkitgettingdataready,
      title={Data-Prep-Kit: getting your data ready for LLM application development}, 
      author={David Wood and Boris Lublinsky and Alexy Roytman and Shivdeep Singh 
      and Constantin Adam and Abdulhamid Adebayo and Sungeun An and Yuan Chi Chang 
      and Xuan-Hong Dang and Nirmit Desai and Michele Dolfi and Hajar Emami-Gohari 
      and Revital Eres and Takuya Goto and Dhiraj Joshi and Yan Koyfman 
      and Mohammad Nassar and Hima Patel and Paramesvaran Selvam and Yousaf Shah  
      and Saptha Surendran and Daiki Tsuzuku and Petros Zerfos and Shahrokh Daijavad},
      year={2024},
      eprint={2409.18164},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2409.18164}, 
}