Shuffle Tokenized Data

[mali-git]

What does this PR do?

This PR introduces functionality to shuffle both the data and index segments in a packed file format. The primary goal is to eliminate the reliance on random reads during later stages when the data is accessed via mmap.

By shuffling the index and aligning it with a shuffled byte stream of the data, this implementation ensures that data can be read sequentially.

General Changes

The shuffle_tokenized_data function loads the input data and index into memory, shuffles the index entries, and processes them in batches to recduce the memory footprint.

Breaking Changes

None.

Checklist before submitting final PR

• My PR is minimal and addresses one issue in isolation
• I have merged the latest version of the target branch into this feature branch
• I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
• I have run a sample config for model training
• I have checked that all tests run through (python tests/tests.py) - Please see Tokenization Test Fails #293
• I have updated the internal changelog (CHANGELOG_DEV.md)

[le1nux]

I gave it a high-level review and I think we should discuss if we really need this multiprocessing setup.
Once decided, I will also look into reading and writing of the pbin files.

[fromm-m]

LGTM

[le1nux]

LGTM :) Nice work!
Left minor comments but nothing major.
Almost mergeable!