Code

Feature dataset checkpoint strategy#194

Code

Merged

plaguss merged 29 commits intomainargilla-io/distilabel:mainfrom feat/dataset-checkpointargilla-io/distilabel:feat/dataset-checkpointCopy head branch name to clipboardJan 8, 2024

+285-22

Commits

Commits on Dec 22, 2023

• Add save and load methods to disk

  plaguss

  committed
  d611cee

• Add DatasetCheckpoint to deal with saves to disk during generation pipeline

  plaguss

  committed
  cd00186

• Add tests for new dataset methods and do_checkpoint strategy

  plaguss

  committed
  952a437

• Add checkpoint strategy to pipeline

  plaguss

  committed
  710ccfa

• Fix error with typing in python 3.8

  plaguss

  committed
  f6718dc

• Merge branch 'main' of https://github.com/argilla-io/distilabel into feat/dataset-checkpoint

  plaguss

  committed
  eda97be

• Add type

  Show description for 769e20a
  

  plaguss

  and

  gabrielmbmb

  authored
  769e20a

• Add type

  Show description for 3487b06
  

  plaguss

  and

  gabrielmbmb

  authored
  3487b06

• Add type

  Show description for d4bda8a
  

  plaguss

  and

  gabrielmbmb

  authored
  d4bda8a

• Add extra test for do_checkpoint and move DatasetCheckpoint to distilabel.dataset

  plaguss

  committed
  0dae781

• Simplify code in pipeline as per code review

  plaguss

  committed
  5ff9180

• Merge incoming changes from github ui

  plaguss

  committed
  2435476

• Add task filename as a constant variable

  plaguss

  committed
  51f4792

• Fix import in pipeline

  plaguss

  committed
  b867f31

• Merge branch 'main' of https://github.com/argilla-io/distilabel into feat/dataset-checkpoint

  plaguss

  committed
  8a409ef

Commits on Jan 2, 2024

• Merge with main

  plaguss

  committed
  28b175b

Commits on Jan 3, 2024

• Use dill instead of pickle to simplify process with custom classes

  plaguss

  committed
  979e1f9

• Add extra check for shape of saved dataset

  plaguss

  committed
  9459b15

• Remove functionality of _build_dataset that prevented the generation of the full dataset

  plaguss

  committed
  44f732d

Commits on Jan 4, 2024

• Add documentatio for the checkpoint strategy

  plaguss

  committed
  d20ffa4

• Rename the default folder to standardize

  plaguss

  committed
  293610d

• Update src/distilabel/pipeline.py

  Show description for dc8603a
  

  plaguss

  and

  alvarobartt

  authored
  dc8603a

Commits on Jan 8, 2024

• Merge branch 'main' of https://github.com/argilla-io/distilabel into feat/dataset-checkpoint

  plaguss

  committed
  b834ca9

• Merge branch 'main' of https://github.com/argilla-io/distilabel into feat/dataset-checkpoint

  plaguss

  committed
  22b4cc4

• Merge with main and corrected typo

  plaguss

  committed
  95dac2c

• Update ultrafeedback from tests

  plaguss

  committed
  b8cc8cc

• Update with comments from the review

  plaguss

  committed
  8db52df

• Merge branch 'main' of https://github.com/argilla-io/distilabel into feat/dataset-checkpoint

  plaguss

  committed
  f8e8606

• Remove dead code

  plaguss

  committed
  d9ad890