Code

"Golden spike" PR#488

Code

Draft

knighton wants to merge 47 commits intomainmosaicml/streaming:mainfrom james/protomosaicml/streaming:james/protoCopy head branch name to clipboard

+3,730-2,701

Commits

Commits on Oct 28, 2023

• Stub out index_delta(), index_lance(), index_parquet().

  knighton

  committed
  a859b58

  Copy full SHA for a859b58

• index_backend().

  knighton

  committed
  bd0208a

  Copy full SHA for bd0208a

• Fix.

  knighton

  committed
  69575bc

  Copy full SHA for 69575bc

• task.py for benchmarking.

  knighton

  committed
  42b59f1

  Copy full SHA for 42b59f1

• generate_datasets.py.

  knighton

  committed
  2fb1b09

  Copy full SHA for 2fb1b09

• Fix.

  knighton

  committed
  11dd673

  Copy full SHA for 11dd673

• Organize/divide streaming/base/util.py:

  Show description for 82737e0
  knighton

  committed
  82737e0

  Copy full SHA for 82737e0

• Completely rip out and rewrite pretty args handling:

  Show description for 3212f66
  knighton

  committed
  3212f66

  Copy full SHA for 3212f66

• Layer several new storage APIs wrapping/complementing streaming/base/storage/.

  Show description for eb93bea
  knighton

  committed
  eb93bea

  Copy full SHA for eb93bea

• Use those APIs to index a Parquet dataset (single-threaded).

  knighton

  committed
  23554ac

  Copy full SHA for 23554ac

• Add cli/index_parquet.py.

  knighton

  committed
  c711567

  Copy full SHA for c711567

• Rename get_list_arg() -> parse_strs() in keeping with parse_str2str(), etc.

  knighton

  committed
  4ea01b2

  Copy full SHA for 4ea01b2

• Rename parse_(args stuff) -> unpack_(args stuff).

  knighton

  committed
  157381a

  Copy full SHA for 157381a

• Long lines.

  knighton

  committed
  c72127f

  Copy full SHA for c72127f

• Populate streaming/examples/ with SD subclasses, also streaming/benchmarks/.

  knighton

  committed
  d2be6a0

  Copy full SHA for d2be6a0

• Fix.

  knighton

  committed
  da6f4af

  Copy full SHA for da6f4af

• Move benchmarks up and out.

  knighton

  committed
  b0fa3d7

  Copy full SHA for b0fa3d7

• Fix.

  knighton

  committed
  4a22638

  Copy full SHA for 4a22638

• Now, rename streaming/base/... -> streaming/....

  knighton

  committed
  cb80865

  Copy full SHA for cb80865

• Update paths accordingly.

  knighton

  committed
  4851888

  Copy full SHA for 4851888

• Update more paths.

  knighton

  committed
  1051474

  Copy full SHA for 1051474

• Formatting.

  knighton

  committed
  65ef0de

  Copy full SHA for 65ef0de

• Fix.

  knighton

  committed
  408999a

  Copy full SHA for 408999a

• Move examples/ to top level.

  knighton

  committed
  b38f8a3

  Copy full SHA for b38f8a3

• Update multimodal.

  knighton

  committed
  ff90826

  Copy full SHA for ff90826

• Update vision dataset sexamples -> kwargs.

  knighton

  committed
  a7808ae

  Copy full SHA for a7808ae

• Update vision datasets to use kwargs (save us from bitrot, o kwargs).

  knighton

  committed
  c857ed6

  Copy full SHA for c857ed6

• Generalize keep_zip argument to keep_packed.

  knighton

  committed
  89d5719

  Copy full SHA for 89d5719

• Add graceful migration from keep_zip to keep_packed.

  knighton

  committed
  c09248c

  Copy full SHA for c09248c

Commits on Oct 29, 2023

• First take on a MDS write_dataset().

  knighton

  committed
  9befaa6

  Copy full SHA for 9befaa6

• Add enough column inference to keep going.

  knighton

  committed
  c4a5094

  Copy full SHA for c4a5094

• WWriting all given samples as one indexless MDS shard, returning its metadata.

  knighton

  committed
  48dce5c

  Copy full SHA for 48dce5c

• Naming.

  knighton

  committed
  b0d1543

  Copy full SHA for b0d1543

• Fixes.

  knighton

  committed
  99ad0c0

  Copy full SHA for 99ad0c0

• cli/hash.py.

  knighton

  committed
  b38fce0

  Copy full SHA for b38fce0

• walk_prefix() including local fs.

  knighton

  committed
  7a9fc90

  Copy full SHA for 7a9fc90

Commits on Nov 5, 2023

• generate_datasets.py: Tabulator.

  knighton

  committed
  5247bfe

  Copy full SHA for 5247bfe

• Fix (passing left, and spacing).

  knighton

  committed
  6dc5e22

  Copy full SHA for 6dc5e22

• Switch to box-drawing chars in Tabulator. Example:

  Show description for 18f6474
  knighton

  committed
  18f6474

  Copy full SHA for 18f6474

• Rewrite task.py.

  knighton

  committed
  52af2cb

  Copy full SHA for 52af2cb

• Fixes.

  knighton

  committed
  bc125b4

  Copy full SHA for bc125b4

• Fix.

  knighton

  committed
  a2ff86f

  Copy full SHA for a2ff86f

• Misc.

  knighton

  committed
  57e7571

  Copy full SHA for 57e7571

• Merge branch 'main' into james/proto

  knighton

  authored
  52dcb42

  Copy full SHA for 52dcb42

• Split out Tabulator.

  knighton

  committed
  f1e10bb

  Copy full SHA for f1e10bb

• Merge branch 'james/proto' of github.com:mosaicml/streaming into james/proto

  knighton

  committed
  56674e8

  Copy full SHA for 56674e8

• Refactor.

  knighton

  committed
  cbfcab3

  Copy full SHA for cbfcab3