Code

Implementation of LakeFS#14

Code

Draft

jameshod5 wants to merge 34 commits intomainukaea/fair-mast-ingestion:mainfrom james/lakefsukaea/fair-mast-ingestion:james/lakefsCopy head branch name to clipboard

+127-64

Commits

Commits on Sep 3, 2024

• lakefs python code for branch creation, committing and merging

  jameshod5

  committed
  7cf606e

  Copy full SHA for 7cf606e

• initial test of lakefs workflow

  jameshod5

  committed
  e0b0fe4

  Copy full SHA for e0b0fe4

Commits on Sep 5, 2024

• added crude versioning to local ingestion

  

  James Hodson

  committed
  eb69a9d

  Copy full SHA for eb69a9d

• data versioning task added

  

  James Hodson

  committed
  06889bf

  Copy full SHA for 06889bf

• removed config requirements, reads from lakectl config now

  

  James Hodson

  committed
  1ae88df

  Copy full SHA for 1ae88df

Commits on Sep 6, 2024

• removed cleanup function and versioning from workflow file

  

  James Hodson

  committed
  aa87edd

  Copy full SHA for aa87edd

• improved versioning to get past mpi processes overlapping

  

  James Hodson

  committed
  27ffe0b

  Copy full SHA for 27ffe0b

• seperate lakefs versioning task, run external to ingestion

  

  James Hodson

  committed
  bb327df

  Copy full SHA for bb327df

Commits on Sep 9, 2024

• removed mpi versioning, now doing seperate versioning process

  

  James Hodson

  committed
  190fa49

  Copy full SHA for 190fa49

• added extra validation for data dir, lakectl install

  

  James Hodson

  committed
  0e0437f

  Copy full SHA for 0e0437f

• added cleanup functionality

  

  James Hodson

  committed
  781df1e

  Copy full SHA for 781df1e

• removed cleanup from workflow, moved to lake_fs.py

  

  James Hodson

  committed
  30831e8

  Copy full SHA for 30831e8

• removed unused versioning data task

  

  James Hodson

  committed
  d866f06

  Copy full SHA for d866f06

Commits on Sep 12, 2024

• create branch for versioning at beginning

  

  James Hodson

  committed
  fd80c08

  Copy full SHA for fd80c08

• commit shot after it is creating to branch ingestion

  

  James Hodson

  committed
  a3b586f

  Copy full SHA for a3b586f

• removed versioning from workflow

  

  James Hodson

  committed
  207074e

  Copy full SHA for 207074e

• working example of versioning, committing each shot at the end of their creation to the branch seperately

  

  James Hodson

  committed
  5cdaed6

  Copy full SHA for 5cdaed6

• organised versioning into functions

  

  James Hodson

  committed
  8613524

  Copy full SHA for 8613524

Commits on Sep 16, 2024

• added option to turn on or off versioning

  

  James Hodson

  committed
  a425809

  Copy full SHA for a425809

• removed args for versioning for now

  

  James Hodson

  committed
  2313959

  Copy full SHA for 2313959

Commits on Sep 17, 2024

• added merge and branch delete

  

  James Hodson

  committed
  42c32f8

  Copy full SHA for 42c32f8

• rework of ingestion workflow to use lakefs instead, does not work fully due to network problems with the lakefs instance. WIP

  

  James Hodson

  committed
  306f868

  Copy full SHA for 306f868

Commits on Sep 18, 2024

• added committing to the branch and cleanup to remove local files after the fact

  

  James Hodson

  committed
  7629c80

  Copy full SHA for 7629c80

• included merge at the end of versioning run

  

  James Hodson

  committed
  5190edd

  Copy full SHA for 5190edd

• upload and commit the shot name instead of the local_path prefix

  

  James Hodson

  committed
  885508c

  Copy full SHA for 885508c

• WIP file. Location of merge and execute functions used in main.

  

  James Hodson

  committed
  ade3c38

  Copy full SHA for ade3c38

• now able to use --upload arg as an option to enable lakefs versioning, requires a repo name

  

  James Hodson

  committed
  a270e28

  Copy full SHA for a270e28

• added branch to config to enable merging of ingestion branch into main at the end

  

  James Hodson

  committed
  950f186

  Copy full SHA for 950f186

• removed excess class

  

  James Hodson

  committed
  d813606

  Copy full SHA for d813606

Commits on Sep 20, 2024

• better info messages

  

  James Hodson

  committed
  5a8919e

  Copy full SHA for 5a8919e

• removed logging info

  

  James Hodson

  committed
  239e971

  Copy full SHA for 239e971

• ruff fixes

  

  James Hodson

  committed
  426e6f9

  Copy full SHA for 426e6f9

• ruff fixes

  

  James Hodson

  committed
  5ea3f84

  Copy full SHA for 5ea3f84

Commits on Oct 7, 2024

• changed job to work with lakefs command

  

  James Hodson

  committed
  fe93dce

  Copy full SHA for fe93dce