Code

Add level2 ingestion into repo#52

Code

Open

samueljackson92 wants to merge 125 commits intomainukaea/fair-mast-ingestion:mainfrom 51-level2-ingestionukaea/fair-mast-ingestion:51-level2-ingestionCopy head branch name to clipboard

+149,473-112,903

Commits

Commits on Dec 11, 2024

• Add new loader code

  samueljackson92

  committed
  5b15ed5

  Copy full SHA for 5b15ed5

• Fix ruff

  samueljackson92

  committed
  7178a9a

  Copy full SHA for 7178a9a

• Update tests to conditionally skip

  samueljackson92

  committed
  76ed719

  Copy full SHA for 76ed719

• Fix ruff

  samueljackson92

  committed
  3184d9b

  Copy full SHA for 3184d9b

• Update tests

  samueljackson92

  committed
  185e8ce

  Copy full SHA for 185e8ce

Commits on Dec 13, 2024

• revert to ignoring tests in CI until mastcodes is ready.

  samueljackson92

  committed
  df8462e

  Copy full SHA for df8462e

• Merge branch 'main' into 34-updated-reader

  samueljackson92

  committed
  01ba84f

  Copy full SHA for 01ba84f

• Skip more tests

  samueljackson92

  committed
  0b23202

  Copy full SHA for 0b23202

• Refactor cli and workflow code

  samueljackson92

  committed
  d32af1c

  Copy full SHA for d32af1c

• refactor pipelines

  samueljackson92

  committed
  0aa1f9a

  Copy full SHA for 0aa1f9a

• Update processing for amc

  samueljackson92

  committed
  763278a

  Copy full SHA for 763278a

• Add fix for names

  samueljackson92

  committed
  c59d689

  Copy full SHA for c59d689

Commits on Dec 16, 2024

• Update renaming dims

  samueljackson92

  committed
  b5207d6

  Copy full SHA for b5207d6

• Update unit mappings

  samueljackson92

  committed
  d6b5e21

  Copy full SHA for d6b5e21

Commits on Dec 17, 2024

• Update workflow code

  samueljackson92

  committed
  9bb6b53

  Copy full SHA for 9bb6b53

• Ruff formatting

  samueljackson92

  committed
  a05baa5

  Copy full SHA for a05baa5

• Update requirements.txt

  samueljackson92

  committed
  200731b

  Copy full SHA for 200731b

• Update mappings

  samueljackson92

  committed
  81d257f

  Copy full SHA for 81d257f

• Update CI

  samueljackson92

  committed
  88d06bc

  Copy full SHA for 88d06bc

• Update tests

  samueljackson92

  committed
  f1d221d

  Copy full SHA for f1d221d

• fix ruff

  samueljackson92

  committed
  3754551

  Copy full SHA for 3754551

Commits on Dec 18, 2024

• Update workflow to upload to s3

  samueljackson92

  committed
  82238e8

  Copy full SHA for 82238e8

• remove unused file

  samueljackson92

  committed
  cbe659f

  Copy full SHA for cbe659f

• Add code to rename groups

  samueljackson92

  committed
  51692ca

  Copy full SHA for 51692ca

• Tidy up name mappings

  samueljackson92

  committed
  418cd84

  Copy full SHA for 418cd84

• Update mappings and pipelines for better harmony

  samueljackson92

  committed
  b44d0a7

  Copy full SHA for b44d0a7

• Add unit test

  samueljackson92

  committed
  255360c

  Copy full SHA for 255360c

• Fix ruff

  samueljackson92

  committed
  9b16462

  Copy full SHA for 9b16462

• Fix historical mappings

  samueljackson92

  committed
  b42f063

  Copy full SHA for b42f063

• Add bes

  samueljackson92

  committed
  8c5b7ba

  Copy full SHA for 8c5b7ba

• Fix tests

  samueljackson92

  committed
  c1f69db

  Copy full SHA for c1f69db

Commits on Jan 2, 2025

• Update install instructions

  samueljackson92

  committed
  ff49aa9

  Copy full SHA for ff49aa9

• Checkpoint work on mappings

  samueljackson92

  committed
  5e5d558

  Copy full SHA for 5e5d558

• Update mappings and trasforms for MAST-U

  samueljackson92

  committed
  c2ed67b

  Copy full SHA for c2ed67b

• remove debug code

  samueljackson92

  committed
  729db7f

  Copy full SHA for 729db7f

• Add support for writing metadatabase

  samueljackson92

  committed
  d99ae3f

  Copy full SHA for d99ae3f

• Fix ruff

  samueljackson92

  committed
  868b043

  Copy full SHA for 868b043

• Fix rank,shape,dims

  Show description for 6ad7669
  samueljackson92

  committed
  6ad7669

  Copy full SHA for 6ad7669

Commits on Jan 3, 2025

• Refactor code structure to support level2

  samueljackson92

  committed
  da4d182

  Copy full SHA for da4d182

• Add level2 code to repo

  samueljackson92

  committed
  246b7ad

  Copy full SHA for 246b7ad

• Add notebooks for checking results

  samueljackson92

  committed
  8ac18f2

  Copy full SHA for 8ac18f2

• Update tests

  samueljackson92

  committed
  48cf3ec

  Copy full SHA for 48cf3ec

• Fix ruff

  samueljackson92

  committed
  cc2df7f

  Copy full SHA for cc2df7f

• Tidy up

  samueljackson92

  committed
  422d11a

  Copy full SHA for 422d11a

• Update gitignore

  samueljackson92

  committed
  27ab443

  Copy full SHA for 27ab443

• Update mappings location

  samueljackson92

  committed
  0265bb7

  Copy full SHA for 0265bb7

• remove unused geom files

  samueljackson92

  committed
  d72af12

  Copy full SHA for d72af12

• Update installation instructions

  samueljackson92

  committed
  b7dd57f

  Copy full SHA for b7dd57f

• Update requirements

  samueljackson92

  committed
  7b376cd

  Copy full SHA for 7b376cd

• Ignore notebook dir in ruff

  samueljackson92

  committed
  0db175e

  Copy full SHA for 0db175e

• Update reqs

  samueljackson92

  committed
  7bf512e

  Copy full SHA for 7bf512e

• Update reqs

  samueljackson92

  committed
  6bdd7af

  Copy full SHA for 6bdd7af

• refactor parallel code

  samueljackson92

  committed
  9707bfd

  Copy full SHA for 9707bfd

• remove code

  samueljackson92

  committed
  56a9a04

  Copy full SHA for 56a9a04

• Ingest in reverse

  samueljackson92

  committed
  18133a9

  Copy full SHA for 18133a9

• Update readme

  samueljackson92

  committed
  ff3a76a

  Copy full SHA for ff3a76a

• Update job scripts

  samueljackson92

  committed
  571027f

  Copy full SHA for 571027f

• Merge branch '51-level2-ingestion' of github.com:ukaea/fair-mast-ingestion into 51-level2-ingestion

  samueljackson92

  committed
  a9fd976

  Copy full SHA for a9fd976

• remove old job scripts

  samueljackson92

  committed
  17abe04

  Copy full SHA for 17abe04

• Add level2 job file

  samueljackson92

  committed
  7aa1377

  Copy full SHA for 7aa1377

• Update to include bottleneck

  samueljackson92

  committed
  4817bf1

  Copy full SHA for 4817bf1

• Update FFT transform

  samueljackson92

  committed
  2e8e18c

  Copy full SHA for 2e8e18c

• Merge branch '51-level2-ingestion' of github.com:ukaea/fair-mast-ingestion into 51-level2-ingestion

  samueljackson92

  committed
  613c1fc

  Copy full SHA for 613c1fc

• add sensible metadata defaults

  samueljackson92

  committed
  086446f

  Copy full SHA for 086446f

• Update the uploader

  samueljackson92

  committed
  76c5c10

  Copy full SHA for 76c5c10

• Fix camera data

  samueljackson92

  committed
  2f8f479

  Copy full SHA for 2f8f479

• Merge branch '51-level2-ingestion' of github.com:ukaea/fair-mast-ingestion into 51-level2-ingestion

  samueljackson92

  committed
  eb0c09c

  Copy full SHA for eb0c09c

• Update to fix subtle bug

  samueljackson92

  committed
  e65ff02

  Copy full SHA for e65ff02

• Fix issue with adding geometry

  samueljackson92

  committed
  e58e2af

  Copy full SHA for e58e2af

• fix geometry names

  samueljackson92

  committed
  1befa0f

  Copy full SHA for 1befa0f

Commits on Jan 6, 2025

• Fix coordinate bug

  samueljackson92

  committed
  9c0c752

  Copy full SHA for 9c0c752

• fix upload syntax

  samueljackson92

  committed
  0bbc992

  Copy full SHA for 0bbc992

• Fix esm mappings

  samueljackson92

  committed
  0fac9bb

  Copy full SHA for 0fac9bb

• Remove debug statement

  samueljackson92

  committed
  ac1b207

  Copy full SHA for ac1b207

• Update metadata parsing script

  samueljackson92

  committed
  e57eada

  Copy full SHA for e57eada

Commits on Jan 8, 2025

• Update level2 mappings for MAST to IMAS

  samueljackson92

  committed
  4c561bf

  Copy full SHA for 4c561bf

Commits on Jan 14, 2025

• Update mappings

  samueljackson92

  committed
  b1d9b48

  Copy full SHA for b1d9b48

• Add connection reset

  samueljackson92

  committed
  4c7d246

  Copy full SHA for 4c7d246

• Add error handling

  samueljackson92

  committed
  3f126a3

  Copy full SHA for 3f126a3

Commits on Jan 15, 2025

• Add more robust code for l1

  samueljackson92

  committed
  3a1bf89

  Copy full SHA for 3a1bf89

• Fix imports

  samueljackson92

  committed
  8a8bb79

  Copy full SHA for 8a8bb79

• Update logger message

  samueljackson92

  committed
  b2cb3b7

  Copy full SHA for b2cb3b7

• Update mappings for ALP

  samueljackson92

  committed
  5b5471e

  Copy full SHA for 5b5471e

• Add better exception handling

  samueljackson92

  committed
  754b573

  Copy full SHA for 754b573

• Force upload to use strings

  samueljackson92

  committed
  e02a6ff

  Copy full SHA for e02a6ff

• Update error logging

  samueljackson92

  committed
  34e21bd

  Copy full SHA for 34e21bd

• Add more robust code

  samueljackson92

  committed
  74630ba

  Copy full SHA for 74630ba

• Update mappings

  samueljackson92

  committed
  d8bb8bf

  Copy full SHA for d8bb8bf

• Update mappings

  samueljackson92

  committed
  ee3f86b

  Copy full SHA for ee3f86b

• Fix race condition

  samueljackson92

  committed
  171219e

  Copy full SHA for 171219e

• Make more robust

  samueljackson92

  committed
  8f0cc8f

  Copy full SHA for 8f0cc8f

• Fix edge case with magnetics

  samueljackson92

  committed
  42a4bdb

  Copy full SHA for 42a4bdb

• Update ruff and tests

  samueljackson92

  committed
  6ed7f48

  Copy full SHA for 6ed7f48

• Update scripts

  samueljackson92

  committed
  b269bc4

  Copy full SHA for b269bc4

• Add fix mast mapping

  samueljackson92

  committed
  cd79622

  Copy full SHA for cd79622

Commits on Jan 16, 2025

• Fix scaling for plasma current

  samueljackson92

  committed
  2e91aee

  Copy full SHA for 2e91aee

Commits on Jan 20, 2025

• Add checkpointing

  samueljackson92

  committed
  36ccf95

  Copy full SHA for 36ccf95

Commits on Jan 23, 2025

• Add upload

  samueljackson92

  committed
  a4679e5

  Copy full SHA for a4679e5

Commits on Jan 24, 2025

• Add updated mappings

  samueljackson92

  committed
  fce53fa

  Copy full SHA for fce53fa

Commits on Jan 27, 2025

• Update logging

  samueljackson92

  committed
  98a217b

  Copy full SHA for 98a217b

• Comment out some magnetics

  samueljackson92

  committed
  926e773

  Copy full SHA for 926e773

Commits on Jan 28, 2025

• Add metadata parsing script

  samueljackson92

  committed
  f391d17

  Copy full SHA for f391d17

• Update for ruff

  samueljackson92

  committed
  9f5cdcd

  Copy full SHA for 9f5cdcd

Commits on Jan 29, 2025

• Update channel loading to use NaN for missing channels

  samueljackson92

  committed
  7833124

  Copy full SHA for 7833124

• Add bes into mappings

  samueljackson92

  committed
  02eeb58

  Copy full SHA for 02eeb58

• Update target units

  samueljackson92

  committed
  afa8fee

  Copy full SHA for afa8fee

• fixed typo in mastcodes git link

  jameshod5

  committed
  f5d5c09

  Copy full SHA for f5d5c09

• Checkpoint mappings

  samueljackson92

  committed
  0c60f39

  Copy full SHA for 0c60f39

• Checkpoint scripts

  samueljackson92

  committed
  7de80e7

  Copy full SHA for 7de80e7

• Checkpoint jobs

  samueljackson92

  committed
  99a2d41

  Copy full SHA for 99a2d41

• Checkpoint configs

  samueljackson92

  committed
  ff174f2

  Copy full SHA for ff174f2

• Update gitignore

  samueljackson92

  committed
  83418c8

  Copy full SHA for 83418c8

• Checkpoint notebook

  samueljackson92

  committed
  02a9be5

  Copy full SHA for 02a9be5

Commits on Jan 30, 2025

• Update consolidate job for freia

  samueljackson92

  committed
  ea868ad

  Copy full SHA for ea868ad

• Update the code to remove metadata writing

  samueljackson92

  committed
  bd6cd47

  Copy full SHA for bd6cd47

• Update metadata jobs

  samueljackson92

  committed
  6b3ef38

  Copy full SHA for 6b3ef38

• Merge branch '51-level2-ingestion' of github.com:ukaea/fair-mast-ingestion into 51-level2-ingestion

  samueljackson92

  committed
  18eac4c

  Copy full SHA for 18eac4c

Commits on Jan 31, 2025

• Remove meta data writing

  samueljackson92

  committed
  9ffe577

  Copy full SHA for 9ffe577

• Update for ruff

  samueljackson92

  committed
  9759268

  Copy full SHA for 9759268

• Speed-up cpf parsing

  samueljackson92

  committed
  ae1fc5d

  Copy full SHA for ae1fc5d

• Update for ruff

  samueljackson92

  committed
  ab66c57

  Copy full SHA for ab66c57

Commits on Feb 3, 2025

• Update metadata jobs

  samueljackson92

  committed
  f683523

  Copy full SHA for f683523

Commits on Feb 7, 2025

• Update mapping files

  samueljackson92

  committed
  ecdc3cf

  Copy full SHA for ecdc3cf

Commits on Feb 18, 2025

• Update code with channel ordering

  samueljackson92

  committed
  9eceb01

  Copy full SHA for 9eceb01

• Update mappings

  samueljackson92

  committed
  40a11ff

  Copy full SHA for 40a11ff