Code

Develop#90

Code

Merged

alubneuski merged 118 commits intomasterd3b-center/image-deid-etl:masterfrom developd3b-center/image-deid-etl:developCopy head branch name to clipboardMar 22, 2024

+4,747

Commits

Commits on Mar 9, 2022

• Import from d3b-imaging-instance-ec2-prd-0 EC2 instance

  Show description for 1345fd4
  

  rbreslow

  and

  afamiliar

  committed
  1345fd4

  Copy full SHA for 1345fd4

Commits on Mar 23, 2022

• Create a reproducible and portable development environment

  rbreslow

  committed
  fc60373

  Copy full SHA for fc60373

• Add flake.lock file

  rbreslow

  committed
  6264583

  Copy full SHA for 6264583

Commits on Mar 24, 2022

• Add program and site args to scripts/test

  rbreslow

  committed
  fe8fb86

  Copy full SHA for fe8fb86

Commits on Mar 25, 2022

• README: clarify everyday usage

  rbreslow

  committed
  7355b6c

  Copy full SHA for 7355b6c

Commits on Mar 29, 2022

• README: clarify the need to refresh AWS credentials

  rbreslow

  committed
  3bc17e4

  Copy full SHA for 3bc17e4

Commits on Mar 31, 2022

• Merge pull request #1 from d3b-center/feature/rb/development-environment

  Show description for 6c4133a
  afamiliar

  authored
  6c4133a

  Copy full SHA for 6c4133a

Commits on Apr 4, 2022

• Add pull request template

  rbreslow

  committed
  fdda2e3

  Copy full SHA for fdda2e3

Commits on Apr 5, 2022

• Store processed UUIDs with SQLAlchemy

  rbreslow

  committed
  9e1f308

  Copy full SHA for 9e1f308

• Add documentation on contributing

  rbreslow

  committed
  2f9ca6f

  Copy full SHA for 2f9ca6f

• Merge pull request #13 from d3b-center/feature/rb/repo-templates

  Show description for d1ed436
  rbreslow

  authored
  d1ed436

  Copy full SHA for d1ed436

• Merge pull request #12 from d3b-center/feature/rb/sql

  Show description for 225f7d2
  rbreslow

  authored
  225f7d2

  Copy full SHA for 225f7d2

Commits on Apr 14, 2022

• Add ability to mark UUIDs as processed

  Show description for a0c7a0d
  rbreslow

  committed
  a0c7a0d

  Copy full SHA for a0c7a0d

• Change cbtn-all source

  Show description for 70ab45c
  

  afamiliar

  authored and

  rbreslow

  committed
  70ab45c

  Copy full SHA for 70ab45c

• Merge pull request #14 from d3b-center/af/cbtn-all-source

  Show description for e04bb5f
  

  afamiliar

  authored and

  rbreslow

  committed
  e04bb5f

  Copy full SHA for e04bb5f

• Update to use fw ingest command for Flywheel upload

  Show description for 4387122
  

  afamiliar

  authored and

  rbreslow

  committed
  4387122

  Copy full SHA for 4387122

Commits on Apr 18, 2022

• Handle study description edge case

  Show description for 1b2bf2b
  

  afamiliar

  authored and

  rbreslow

  committed
  1b2bf2b

  Copy full SHA for 1b2bf2b

• Use the Flywheel CLI from Nixpkgs

  Show description for 5836d53
  rbreslow

  committed
  5836d53

  Copy full SHA for 5836d53

• Disable support for the deprecated fw import command

  rbreslow

  committed
  c5db489

  Copy full SHA for c5db489

Commits on Apr 19, 2022

• Merge pull request #15 from d3b-center/feature/af/fw-ingest

  Show description for 44e2dee
  afamiliar

  authored
  44e2dee

  Copy full SHA for 44e2dee

Commits on Apr 20, 2022

• Split up the functionality of run_pipeline

  Show description for def8735
  rbreslow

  committed
  def8735

  Copy full SHA for def8735

Commits on Apr 22, 2022

• Merge pull request #20 from d3b-center/feature/rb/split-pipeline-functionality

  Show description for 5cb7aa1
  rbreslow

  authored
  5cb7aa1

  Copy full SHA for 5cb7aa1

Commits on Apr 28, 2022

• Switch to a container-based development environment

  Show description for da845bf
  rbreslow

  committed
  da845bf

  Copy full SHA for da845bf

Commits on May 6, 2022

• Merge pull request #24 from d3b-center/feature/rb/containerize-the-app

  Show description for a6d33a6
  rbreslow

  authored
  a6d33a6

  Copy full SHA for a6d33a6

Commits on May 10, 2022

• Fix missing birthdays and mismatching study descriptions from DICOM tags

  

  afamiliar

  authored and

  rbreslow

  committed
  a621295

  Copy full SHA for a621295

• Account for different study descriptions due to special characters

  

  afamiliar

  authored and

  rbreslow

  committed
  d1ad0f2

  Copy full SHA for d1ad0f2

• Adding body-part string match (session_label)

  

  afamiliar

  authored and

  rbreslow

  committed
  769b774

  Copy full SHA for 769b774

• add special character handling for acquisition labels when injecting flywheel metadata after upload

  afamiliar

  committed
  ad2366d

  Copy full SHA for ad2366d

Commits on May 12, 2022

• Remove stray link from README ToC

  rbreslow

  committed
  73d0cb2

  Copy full SHA for 73d0cb2

• Add Terraform configuration for VPC and RDS

  Show description for 7b473a7
  rbreslow

  committed
  7b473a7

  Copy full SHA for 7b473a7

Commits on May 13, 2022

• Merge pull request #30 from d3b-center/feature/rb/vpc-and-rds

  Show description for bf5240e
  rbreslow

  authored
  bf5240e

  Copy full SHA for bf5240e

• Build and publish container image to Amazon ECR

  Show description for a05410d
  rbreslow

  committed
  a05410d

  Copy full SHA for a05410d

• Merge pull request #31 from d3b-center/feature/rb/publish-container-image

  Show description for af3a49b
  rbreslow

  authored
  af3a49b

  Copy full SHA for af3a49b

Commits on May 16, 2022

• Change try-except to if-else in dob handling

  afamiliar

  committed
  e21be45

  Copy full SHA for e21be45

• Moved orthanc-related lines within corresponding function

  afamiliar

  committed
  c4ea925

  Copy full SHA for c4ea925

• Merge pull request #29 from d3b-center/feature/af/missing-metadata

  Show description for 55beb9f
  afamiliar

  authored
  55beb9f

  Copy full SHA for 55beb9f

Commits on May 17, 2022

• Add Terraform configuration for AWS Batch

  Show description for a1ed59f
  rbreslow

  committed
  a1ed59f

  Copy full SHA for a1ed59f

• Write deployment documentation

  rbreslow

  committed
  bdcfc73

  Copy full SHA for bdcfc73

• Add flag to CLI that offloads processing to Batch

  rbreslow

  committed
  4b002fd

  Copy full SHA for 4b002fd

• Read subject ID mapping via Pandas API

  Show description for 5c27460
  rbreslow

  committed
  5c27460

  Copy full SHA for 5c27460

Commits on May 21, 2022

• Merge pull request #34 from d3b-center/feature/rb/batch

  Show description for 27e175a
  rbreslow

  authored
  27e175a

  Copy full SHA for 27e175a

Commits on Jun 6, 2022

• Use Rollbar for error tracking and notification

  Show description for cd25b36
  rbreslow

  committed
  cd25b36

  Copy full SHA for cd25b36

• Include study UUIDs in Rollbar payload

  rbreslow

  committed
  fd68034

  Copy full SHA for fd68034

Commits on Jun 13, 2022

• Merge pull request #41 from d3b-center/feature/rb/rollbar

  Show description for 1b8c5de
  rbreslow

  authored
  1b8c5de

  Copy full SHA for 1b8c5de

Commits on Jun 15, 2022

• Exit if there are no valid DICOMs found

  afamiliar

  committed
  0c442f0

  Copy full SHA for 0c442f0

Commits on Jun 16, 2022

• use series-number in inject_sidecar_metadata

  afamiliar

  committed
  a4273ad

  Copy full SHA for a4273ad

• Change error description bc too specific

  afamiliar

  committed
  b36020b

  Copy full SHA for b36020b

• Merge pull request #44 from d3b-center/feature/af/exit-if-no-dicoms

  Show description for 8941d00
  afamiliar

  authored
  8941d00

  Copy full SHA for 8941d00

Commits on Jun 20, 2022

• Update diagnosis_mapping.json

  Show description for be7f6a3
  afamiliar

  committed
  be7f6a3

  Copy full SHA for be7f6a3

• Add fuzzywuzzy to requirements.txt

  afamiliar

  committed
  eb0cc0a

  Copy full SHA for eb0cc0a

• Change to matching SeriesDesc not SeriesNum

  afamiliar

  committed
  62bf5dd

  Copy full SHA for 62bf5dd

Commits on Jun 21, 2022

• Show Terraform warning messages in compact form

  rbreslow

  committed
  c30fe21

  Copy full SHA for c30fe21

• Revert default_tags and tag all resources by-hand

  Show description for 5d6585f
  rbreslow

  committed
  5d6585f

  Copy full SHA for 5d6585f

• Set Batch job platform_capabilities

  Show description for 5725ce2
  rbreslow

  committed
  5725ce2

  Copy full SHA for 5725ce2

Commits on Jun 22, 2022

• Merge pull request #47 from d3b-center/feature/rb/revert-default-tags

  Show description for 76e57a2
  rbreslow

  authored
  76e57a2

  Copy full SHA for 76e57a2

• Add input directory for local CSV, use both series-desc and series-num for matching

  afamiliar

  committed
  4587e46

  Copy full SHA for 4587e46

Commits on Jun 24, 2022

• Add GitHub Actions CI workflow (#50)

  Show description for cbf5f42
  rbreslow

  authored
  cbf5f42

  Copy full SHA for cbf5f42

• Add DNS zone for ETL-related services

  rbreslow

  committed
  54c44c2

  Copy full SHA for 54c44c2

• Draft documentation for tunneling to RDS

  rbreslow

  committed
  48b0c06

  Copy full SHA for 48b0c06

Commits on Aug 5, 2022

• Add flag to disable audit logs in fw ingest folder

  afamiliar

  committed
  e2b7feb

  Copy full SHA for e2b7feb

Commits on Sep 23, 2022

• License repository under Apache 2.0 (#61)

  Show description for 88d8361
  rbreslow

  authored
  88d8361

  Copy full SHA for 88d8361

• Document how we define our database schema

  rbreslow

  committed
  c3af675

  Copy full SHA for c3af675

• Merge pull request #51 from d3b-center/feature/rb/tunneling

  Show description for fb9acb5
  rbreslow

  authored
  fb9acb5

  Copy full SHA for fb9acb5

• Add Step Functions workflow that wraps the ETL

  Show description for 9c9b530
  rbreslow

  committed
  9c9b530

  Copy full SHA for 9c9b530

Commits on Sep 26, 2022

• Add workflow trigger for manual deployment

  afamiliar

  committed
  58d4f10

  Copy full SHA for 58d4f10

• Merge pull request #62 from d3b-center/feature/af/add-new-workflow

  Show description for 71ed988
  afamiliar

  authored
  71ed988

  Copy full SHA for 71ed988

• Add fuzzywuzzy package requirement

  afamiliar

  committed
  275d7c8

  Copy full SHA for 275d7c8

• Moved package import to top of file

  afamiliar

  committed
  22d4f14

  Copy full SHA for 22d4f14

• Removed commented lines in custom_flywheel.py

  afamiliar

  committed
  5def783

  Copy full SHA for 5def783

Commits on Sep 27, 2022

• get Flywheel path from directory names, instead of sub-mapping CSV

  afamiliar

  committed
  154009e

  Copy full SHA for 154009e

• Merge pull request #57 from d3b-center/feature/af/acq-label-mismatch

  Show description for 3326b1c
  afamiliar

  authored
  3326b1c

  Copy full SHA for 3326b1c

• Merge pull request #58 from d3b-center/feature/af/disable-audit-logs

  Show description for c1ce2b8
  afamiliar

  authored
  c1ce2b8

  Copy full SHA for c1ce2b8

• force format of DOB column with pandas

  afamiliar

  committed
  cd9ace0

  Copy full SHA for cd9ace0

• Merge pull request #65 from d3b-center/bug/af/dob-format-handling

  Show description for 37f6661
  afamiliar

  authored
  37f6661

  Copy full SHA for 37f6661

• add 'knee' to body part string matching

  afamiliar

  committed
  f7f0e7f

  Copy full SHA for f7f0e7f

• add error handling to tease apart missing session/subject information

  afamiliar

  committed
  c2f8c50

  Copy full SHA for c2f8c50

• Merge pull request #66 from d3b-center/bug/af/update-anatomy-mapping

  Show description for 72ec28a
  afamiliar

  authored
  72ec28a

  Copy full SHA for 72ec28a

• remove leading zeros from cbtn-all MRN column

  afamiliar

  committed
  e204018

  Copy full SHA for e204018

• update error types to be appropriate for given case

  afamiliar

  committed
  ab4f681

  Copy full SHA for ab4f681

• assign variables regardless of 'validate' mode

  Show description for 29f88bd
  afamiliar

  committed
  29f88bd

  Copy full SHA for 29f88bd

• exit processing at error

  afamiliar

  committed
  6a9e453

  Copy full SHA for 6a9e453

• Merge pull request #67 from d3b-center/bug/af/pandas-merge-error

  Show description for 810ac5c
  afamiliar

  authored
  810ac5c

  Copy full SHA for 810ac5c

• Merge pull request #68 from d3b-center/bug/af/handle-mrns-leading-zeros

  Show description for dad0328
  afamiliar

  authored
  dad0328

  Copy full SHA for dad0328

Commits on Sep 28, 2022

• exit upon error - missing cbtn-all file

  afamiliar

  committed
  035297f

  Copy full SHA for 035297f

• handle sessions with no DICOMs to process, add acq modalities to delete

  afamiliar

  committed
  8b47598

  Copy full SHA for 8b47598

• missing subject mapping error after file conversion

  afamiliar

  committed
  c4bb107

  Copy full SHA for c4bb107

• Merge pull request #69 from d3b-center/feature/af/handle-invalid-sessions

  Show description for 5279e6e
  afamiliar

  authored
  5279e6e

  Copy full SHA for 5279e6e

• Merge branch 'develop' into feature/af/update-error-info

  afamiliar

  authored
  da52511

  Copy full SHA for da52511

• Merge pull request #70 from d3b-center/feature/af/update-error-info

  Show description for 64be871
  afamiliar

  authored
  64be871

  Copy full SHA for 64be871

• adding more descriptive error for invalid study uuid

  afamiliar

  committed
  bb09c4d

  Copy full SHA for bb09c4d

• add comments designating modality type

  afamiliar

  committed
  cfe1208

  Copy full SHA for cfe1208

Commits on Sep 29, 2022

• add libdcm-tools to dockerfile, and add comments

  afamiliar

  committed
  3073ac9

  Copy full SHA for 3073ac9

Commits on Sep 30, 2022

• use gdcmconv on broken DICOMs (e.g., JPEG2000 errors) before dcm2niix convert

  afamiliar

  committed
  cdb499f

  Copy full SHA for cdb499f

• add comments on gdcmconv

  afamiliar

  committed
  91f6c6e

  Copy full SHA for 91f6c6e

• Switch to new stable version of dcm2niix

  afamiliar

  committed
  bfcf2cd

  Copy full SHA for bfcf2cd

• Merge pull request #72 from d3b-center/feature/af/fix-jpeg2000-encoding

  Show description for 89b955c
  afamiliar

  authored
  89b955c

  Copy full SHA for 89b955c

• add study-time from DICOM tag to fw session label

  afamiliar

  committed
  1685654

  Copy full SHA for 1685654

• Merge pull request #73 from d3b-center/feature/af/add-acq-time-to-session-label

  Show description for ae04e21
  afamiliar

  authored
  ae04e21

  Copy full SHA for ae04e21

• check for niftis prior to fw upload, if none then exit

  afamiliar

  committed
  eba4cb0

  Copy full SHA for eba4cb0

• switch to checking for nifti files instead of acq-dirs

  afamiliar

  committed
  1a99737

  Copy full SHA for 1a99737

• Merge pull request #74 from d3b-center/feature/af/handle-invalid-niftis

  Show description for e020f2d
  afamiliar

  authored
  e020f2d

  Copy full SHA for e020f2d

Commits on Oct 5, 2022

• add functionality for dataset version numbers (fw project label)

  afamiliar

  committed
  6d6d967

  Copy full SHA for 6d6d967

• Merge pull request #75 from d3b-center/feature/af/add-fw-proj-version

  Show description for 4a10817
  afamiliar

  authored
  4a10817

  Copy full SHA for 4a10817

• Merge pull request #76 from d3b-center/feature/af/add-orthanc-error-handling

  Show description for 0a38286
  afamiliar

  authored
  0a38286

  Copy full SHA for 0a38286

• Create continuous_integration_manual.yml

  afamiliar

  committed
  67a951e

  Copy full SHA for 67a951e

• Merge pull request #77 from d3b-center/feature/af/add-github-workflow

  Show description for 6a292f2
  afamiliar

  authored
  6a292f2

  Copy full SHA for 6a292f2

• Update continuous_integration_manual.yml

  afamiliar

  committed
  64491a9

  Copy full SHA for 64491a9

• Update custom_flywheel.py

  afamiliar

  committed
  1555852

  Copy full SHA for 1555852

• Merge pull request #78 from d3b-center/bug/af/fix-dictionary-key

  Show description for af46696
  afamiliar

  authored
  af46696

  Copy full SHA for af46696

Commits on Nov 11, 2022

• Merge pull request #53 from d3b-center/feature/rb/step-functions

  Show description for 0f0626b
  alubneuski

  authored
  0f0626b

  Copy full SHA for 0f0626b

Commits on Nov 15, 2022

• remove dependency on 'accession number' field

  Show description for da560ae
  afamiliar

  committed
  da560ae

  Copy full SHA for da560ae

• Merge pull request #83 from d3b-center/remove-accession-num-dependency

  Show description for f4793dd
  afamiliar

  authored
  f4793dd

  Copy full SHA for f4793dd

Commits on Dec 28, 2022

• 🔧 Added scheduled invokation for imaging workflow

  

  Alex Lubneuski

  committed
  555ed89

  Copy full SHA for 555ed89

• 🔧 Added IAM event bus role

  

  Alex Lubneuski

  committed
  0c58175

  Copy full SHA for 0c58175

Commits on Mar 11, 2024

• 📌 Update postgresql from 14.2 -> 14.11

  devbyaccident

  committed
  5b47577

  Copy full SHA for 5b47577

• Merge pull request #88 from d3b-center/chore/cbl/postgres-minor-version

  Show description for 955e496
  devbyaccident

  authored
  955e496

  Copy full SHA for 955e496

• 📌 Update rollbar/github-deploy-action 2.1.1 -> 2.1.2

  devbyaccident

  committed
  b35c4ea

  Copy full SHA for b35c4ea

Commits on Mar 20, 2024

• Merge pull request #89 from d3b-center/chore/cbl/update-rollbar-action

  Show description for e7f4837
  alubneuski

  authored
  e7f4837

  Copy full SHA for e7f4837