Code

Clean#5

Code

Merged

Oufattole merged 122 commits intomainmmcdermott/MEDS_Tabular_AutoML:mainfrom cleanmmcdermott/MEDS_Tabular_AutoML:cleanCopy head branch name to clipboardJun 5, 2024

+3,583-55

Commits

Commits on May 25, 2024

• added test case and initial testing and and code for shardingoutput tabular data

  Oufattole

  committed
  d5ff0df

Commits on May 26, 2024

• added static feature pivoting

  Oufattole

  committed
  4a486aa

• added docstrings and a smiple test case checking the number of subjects is correct when producing static representations

  Oufattole

  committed
  19f0f4e

Commits on May 27, 2024

• Refactor scripts into separate modules for improved clarity:

  Show description for fd1731f
  Oufattole

  committed
  fd1731f

• fixed doctests and updated github workflow tests to use python 3.12

  Oufattole

  committed
  63b9ba6

• Implement data processing for MEDS format to pivot tables into two indexed file types:

  Show description for cd067f8
  Oufattole

  committed
  cd067f8

• Enhance data aggregation framework with dynamic window and aggregation handling

  Show description for c8ca3bb
  Oufattole

  committed
  c8ca3bb

• Update src/MEDS_tabular_automl/generate_summarized_reps.py

  Show description for 7fdc37d
  

  mmcdermott

  and

  coderabbitai[bot]

  authored
  7fdc37d

• Update src/MEDS_tabular_automl/generate_summarized_reps.py

  Show description for 4dd3cad
  

  mmcdermott

  and

  coderabbitai[bot]

  authored
  4dd3cad

• Update src/MEDS_tabular_automl/generate_summarized_reps.py

  Show description for 720a533
  

  mmcdermott

  and

  coderabbitai[bot]

  authored
  720a533

• Added doctest and updated docstrings in identiy_columns.py. [WIP] addressing comments in other files

  Oufattole

  committed
  548e29a

• Merge branch 'esgpt_caching' of github.com:mmcdermott/MEDS_Tabular_AutoML into esgpt_caching

  Oufattole

  committed
  d9ba7e7

Commits on May 28, 2024

• working on xgboost

  teyaberg

  committed
  f0b1cbb

• current state

  Oufattole

  committed
  ba954ef

• Removed tqdm, fixed deprecated groupbys, fixed doctest long-line issue.

  mmcdermott

  committed
  df2750a

• Fixed one of the summary doctests.

  mmcdermott

  committed
  4bbbc20

• updates based on formats... still many to dos

  teyaberg

  committed
  d39bf1a

Commits on May 29, 2024

• using sparse matrices for generating time series representations

  Oufattole

  committed
  41fe4b4

• still working on sparse matrix to external memory xgboost

  teyaberg

  committed
  cb5f689

• same problem

  teyaberg

  committed
  8bc9a16

• cleaned some testing

  teyaberg

  committed
  1e27526

• sped up the tabularize_ts script by about 30% by concatenating the sparse matrices instead of the sparse dataframes for values and codes. Also started modifying the summarization script to work wit…

  Show description for 97938a8
  Oufattole

  committed
  97938a8

• got iterator working with csr_matrices for X and numpy arrays for y

  teyaberg

  committed
  c28e6b2

• added support for sparse aggregations

  Oufattole

  committed
  6f3b1ec

• passing unit tests for sparse aggregations (only code/count and value/sum implemented at the moment)

  Oufattole

  committed
  6753609

• added significant speed improvements for rolling window aggregations

  Oufattole

  committed
  f125600

• improved speed, by removing conversion from sparse scipy matrix to sparse pandas array for each patient, now we just use sparse scipy matrices

  Oufattole

  committed
  2acc3bc

Commits on May 30, 2024

• takes about an hour to run through a shard. The speed gain is from merging rows that occur at the same time based on the current aggregation strategy. For example if the aggregation is sum, we sum …

  Show description for eec05e2
  Oufattole

  committed
  eec05e2

• added scripts to the readme

  Oufattole

  committed
  bd9bdae

• save before breaking it

  teyaberg

  committed
  29c8c5f

• added support for parallelism using mapper warp function. We cache feature frequencies at the beginning of the identify columns script as well. summarize over windows can be parallelized over aggre…

  Show description for 4c7d3e7
  Oufattole

  committed
  4c7d3e7

• wip

  teyaberg

  committed
  ba796e5

• automl

  teyaberg

  committed
  3678d30

• Merge branch 'esgpt_caching' into xgboost

  Oufattole

  committed
  82b3903

Commits on May 31, 2024

• working on collect_in_memory

  teyaberg

  committed
  ffa0f3c

• collect in memory fixed

  teyaberg

  committed
  c8f26ea

• added hf_cohort scripts

  

  Nassim Oufattole

  committed
  f6a3751

• Apply suggestions from code review

  Show description for 2ec1860
  

  mmcdermott

  and

  coderabbitai[bot]

  authored
  2ec1860

• cleaning

  teyaberg

  committed
  db18dc5

• local WIP--changing to sparse matrix implementation

  teyaberg

  committed
  abba3d2

• added merging of static and time series data

  

  Nassim Oufattole

  committed
  77f296f

• Merge branch 'esgpt_caching' of github.com:mmcdermott/MEDS_Tabular_AutoML into esgpt_caching

  

  Nassim Oufattole

  committed
  e8f26eb

• merging script runs, but the output is 50GB

  Oufattole

  committed
  958906d

• merging script works and is efficient

  Oufattole

  committed
  7668382

• fixed bug with sparse matrix shape being too small for merging static and time series dataframs

  Oufattole

  committed
  b6b8d43

Commits on Jun 1, 2024

• changed to sparse format

  teyaberg

  committed
  e6a88a7

• added script for extracting tasks using aces

  Oufattole

  committed
  e8d64fd

• merged xgboost code

  Oufattole

  committed
  5b2f7f7

• added dependencies

  Oufattole

  committed
  5c5dc8e

• Merge branch 'xgboost' into esgpt_caching

  Oufattole

  committed
  357845e

• added support for loading cached labels and event indexes

  Oufattole

  committed
  d99e274

• updated readme

  Oufattole

  committed
  cadc603

• size issues for loading sparse matrix

  teyaberg

  committed
  285ccbf

• push updates

  teyaberg

  committed
  795b532

• 4x speed increase for tabularization to sparse matrix by caching window sizes, using a csr_matrix for the input data, and using adding he aggregation outputs to lists as we iterate through windows …

  Show description for b9d057b
  Oufattole

  committed
  b9d057b

• Merge branch 'xgboost' into esgpt_caching

  Oufattole

  committed
  85f38b5

Commits on Jun 2, 2024

• standardized file storage using file_name.py and updated from using npy files for storing sparse matrices to using npz files that contain the array with data,row,cols and contain the shape of the s…

  Show description for 7ea3230
  Oufattole

  committed
  7ea3230

• cleaned up file paths so we can load all aggregations selectively and concatenate them. Now the get_feature_names and get_feature_indices functions will load the feature names and indices of the co…

  Show description for 23a2e3b
  Oufattole

  committed
  23a2e3b

• fixed bug with codes that are only in the test and validation set (not in the training set) crashing the window summarization. Now we throw those events out.

  

  Nassim Oufattole

  committed
  c225c47

• fixed bug with summarization script crashing for min and max value aggregations due to a coo matrix being returned rather than a dense matrix as with sum and count operations

  

  Nassim Oufattole

  committed
  cb21821

• removed overwrite killing of jobs which causes errors in multirun

  

  Nassim Oufattole

  committed
  3a412a0

• Xgboost is able to load all concatenated windows and aggregations. Fixed bugs related to event ids and column ids being incorrect.

  Oufattole

  committed
  a4f1843

• fixed timedelta overflow bug

  

  Nassim Oufattole

  committed
  800ab7e

• Merge branch 'esgpt_caching' of github.com:mmcdermott/MEDS_Tabular_AutoML into esgpt_caching

  

  Nassim Oufattole

  committed
  820e194

• fixed bug with loading feature columns json for aces task script

  

  Nassim Oufattole

  committed
  4b0637a

• added memory profiling to hf_cohort e2e script

  Oufattole

  committed
  127d04a

• Merge branch 'esgpt_caching' of github.com:mmcdermott/MEDS_Tabular_AutoML into esgpt_caching

  Oufattole

  committed
  23877ad

• Made tests ignore the hf_cohort directory

  mmcdermott

  committed
  36f54a3

• Pre-commit fixes

  mmcdermott

  committed
  81bf2d9

• Resolving deprecation warnings

  mmcdermott

  committed
  83c4eec

• Fixed test installation instructions.

  mmcdermott

  committed
  e7a85ba

• Merge branch 'esgpt_caching' into mmd_changes

  mmcdermott

  committed
  35acb97

• Resolved one error (or, rather, shifted it) by making some things properties, but don't know what the desired behavior is for the error

  mmcdermott

  committed
  bef63b6

• Shifted more test errors around, but the failures are deeper than expected.

  mmcdermott

  committed
  e9775e2

• Merge pull request #4 from mmcdermott/mmd_changes

  Show description for c8f4144
  mmcdermott

  authored
  c8f4144

• update iterators to reference correct data

  teyaberg

  committed
  91a2056

• Merge branch 'esgpt_caching' into xgboost

  teyaberg

  committed
  f7605b5

• timeable

  teyaberg

  committed
  0bbe879

• Update xgboost.py

  mmcdermott

  authored
  e091432

• Add time tracking around row selection.

  mmcdermott

  authored
  dc8a093

• sparse updates

  teyaberg

  committed
  45470ae

Commits on Jun 3, 2024

• speed up?

  teyaberg

  committed
  84f5a77

• wip

  teyaberg

  committed
  a2fffa4

• config updates

  teyaberg

  committed
  d7c1c13

• fixed tests and added starter code for caching csc matrix

  Oufattole

  committed
  f6786f7

• time profiling

  teyaberg

  committed
  3c1b910

• Merge branch 'xgboost' of github.com:mmcdermott/MEDS_Tabular_AutoML into xgboost

  teyaberg

  committed
  99c107a

• added code to reduce dtype and filter out zero or nan values from sparse matrix

  Oufattole

  committed
  fce9bcc

• Merge branch 'xgboost' of github.com:mmcdermott/MEDS_Tabular_AutoML into xgboost

  teyaberg

  committed
  6f9aa32

• updated fixes for yaml

  Oufattole

  committed
  136204e

• updated scripts for launching hf cohort meds formatting and e2e tabularization

  Oufattole

  committed
  266d785

• updated tests and added task specific caching script

  Oufattole

  committed
  0606a78

• fixed misspelled variable in yaml

  

  Nassim Oufattole

  committed
  9aa2d73

• all working

  teyaberg

  committed
  51c86c6

• Merge branch 'xgboost' of github.com:mmcdermott/MEDS_Tabular_AutoML into xgboost

  teyaberg

  committed
  425b79d

• updates for task cached shard

  teyaberg

  committed
  95f5694

• added cli base code

  Oufattole

  committed
  15f045d

• Merge branch 'xgboost' of github.com:mmcdermott/MEDS_Tabular_AutoML into xgboost

  Oufattole

  committed
  5f5705f

• added cli tests and started adding config hierarchy

  Oufattole

  committed
  a7b1c14

• added messy configs

  Oufattole

  committed
  d225961

• added starter configs

  mmcdermott

  committed
  447d23b

• updated configs to include all tasks

  Oufattole

  committed
  c2b881b

Commits on Jun 4, 2024

• updated describe_codes

  Oufattole

  committed
  9fd33f4

• updated configs for the tabularization

  Oufattole

  committed
  e122d66

• updated task caching to use the updated config

  Oufattole

  committed
  43ac162

• added xgboost support for the updated configs

  Oufattole

  committed
  6856ee8

• updated hydra to reference the correct config files, cli seems to be working

  

  Nassim Oufattole

  committed
  f426263

• updated e2e script for heart failure cohort using the cli

  

  Nassim Oufattole

  committed
  6d68a1e

• fixed polars string cache bug

  

  Nassim Oufattole

  committed
  1cda7f5

• added caching of filtered codes

  

  Nassim Oufattole

  committed
  0111aba

Commits on Jun 5, 2024

• updated cli to take min code frequency

  

  Nassim Oufattole

  committed
  3e5ebae

• [wip] debugging config path resolutions

  Oufattole

  committed
  9f6b0d3

• Added setuptools scm plugin

  mmcdermott

  committed
  bdbe1fc

• added integration test that reproduces the Could not override 'tabularization.min_code_inclusion_frequency'. error

  Oufattole

  committed
  3f5a23d

• Added init file so tabularize default config path is resolved

  Oufattole

  committed
  f1e5647

• passing integration tests now

  Oufattole

  committed
  1685768

• added fix for model.json saving check in integration test

  Oufattole

  committed
  b1f5db6

• added xgboost sweeping support

  Oufattole

  committed
  19f7ce7

• updated to mapper that is more robust to race conditions by using a lock with a timestamp

  Oufattole

  committed
  ed193f0

• removed experiment scripts and old bash cli

  

  Nassim Oufattole

  committed
  4047d7c

• cleaned up formatting for codebase

  

  Nassim Oufattole

  committed
  a19ad3e

• Merge branch 'main' into clean

  Oufattole

  authored
  ec73910