WG meetings 2023

November 3, 2023

Update

1. We've merged the store results in time PR
2. Tentatively: everything having to do with dropbox links is fixed.
3. Saurabh has run packings on endosomes
4. Ruge has merged the inherit key PR
5. can specify which grid file to load by path
6. s3 upload: separated out uploading and opening the results in a new tab.
7. lifecycle rule for s3 setup

TODO:

1. Megan needs to review Ruge's PR
2. run analysis and slides for working group on Tuesday and ASCB practice talk on Thursday 9th

October 10, 2023

Update

1. Megan hasn't yet checked on OBJ loader in viewer, but did look at how the viewer was handling dropbox links.
2. Ruge has PRs open for the inherit key recipe and save metadata
3. Saurabh updated gradients with PR for inversion gradients
4. PR results in memory breaks image creation, need to check on how it's called. maybe add tests for voxelization functions?

TODO:

• if PRs are dependent on each other, keep one as a draft so people know which one to review first.
• Ruge - working on delete old files
• Review open PRs
• group programming: work on multiple files saved in Simularium time.
• analysis - reconstruct 3D packings from PILRs good topic for WG meeting
• start thinking about what we want to present at the cellMorph meeting on 10/26 (Thursday).
• make talk for Wednesday lunch on the 18th

September 26th, 2023

Update

1. We finished variable sized packing, in review, includes a cleanup of how we're saving results in memory
2. Fix to assignment for compartments
3. Fix to inversion in gradients
4. Add upload config option
5. Worked on the dropbox bug, but still not loading in Simularium
6. In progress: packing peroxisomes in real cells with inverted gradients
7. Onboarded Karthik

TODO:

1. Megan - Check on objloader in Simularium viewer for new dropbox link format
2. Ruge is continuing to work on the inherit key for firebase recipes
3. Ruge - delete old results from amazon and firebase
4. Ruge - save metadata of results to firebase
5. Friday group programming start on saving multiple results to one Simularium file using time (hand of to Karthik to finish)
6. Saurabh - analyze the new packing results with peroxisomes

August 29th, 2023

Update

1. Presented in cellMorph meeting, got a lot of TODOs

• created an analysis repo with the TODOs as issues

2. We moved the group programming to Fridays
3. Saurabh - has does some of the initial investigations of the PaCMAP results

TODOs

1. Ruge - Separate out upload to s3 PR with just the needed changes so it's not dependent on merging the run from firebase code
2. Saurabh - Pack peroxisomes in real cell shapes

August 15th, 2023

Update

1. Got code from Matheus on how to reshape PILR into 3D

• plot PILR in 3D
• discuss how to compare areas in 3D, start averaging over the phi angle

2. Need a working group meeting. How many packings per each shape.
3. Approved 2 open PRs, need a few changes on the upload to s3 PR

TODOs:

1. Saurabh - Open 1 line PR to fix linting issue: DONE
2. Ruge - Address changes on upload to s3 PR - still working, merge approved PRs - DONE
3. Megan - go through the backlog and assign/close/edit issues -DONE

August 1st, 2023

Update

1. PaCMAP on individual PILRs does not show any trends
   • Try basic PCA
2. Waiting for 3D PILR reshaping from Matheus
3. PR for vectorizing mesh inside-outside points
   • Compromise between speed and memory usage - make it an option (?)
4. Try packing peroxisomes in actual cell shapes
   • have the meshes ready
   • script is ready to run
   • waiting to start a packing run for all peroxisome cell shapes
5. PR to check if PR exists before running workflow is ready for review
6. Submitted abstract to ASCB
   • submitted for talk AND poster

TO-DO:

1. Pack peroxisomes in cell shapes
   • use variable count?
   • replicates/randomness
   • discuss with WG to figure out best next steps

July 18th, 2023

Update

1. We had two meetings about problems and future directions of correlations analysis.

• Suggestions:
  • PCA on individual PILRs
  • 3D PILR, with progressive subdivisions. Average over phi

2. Can run multiple packings in parallel on the same recipe with meshes now
3. In progress: upload results to AWS automatically and open in Simularium browser window

June 20th, 2023

Update:

1. We are presenting at CellularMorph on Thursday: mostly on correlation analysis for punctuate structures
2. Review of the database upload/download system
3. Set up creds system for firebase

TODO:

1. Include 1 and 3 from below in a slide about our analysis progress
2. Megan: make problem statement slide for CellMorph WG meeting
3. Megan: review open PRs
4. Saurabh: Open 1 more PR
5. Ruge: new issue, setting up S3 write (could use the same creds setup for AWS credentials)

June 6, 2023

Updates:

1. Can run PILR analysis on a dockerized image in parallel: example run for 305 peroxisome packing
2. Can upload gradients to firebase - waiting for review/feedback
3. Can pack ingredients with variable counts when running multiple packings

TODO:

1. TIFF image PR comments, refactoring - Saurabh
2. Parallel packing with meshes
3. Markdown writer updates - Aditya
4. Add tests for variable counts, use the same random seed for packing - Saurabh
5. Run recipes from firebase (waiting for upload PR review/merge) - Ruge

Ideas:

1. Gradually increase resolution of the radial PILR correlation, starting with 2 bins to separate nuclear vs cytosolic structures, continue to finer bins for radial variation.

May 23, 2023

Updates

1. TIFF output is done
2. Ruge waiting on reviews on upload tests
3. Markdown writer - has a path bug that needs to be fixed
4. Saurabh has packings running in parallel

TODO:

April 25, 2023

Updates

1. Megan out until the 10th.
2. Saurabh has first version of TIFF output - spheres and meshes
3. Need to review Ruge's PR
4. Aditya has been working on the markdown writer

TODO:

1. Saurabh will look into gradient error
2. Aditya will work on sphere tree implementation
3. Saurabh will be presenting cellPACK at lunch meeting

April 11, 2023

Updates

1. Aditya finished updates to analysis PR workflow
2. Megan and Saurabh combined changes to Env and Analysis code
3. Small changes: Saurabh fixed ploty plots for multi runs and Megan added recipe info to simularium metadata file header
4. Ruge added unit tests for upload PR - needs review now
5. Ruge: separate PR for additional tests, also ready for review
6. Saurabh has new analysis code, Megan is working on a fix to partner packing schema still
7. Saurabh PR for live analysis

TODO/Discussion points:

1. Megan - add Jessica to TIFF meeting

March 28, 2023

Updates:

1. Presented at WG meeting
2. Agreed jitter as is does not work - Draft PR has debugging changes that we can make more formalized
3. Worked on upload duplicate check
4. Analysis code: able to be run for each ingredient

TODO/discussion Points:

1. Discuss the possibility of implementing a feature to save voxelized images
   • Dan, Alli, Jessica
2. Live analysis mode - options from the analysis config file
3. Schedule pair programming sessions to work on saving voxeling outputs - Megan
4. Megan - fix partner packing using keys
5. Ruge - unit test for upload
6. Aditya - PR for workflow code
7. Saurabh - PR for new analysis code

Mar 14, 2023:

Update:

1. Finished partner packing
2. Gradient packing new schema is in review
3. Able to identify first collision in jitter

TODO/discussion points:

1. discuss save voxelized image/ schedule pair programming?
2. Present updates at march 23rd meeting
3. New partner packing: also be able to define a compartment as a partner
4. combinations of stochastic packing and gradients

Mar 8, 2023:

Updates:

1. Vector Gradient work
2. Partner packing (almost) ready for PR
3. Firebase upload (almost) ready for PR

TODOs:

1. Tests for Partner Packing - Megan - DONE
   1. Make sure repelling works - DONE
2. Make "center" optional for vector gradient mode - Saurabh - DONE
3. Format version migration for new gradient settings -Saurabh (after partners PR) - DONE
4. Analysis report for partners: get partner distance distribution - Saurabh - in progress
5. Code cleanup - Ruge - in progress

Mar 2, 2023:

Updates:

1. Work on GradientData class to validate gradient options

TODOs:

1. look into build_directional_weight_map and get_direction_length. Refer to new gradient schema wiki for details on directional gradient. - DONE

Jan 31, 2023:

Updates

1. PR Updates:

• Finished the scaled distance PR
• Any outstanding issues from Ruge's PR (megan will review today)

2. Update and continued discussion of jitter error issue - debugging idea to run analysis at each step to identify the first ingr that is packed too close to identify issue

notes from discussion:

• are point distances being updated with a -1 when a packing failed at that location?
• effects of jitter attempts number, if we increase it a large amount will the error go away?
• what are the biases in how we are sampling the jitter space around a grid point? square root of r to avoid more likely sampling at the center of the sphere.
• more efficient selection of point within sphere without having to reject the outside of the sphere
• debug sparse grid spacing aberration

TODOs:

1. run similarity analysis on new scaled distance code (Saurabh)
   • consider pulling packing id from recipe version to display in similarity heatmaps
2. post images to AWS for auto validation (Megan)
3. compare contents of local items and database items instead of just strings (Ruge)