From dbb5dc0d46f52fec68b38485f0960ab6b0833a04 Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Thu, 21 Mar 2024 11:13:02 -0400 Subject: [PATCH 01/13] Copy FlyEM workflow files to cellmap --- doc/cellmap/recon_p01_render_import.md | 118 ++++++++++++ doc/cellmap/recon_p02_alignment_prep.md | 177 +++++++++++++++++ doc/cellmap/recon_p03_review_matches.md | 234 +++++++++++++++++++++++ doc/cellmap/recon_p04_alignment_solve.md | 66 +++++++ 4 files changed, 595 insertions(+) create mode 100644 doc/cellmap/recon_p01_render_import.md create mode 100644 doc/cellmap/recon_p02_alignment_prep.md create mode 100644 doc/cellmap/recon_p03_review_matches.md create mode 100644 doc/cellmap/recon_p04_alignment_solve.md diff --git a/doc/cellmap/recon_p01_render_import.md b/doc/cellmap/recon_p01_render_import.md new file mode 100644 index 00000000..88b67fe9 --- /dev/null +++ b/doc/cellmap/recon_p01_render_import.md @@ -0,0 +1,118 @@ +# Reconstruction Part 1: Render Import +## Common Parameters +```bash +export NEARLINE_HOST="e06u14.int.janelia.org" +export BASE_WORK_DIR="/groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m" +``` +## Setup Work Area +```bash +# Run on host with access to /nearline because that is where .dat files are stored. +ssh ${NEARLINE_HOST} + +cd ${BASE_WORK_DIR} + +# Create work area ${BASE_WORK_DIR}/${REGION}/${TAB} with copies of pipeline scripts. +# +# USAGE: ./setup.sh ( e.g. ./setup.sh VNC Sec32 ) +./setup.sh VNC Sec26 +``` +## Validate and Correct Scope File Transfers +VNC Sec26 Processing Time: 15 minutes using 1 core *(can take much longer if there are many transfer issues)* + +The Fly EM image transfer process continuously polls the EM scopes for new data +and then copies (and converts) that data to centralized network filesystems. +The process works most of the time, but occasionally files are not transferred. +It is important to ensure that all data has been properly transferred +before deriving and importing metadata into the Render web services. + +### Transfer Process (for context) +![FIBSEM transfer flow diagram](fibsem-transfer-flow.png) + +### Validation Steps +```bash +# While still on nearline host ... +cd VNC/Sec26 # cd ${BASE_WORK_DIR}/${REGION}/${TAB} + +# Run as flyem user so that scopes can be reached and corrections can be written. +su flyem + +# Use date range from transferred files to go back to scope and get listing of +# all files acquired during that time. +./01_gen_scope_dat_file_lists.sh + +# Look for missing and/or misplaced files. +# InLens png, Transferred dat, Archived dat, Scope dat, and Logs counts should be the same. +# Nothing should be missing. +./02_check_tabs.sh + +| ==================================================== +| Tab: Z0720-07m_VNC_Sec26 +| +| InLens png Transferred dat Archived dat Scope dat Logs +| ---------- --------------- ------------ --------- --------- +| 119187 119187 119187 119208 119187 +| +| 21 missing InLens png: ['Merlin-6285_21-08-23_084156_0-0-0', 'Merlin-6285_21-08-23_084156_0-0-1', 'Merlin-6285_2 +| 21 missing dat: ['Merlin-6285_21-08-23_084156_0-0-0', 'Merlin-6285_21-08-23_084156_0-0-1', 'Merlin-6285_2 +| 0 unarchived dat: [] +| 0 non-standard dat: [] + +# In this example, 21 dat files (along with their corresponding png files) are missing +# so we need to try to transfer the dat files again. +# This script serially scp's each missing dat file and converts it to png. +./03_fetch_missing_dat.sh + +# Re-run the check: nothing should be missing now unless the source files are corrupted in some way. +./02_check_tabs.sh + +| ==================================================== +| Tab: Z0720-07m_VNC_Sec26 +| +| InLens png Transferred dat Archived dat Scope dat Logs +| ---------- --------------- ------------ --------- --------- +| 119208 119208 119208 119208 119208 +| +| 0 missing InLens png: [] +| 0 missing dat: [] +| 0 unarchived dat: [] +| 0 non-standard dat: [] + +# Verify the total dat file count and the first and last dat file names +# match what Wei thinks they should be. Wei typically emails the names and counts +# but sometimes you need to ask Wei to send the information. +# +# TODO: merge this information into the check tabs tool +./list_first_and_last_dats.sh + +# Once everything looks good, exit from the flyem shell. +exit +``` + +### Move Bad dat Files + +```bash +# If dat files are corrupted such that png files cannot be generated for them ... +NEARLINE_TAB_DIR="/nearline/flyem2/data/Z0720-07m_${REGION}_${TAB}" +BAD_DAT_FILE="???" +mkdir ${NEARLINE_TAB_DIR}/dat_bad +mv ${NEARLINE_TAB_DIR}/dat/${BAD_DAT_FILE} ${NEARLINE_TAB_DIR}/dat_bad + +# Note that the render import process will automatically +# patch partial layers with tiles from complete adjacent layers. +``` + +## Import Metadata into Render Web Services +VNC Sec26 Processing Time: 2 minutes using 32 cores + +```bash +# While still on nearline host in ${BASE_WORK_DIR}/${REGION}/${TAB} and as yourself ... + +# Run script that parses dat headers, generates JSON tile specs, and imports them into render. +# Current process sets up a 32-worker dask cluster to do the import, so you may need to +# reconfigure the number of dask workers if your nearline host does not have enough cores. +./06_dat_to_render.sh + +# Once import completes everything else needs to launch from an LSF submit host, +# so exit from the nearline host. +exit +``` diff --git a/doc/cellmap/recon_p02_alignment_prep.md b/doc/cellmap/recon_p02_alignment_prep.md new file mode 100644 index 00000000..51ea411c --- /dev/null +++ b/doc/cellmap/recon_p02_alignment_prep.md @@ -0,0 +1,177 @@ +# Reconstruction Part 2: Alignment Preparation +## Common Parameters +```bash +export BSUB_HOST="login1.int.janelia.org" +export BASE_WORK_DIR="/groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m" +export WORK_DIR="${BASE_WORK_DIR}/VNC/Sec26" # more generically: ${BASE_WORK_DIR}/${REGION}/${TAB} +``` +## Validate Tile Specs +VNC Sec26 Processing Time: 1 minute using 10 cores + +```bash +# Run on LSF / bsub submit host because most work will be done using LSF cluster. +ssh ${BSUB_HOST} +cd ${WORK_DIR} + +# Setup work area and parameters for LSF array job to validate recently imported render tile specs. +./07_gen_validate_spec_run.sh + +| Retrieving z values from http://10.40.3.162:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v1_acquire/zValues +| +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_060707_37_ValidateTiles/job_specific_parameters.txt +| .......... +| +| Common parameters for array job are: +| --baseDataUrl http://10.40.3.162:8080/render-ws/v1 --owner Z0720_07m_VNC --project Sec26 --stack v1_acquire --validatorClass org.janelia.alignment.spec.validator.TemTileSpecValidator --validatorData minCoordinate:-999999,maxCoordinate:999999,minSize:1000,maxSize:99999 +| +| Created bsub array script for 10 jobs (z or z batches) in: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_060707_37_ValidateTiles/bsub-array.sh +| +| Logs will be written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_060707_37_ValidateTiles/logs + +# If common parameters look ok (they should), run submit script (full path is output above): +run_20211217_060707_37_ValidateTiles/bsub-array.sh + +| This job will be billed to flyem +| Job <113403049> is submitted to default queue . +| This job will be billed to flyem +| Job <113403050> is submitted to default queue . +| This job will be billed to flyem +| Job <113403051> is submitted to default queue . + +# The submit script launches 3 chained jobs: +# 1. The first one runs just the first of the array job inputs [1-1]. +# 2. If that succeeds, the second job runs the remaining array job inputs [2-10]. +# 3. When the second job group completes, the third job runs a log check script. + +# Output from the log check script is emailed to you (the job launcher) and looks like this: + +| Job 113403051: in cluster Done +| ... +| The output (if any) follows: +| All log files in /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_060707_37_ValidateTiles/logs look okay. +``` + +## Generate Mipmaps +VNC Sec26 Processing Time: 17 minutes using 1,036 cores + +```bash +# Setup work area and parameters for LSF array job to generate down-sampled mipmaps for all source images. +./08_gen_mipmap_run.sh + +# Output (including full path to submit script) is similar to validation example above. + +# If common parameters look ok (they should), run submit script (full path is gen script output): +run_20211217_063059_153_Mipmap/bsub-array.sh + +# Output and 3-chained-job pattern is similar to validation example above. +# As with validation, output from the log check script is emailed to you. +``` + +## Generate Tile Pairs +VNC Sec26 Processing Time: 1 minute using 1 core + +```bash +# Create tile pair metadata (JSON files) to be used as input for match. +# This java process runs directly on the submit host which technically you are not supposed to do. +# The process is relatively short (a minute or so) and has never been an issue, so I just run it on the submit host. +# If anyone ever complains, it is easy enough to use an interactive node (busb -i) and run there. +./11_gen_new_pairs.sh + +| ------------------------------------------------------------------------ +| Setting up montage pairs (log is captured so wait for results) ... +| +| MAX_PAIRS_PER_FILE was set to 50 +| +| Full pair generation output written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/tile_pairs-20211217_071324.log +| ... +| ------------------------------------------------------------------------ +| Setting up cross pairs (log is captured so wait for results) ... +| ... +| 07:14:59.399 [main] INFO [org.janelia.alignment.util.FileUtil] saveJsonFile: exit, wrote data to /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/pairs_cross/tile_pairs_cross_p11919.json.gz +| 07:14:59.399 [main] INFO [org.janelia.render.client.TilePairClient] deriveAndSaveSortedNeighborPairs: exit, saved 715143 total pairs +| 07:14:59.399 [main] INFO [org.janelia.render.client.ClientRunner] run: exit, processing completed in 0 hours, 0 minutes, 57 seconds +``` + +## Generate Match Data +VNC Sec26 Processing Time: 75 minutes using 13,787 cores + +```bash +# Setup work areas and parameters for LSF array jobs to generate same layer and cross layer matches. +./12_gen_staged_match_run.sh + +| ------------------------------------------------------------------------ +| Setting up job for run_20211217_071528_16_multi_stage_match_montage ... +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/common_parameters.txt +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/job_specific_parameters.txt +| .................. +| Created bsub array script for 1867 jobs in /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/bsub-array.sh +| Logs will be written to /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/logs +| +| ------------------------------------------------------------------------ +| Setting up job for run_20211217_071529_994_multi_stage_match_cross ... +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/common_parameters.txt +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/job_specific_parameters.txt +| ....................................................................................................................... +| Created bsub array script for 11920 jobs in /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/bsub-array.sh +| Logs will be written to /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/logs + +# Run both montage and cross bsub-array.sh scripts (full paths are in gen script output). +run_20211217_071528_16_multi_stage_match_montage/bsub-array.sh +run_20211217_071529_994_multi_stage_match_cross/bsub-array.sh + +# As with previous array jobs, output from a log check script is emailed to you +# for both the same layer (montage) and cross layer match processes. + +# The check script provides summary statistics for each stage of ... + +# the montage multi-stage process: + +| Stage parameters loaded from /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/stage_parameters.montage.json +| ......... +| [stage name]_ +| SIFT SIFT Combined Combined SIFT_s[render scale]e[epsilon]_i[inliers]_c[coverage]pct_ +| Total Total Percent SIFT Combined Poor Poor Poor Poor GEO_s[render scale]e[epsilon]_i[combined inliers]_c[combined coverage]pct_ +| Stage Processed Saved Saved Saved Saved Quantity Coverage Quantity Coverage Slug: d[delta z]_h[parameter hash] +| --------------- --------- --------- --------- --------- --------- --------- --------- --------- --------- ----------------------------------------------------------------------------------------------------------- +| montagePass1 93341 73064 78.3 73064 0 17870 2407 0 0 montagePass1_SIFT_s0.40e02_i025_c070pct_GEO_none----_----_-------_d000_h8da6a90ff657806ebc8f4b9ff86f3340 +| montagePass2 20277 608 3.0 582 26 15703 0 3852 114 montagePass2_SIFT_s0.50e02_i025_c070pct_GEO_s1.00e05_i025_c070pct_d000_h83331cf1918bd1cd1235c1093e71caeb +| montagePass3 19669 1155 5.9 1111 44 0 0 16494 2020 montagePass3_SIFT_s0.60e03_i025_c070pct_GEO_s1.00e05_i000_c060pct_d000_ha8feba4c49ed328a2d1ac7decdb756b6 +| montagePass4 18514 12 0.1 0 12 0 0 16279 2223 montagePass4_SIFT_s0.60e03_i025_c070pct_GEO_s1.00e05_i000_c060pct_d000_hf170feeae225c5014591c47b5567613b +| montagePass5 18502 1296 7.0 1128 168 0 0 16162 1044 montagePass5_SIFT_s0.60e03_i012_c050pct_GEO_s1.00e03_i000_c050pct_d000_h118513a845169760a041936c8ff80f70 +| montagePass6 17206 3190 18.5 3190 0 14016 0 0 0 montagePass6_SIFT_s1.00e05_i012_c000pct_GEO_none----_----_-------_d000_hb1cf242963c833a4b1f1e05f2c673e91 +| +| processed_pair_count: 93341 +| saved_pair_count: 79325 +| average_seconds_per_pair: 13.26 +| +| jobs: 1867 +| average_minutes_per_job: 11.05 +| max_job_minutes: 71.93 + +# and the cross multi-stage process: + +| Stage parameters loaded from /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/stage_parameters.cross.json +| ........................................................... +| [stage name]_ +| SIFT SIFT Combined Combined SIFT_s[render scale]e[epsilon]_i[inliers]_c[coverage]pct_ +| Total Total Percent SIFT Combined Poor Poor Poor Poor GEO_s[render scale]e[epsilon]_i[combined inliers]_c[combined coverage]pct_ +| Stage Processed Saved Saved Saved Saved Quantity Coverage Quantity Coverage Slug: d[delta z]_h[parameter hash] +| --------------- --------- --------- --------- --------- --------- --------- --------- --------- --------- ----------------------------------------------------------------------------------------------------------- +| crossPass1 715143 156530 21.9 156530 0 558613 0 0 0 crossPass1_SIFT_s0.05e00_i020_c050pct_GEO_s0.10e00_i150_c050pct_d006_h7c57eb6f14fd7006563010950b1a8c4c +| crossPass2 558613 374210 67.0 374210 0 134499 0 49904 0 crossPass2_SIFT_s0.10e01_i020_c050pct_GEO_s0.15e00_i150_c050pct_d006_h43fb6938fc19a0a7bef73756f1dd03c2 +| crossPass3 184403 8875 4.8 8875 0 103158 0 72370 0 crossPass3_SIFT_s0.15e01_i020_c050pct_GEO_s0.25e01_i150_c050pct_d006_h888d17a7cd1c7380997dd41b256b77c9 +| crossPass4 85288 31243 36.6 31243 0 38266 15779 0 0 crossPass4_SIFT_s0.25e02_i150_c030pct_GEO_none----_----_-------_d003_h9dbf8fa30fbb5f6ccf6330362cdedc20 +| crossPass5 35823 15901 44.4 15901 0 0 0 19922 0 crossPass5_SIFT_s0.25e02_i020_c050pct_GEO_s0.50e02_i075_c000pct_d002_hc9678dff8a758fa34fa42d4f78416501 +| crossPass6 9500 8070 84.9 8070 0 1430 0 0 0 crossPass6_SIFT_s0.25e02_i020_c000pct_GEO_none----_----_-------_d001_h97ce9c637dced2411d673cb2865ba58c +| +| processed_pair_count: 715143 +| saved_pair_count: 594829 +| average_seconds_per_pair: 3.07 +| +| jobs: 11920 +| average_minutes_per_job: 3.07 +| max_job_minutes: 12.92 +``` \ No newline at end of file diff --git a/doc/cellmap/recon_p03_review_matches.md b/doc/cellmap/recon_p03_review_matches.md new file mode 100644 index 00000000..13c4e42b --- /dev/null +++ b/doc/cellmap/recon_p03_review_matches.md @@ -0,0 +1,234 @@ +# Reconstruction Part 3: Review Matches and Fix Connectivity Issues +## Common Parameters +```bash +export BSUB_HOST="login1.int.janelia.org" +export BASE_WORK_DIR="/groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m" +export WORK_DIR="${BASE_WORK_DIR}/VNC/Sec26" # more generically: ${BASE_WORK_DIR}/${REGION}/${TAB} +``` +## Generate Connected Tile Cluster Data +VNC Sec26 Processing Time: 5 minutes using 1 core + +```bash +# Continue on LSF / bsub submit host because most work will be done using LSF cluster. +ssh ${BSUB_HOST} +cd ${WORK_DIR} + +# Use a single node interactive session because cluster counting usually +# takes a few minutes and you don't want to cause trouble on the submit host. +bsub -n 1 -P flyem -Is /bin/bash --login + +# Extract match counts for entire volume and identify all connected tile clusters. +# Ultimately, there should be one and only one connected tile cluster. +./21_count_acquire_clusters.sh + +| Connected tile cluster counts will be written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/cluster_count.v1_acquire.log +| +| Be patient, this could take a few minutes ... +| +| Parsing /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/cluster_count.v1_acquire.log to create: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/excluded_cells.json + +# Exit interactive session and return to the submit host to review results. +exit +``` + +## Trim Unconnected Resin Tiles +VNC Sec26 Processing Time: 15 minutes using 1 core (review), 1 minute using 52 cores (create trimmed stack) + +Most volumes have some tiles with just resin in them, especially at the beginning and end of an +acquisition. If the resin isn't dirty enough, those tiles won't produce point match correspondences +with adjacent tiles creating smaller connected clusters or "islands". The alignment solve process +requires a single connected set of tiles as input, so we need to remove clusters of purely resin tiles +and fix/patch missing connections for tiles that have tissue. + +The `excluded_cells.json` file produced by `./21_count_acquire_clusters.sh` contains cell and z range +information for all connected clusters. There are usually many many tiny clusters in resin areas that +can be combined into one or just a few larger clusters for removal (trimming). Unfortunately, +combining/reducing the small resin clusters is currently a manual process. + +The `excluded_cells.json` file looks like this: +```javascript +[ + { "cellIds": [ + "0,0", + "0,0" + ], + "minZ": 1, + "maxZ": 4244 + }, + { + "cellIds": [ + "0,0", + "0,0" + ], + "minZ": 4245, + "maxZ": 4455 + }, + { + "cellIds": [ + "0,0", + "0,0" + ], + "minZ": 4456, + "maxZ": 4459 + }, ... +``` +To combine/reduce resin clusters, use CATMAID (or any tile viewer) to confirm that the cell only contains +resin tiles and then edit `excluded_cells.json` accordingly. + +![](resin-tiles-in-catmaid.png) + +For this mini-example, the combined json for cell 0,0 would look like: +```javascript +[ + { "cellIds": ["0,0"], "minZ": 1, "maxZ": 4459 }, + ... +] +``` + +After combining/reducing all resin tile cells in VNC Sec26, the final `excluded_cells.json` file +looks like this: +```javascript +[ + { "cellIds": [ "0,0" ], "minZ": 1, "maxZ": 4471 }, + { "cellIds": [ "0,0" ], "minZ": 23965, "maxZ": 24154 }, + { "cellIds": [ "0,0" ], "minZ": 25777, "maxZ": 25867 }, + { "cellIds": [ "0,1" ], "minZ": 1, "maxZ": 2900 }, + { "cellIds": [ "0,1" ], "minZ": 25245, "maxZ": 25365 }, + { "cellIds": [ "0,3" ], "minZ": 21797, "maxZ": 21911 }, + { "cellIds": [ "0,4" ], "minZ": 18750, "maxZ": 21911 } +] +``` + +Once `excluded_cells.json` is ready ... +```bash +# Generate a trimmed (clip) stack by submitting a bsub array job. +./22_gen_clip_stack.sh +``` + +## Review Connectivity for Trimmed Stack +VNC Sec26 Processing Time: 30 minutes using 1 core *(time can vary widely depending upon connection issues)* + +Once the trimmed stack has been created and completed (you should receive an email) ... +```bash +# Re-count connected tile clusters for the trimmed stack. +./23_count_trimmed_clusters.sh + +| Connected tile cluster counts will be written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/cluster_count.v1_acquire_trimmed.log +| +| Be patient, this could take a few minutes ... +| +| 20:56:16.141 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClustersForSlab: slab z 24995.0 to 25776.0 has 1 connected tile sets with sizes [1032] +| 20:56:16.141 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: found 1 connected tile set with size [108155] +| 20:56:16.291 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: 108155 tile set with first tile 21-08-19_194752_0-0-2.1.0 and last tile 21-09-14_134854_0-0-0.25776.0 +| 20:56:16.291 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: found 3 completely unconnected tiles: [21-09-06_113017_0-0-4.18749.0, 21-09-13_091915_0-0-1.23966.0, 21-09-13_091915_0-0-2.23966.0] +| 20:56:16.291 [main] INFO [org.janelia.render.client.ClientRunner] run: exit, processing completed in 0 hours, 1 minutes, 20 seconds + +# In this case, the count log output shows that there is now only 1 cluster (that's good) but +# that there are 3 completely unconnected tiles. These tiles need to be checked and fixed ... +``` + +The first unconnected tile `21-09-06_113017_0-0-4.18749.0` was orphaned by our earlier trim and can be +fixed by simply editing `excluded_cells.json` +```javascript +{ "cellIds": [ "0,4" ], "minZ": 18750, "maxZ": 21911 } -> +{ "cellIds": [ "0,4" ], "minZ": 18749, "maxZ": 21911 } +``` + +The other two unconnected tiles `21-09-13_091915_0-0-1.23966.0` and `21-09-13_091915_0-0-2.23966.0` happened to +be corrupted during acquisition: + +![](problem-tiles.png) + +To fix this kind of problem, we usually patch the corrupted tiles with tiles from an adjacent layer. + +## Patch Corrupted Tiles +VNC Sec26 Processing Time: 45 minutes using essentially 1 core + +The current patch process has several manual steps and is a little messy. + +Run `./setup_patch.sh` to create a patch subdirectory with the scripts needed for patching: + +```bash +# USAGE: ./setup_patch.sh [patch z] ... +# In this case, the z to patch is 23966 (from unconnected tile info). +./setup_patch.sh 23966 + +| Copy these patch tile id lines into +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/v2_patch/06_patch_tile_specs.py: +| +| ("21-09-13_091915_0-0-1.23966.0", -1), # patch from prior layer +| ("21-09-13_091915_0-0-2.23966.0", -1), # patch from prior layer +| +| After patching, generate match pairs by running: +| ./11_gen_new_pairs.sh 23966 + +# move into the patch subdirectory +cd v2_patch +``` + +Edit `06_patch_tile_specs.py` to insert the tile ids (copied from the setup script output) to +be patched. The setup script defaults to patching from the previous layer `(tile_id, -1)` but in +this case the following layer looks better so the data was changed to `(tile_id, 1)`. It is a good +idea to view the area around the patch in CATMAID before deciding the best way to patch it. +```python +def main(): + owner = "Z0720_07m_VNC" # TODO: update with BR or VNC + project = "Sec26" # TODO: update with Sec number + stack = "v2_acquire_trimmed" # TODO: update with patch version number + + # TODO: update tile ids + tile_ids_to_patch = [ + ("21-09-13_091915_0-0-1.23966.0", 1), # patch from next layer + ("21-09-13_091915_0-0-2.23966.0", 1), # patch from next layer + ] +``` + +Copy the `v1_acquire_trimmed` stack to `v2_acquire_trimmed` (with the updated excluded_cells.json): +```bash +# Create and immediately submit copy job: +./01_gen_new_trimmed_stack.sh + +# Once the copy job completes, patch tiles in the `v2_acquire_trimmed` stack: +./06_patch_tile_specs.py + +| submitting PUT http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/state/LOADING +| submitting DELETE http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/tile/21-09-13_091915_0-0-1.23966.0 +| submitting DELETE http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/tile/21-09-13_091915_0-0-2.23966.0 +| submitting PUT http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/resolvedTiles for 2 tile specs +| submitting PUT http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/state/COMPLETE + +# Generate matches for the patched tiles: +./11_gen_new_pairs.sh 23966 + +| ... (similar output to when this script was originally run for alignment prep) + +./12_gen_staged_match_run.sh + +| ... (similar output to when this script was originally run for alignment prep) + +# Run both montage and cross bsub-array.sh scripts (full paths are in gen script output). +run_20220206_093430_479_multi_stage_match_montage/bsub-array.sh +run_20220206_093430_511_multi_stage_match_cross/bsub-array.sh + +# The match jobs should finish quickly since they only cover a few tiles. +# After receiving email confirmation of their completion, +# re-count connected tile clusters for the v2 trimmed stack. +./23_count_trimmed_clusters.sh + +| Connected tile cluster counts will be written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/cluster_count.v2_acquire_trimmed.log +| +| Be patient, this could take a few minutes ... +| +| 09:57:35.971 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClustersForSlab: slab z 24995.0 to 25776.0 has 1 connected tile sets with sizes [1032] +| 09:57:35.971 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: found 1 connected tile set with size [108102] +| 09:57:36.073 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: 108102 tile set with first tile 21-08-19_194752_0-0-2.1.0 and last tile 21-09-14_134854_0-0-0.25776.0 +| 09:57:36.073 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: found 0 completely unconnected tiles: [] +| 09:57:36.073 [main] INFO [org.janelia.render.client.ClientRunner] run: exit, processing completed in 0 hours, 1 minutes, 9 seconds + +# The count log output now shows that there is only 1 cluster and no unconnected tiles +# We are finally ready to run the alignment solve process. +``` \ No newline at end of file diff --git a/doc/cellmap/recon_p04_alignment_solve.md b/doc/cellmap/recon_p04_alignment_solve.md new file mode 100644 index 00000000..7cc6aa91 --- /dev/null +++ b/doc/cellmap/recon_p04_alignment_solve.md @@ -0,0 +1,66 @@ +# Reconstruction Part 4: Align Volume +## Common Parameters +```bash +export BSUB_HOST="login1.int.janelia.org" +export BASE_WORK_DIR="/groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m" +export WORK_DIR="${BASE_WORK_DIR}/VNC/Sec26" # more generically: ${BASE_WORK_DIR}/${REGION}/${TAB} +``` +## Generate Alignment Transformations (Run Solve) +VNC Sec26 Processing Time: 215 minutes using Spark with 18 eleven-core workers, 200 cores in all + +```bash +# Continue on LSF / bsub submit host because most work will be done using LSF cluster. +ssh ${BSUB_HOST} +cd ${WORK_DIR} + +# Run solve on Spark ... +# +# Notes: +# - Default block size for distributed solve is 500 z layers +# so it is easy to allocate too many workers. +# - Using 12/18/24 workers for small/medium/large tabs works well. +./41_alignment_solve.sh 18 +``` + +## Generate Thickness Correction Data +VNC Sec26 Processing Time: 112 minutes using 215 cores + +The `./41_alignment_solve.sh` script submits a job to run `./42_gen_z_corr_run.sh` +when the alignment solve job has completed - so there is no need to manually launch +thickness correction. The thickness correction data generation process involves a +distributed z layer cross correlation calculation followed by a solve. When the +thickness correction solve completes, interactive plots are generated for cross correlation +and z correction values and an email with the Zcoords.txt thickness correction data file path +and URLs for the interactive plots is sent. The email looks like this: + +![](Sec26-v2-z-corr-solve-email.png) + +## Review Thickness Correction Data +VNC Sec26 Processing Time: 30 minutes using 1 core *(time will vary, does not include patch time for problems)* + +Although we ultimately use the thickness correction data to properly render the volume, +we also use the cross correlation information to identify alignment problems. +For the Fly EM volumes, we've found that properly aligned layers typically have a +cross correlation value `>= 0.98`. + +This plot of the `v2_acquire_trimmed` alignment shows that something is wrong around `z 552`: + +![](Sec26-v2-correlation-plot.png) + +Clicking on the problem plot point, opens a CATMAID browser window for the z layer. +You can then navigate through the stack to determine what went wrong during alignment. + +![](Sec26-v2-poorly-aligned-tiles.png) + +In this case, `z 552` has a big wave going through it and the third tile (in cell 0,2) is not +connected to its neighbor. That cell remains disconnected until the `z 1200` area. To correct +the problem, we can either remove/trim the cell tiles or add point matches to better connect the tiles +and prevent them from drifting apart during the solve process. Removal is the best option +when the tiles do not contain any useful information. In all other cases, the tile connections +should be improved. + +## Correct Alignment Issues +If alignment issues are found, corrections are performed +using the [patch process](recon_p03_review_matches.md#patch-corrupted-tiles) +with an incremented patch version number. You then repeat the solve and review processes +until you have a clean alignment. \ No newline at end of file From 8a93be33f65b671763d42bfce67a42dafc343480 Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Mon, 25 Mar 2024 12:00:19 -0400 Subject: [PATCH 02/13] Add diagram and description of project management --- doc/cellmap/project_management.md | 49 +++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 doc/cellmap/project_management.md diff --git a/doc/cellmap/project_management.md b/doc/cellmap/project_management.md new file mode 100644 index 00000000..55e90bb7 --- /dev/null +++ b/doc/cellmap/project_management.md @@ -0,0 +1,49 @@ +# Project management +All projects are tracked on the [recon_fibsem repository](https://github.com/JaneliaSciComp/recon_fibsem) on GitHub. +This repository is used for documenting and exchanging the information necessary to ingest and process new datasets, tracking the progress of the processing, and reviewing the processed datasets. + +```mermaid +sequenceDiagram + participant F as FIB-SEM
shared resource + participant S as SciComp + participant G as GitHub + participant C as Collaborator + + F->>G: Create new issue + G-->>S: Notify + activate S + S->>G: Include in project board + Note over S: Ingestion and processing + loop Review + S->>G: Post updates and questions + G-->>C: Notify + C->>G: Review and feedback + G-->>S: Notify + end + S->>C: Finalize and hand off + deactivate S +``` + +## Issues +For every new dataset, an issue is opened. What information should be included and what processes this triggers is detailed in [part 1 of the pipeline documentation](recon_p01_render_import.md). Here, we concentrate on the management of the issues. + +Generally, GitHub is mainly used to track the progress of the processing and to communicate with the collaborators. This can be sped up by assigning the issue to the responsible person, by using labels, and by tagging people from which feedback is needed. + +### Labels + +There are three different categories of labels: + +- **Workflow labels**: `00: imaging in progress`, `01: prep done`, `02: alignment done`, `03: alignment review done`, `04: z correction done`, `05: streak correction done`, `06: intensity correction done`, `07: straightening done`, `09: clean-up done` +- **Collaborator labels**: `Cell Map`, `eFIB-SEM SR` (the FIB-SEM shared resource), `Fly EM` +- **TODO labels**: `needs alignment`, `needs clean-up`, `needs cross-correlation`, `needs N5 creation`, `needs review`, `needs straightening`, `needs streak correction`, `non-rigid tile deformations`, `alignment review concerns` + +TODO: What labels are required and what’s the logic? + +### Project board + +All issues are collected on the [project board](https://github.com/JaneliaSciComp/recon_fibsem/projects/1), which has four columns: + +- **Alignment**: All new issues are automatically entered in this column and stay there during the all standard processing steps. +- **Review**: Once processing is done, the issue is moved to this column, where Preibisch and the collaborators have a final look at it. If further processing is necessary, the issue stays in this column for the time necessary to address all issues. +- **Done**: Once the dataset passes the review process, it is moved to this column to signal that we don’t do further processing on the dataset. The dataset is now ready to be handed off to the collaborator. +- **Cleaned Up**: TODO: find out what exactly this means. \ No newline at end of file From f012bb3d7f251f5b351021d6fea1add5839eaddc Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Mon, 25 Mar 2024 15:30:10 -0400 Subject: [PATCH 03/13] Clarify questions in project management --- doc/cellmap/project_management.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/cellmap/project_management.md b/doc/cellmap/project_management.md index 55e90bb7..c7424993 100644 --- a/doc/cellmap/project_management.md +++ b/doc/cellmap/project_management.md @@ -37,7 +37,7 @@ There are three different categories of labels: - **Collaborator labels**: `Cell Map`, `eFIB-SEM SR` (the FIB-SEM shared resource), `Fly EM` - **TODO labels**: `needs alignment`, `needs clean-up`, `needs cross-correlation`, `needs N5 creation`, `needs review`, `needs straightening`, `needs streak correction`, `non-rigid tile deformations`, `alignment review concerns` -TODO: What labels are required and what’s the logic? +Each dataset should have the right collaborator label. Apart from that, there are no strict rules about labeling. Use the TODO labels as needed to keep track of what needs to be done. The workflow labels are used to document the processing steps that have been done. When a dataset is done processing, usually all workflow labels below 06 are removed, as they are recorded in the issue history anyway. ### Project board @@ -46,4 +46,4 @@ All issues are collected on the [project board](https://github.com/JaneliaSciCom - **Alignment**: All new issues are automatically entered in this column and stay there during the all standard processing steps. - **Review**: Once processing is done, the issue is moved to this column, where Preibisch and the collaborators have a final look at it. If further processing is necessary, the issue stays in this column for the time necessary to address all issues. - **Done**: Once the dataset passes the review process, it is moved to this column to signal that we don’t do further processing on the dataset. The dataset is now ready to be handed off to the collaborator. -- **Cleaned Up**: TODO: find out what exactly this means. \ No newline at end of file +- **Cleaned Up**: Once the collaborator has taken over the dataset and has indicated that no further processing is necessary, the issue is moved to this column. At this stage all intermediate data that was used for processing (including 8bit images for alignment) are deleted. This clean up is only done irregularly if disk space is scarce. \ No newline at end of file From fdb8bfc1cf5bb42b3737ecb0399a65781f16974c Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Mon, 25 Mar 2024 15:34:40 -0400 Subject: [PATCH 04/13] Update and improve transfer flow diagram --- doc/cellmap/fibsem-transfer-flow.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/cellmap/fibsem-transfer-flow.md b/doc/cellmap/fibsem-transfer-flow.md index ea701468..94f042de 100644 --- a/doc/cellmap/fibsem-transfer-flow.md +++ b/doc/cellmap/fibsem-transfer-flow.md @@ -1,13 +1,13 @@ ```mermaid graph TD - SCOPE_DAT_A[jeiss scope
z1_0-0-0.dat, z1_0-0-1.dat,
z1_0-1-0.dat, z1_0-1-1.dat] --> TRANSFER_DAT - SCOPE_DAT_B[jeiss scope
zn_0-0-0.dat, zn_0-0-1.dat,
zn_0-1-0.dat, zn_0-1-1.dat] --> TRANSFER_DAT - TRANSFER_DAT{transfer} --> DM11_DAT_A & DM11_DAT_B - DM11_DAT_A[dm11
z1_0-0-0.dat, z1_0-0-1.dat,
z1_0-1-0.dat, z1_0-1-1.dat ] --> DAT_TO_H5 - DM11_DAT_B[dm11
zn_0-0-0.dat, zn_0-0-1.dat
zn_0-1-0.dat, zn_0-1-1.dat ] --> DAT_TO_H5 - DAT_TO_H5{convert and
remove dm11 dats
after verification} --> DM11_RAW_H5 & NRS_ALIGN_H5 - DM11_RAW_H5[dm11
z1.raw.h5, ...
zn.raw.h5 ] --> ARCHIVE_H5 - NRS_ALIGN_H5[nrs
z1.uint8.h5, ...
zn.uint8.h5] - ARCHIVE_H5{archive and
remove dm11 raw h5s
after verification} --> NEARLINE_RAW_H5 - NEARLINE_RAW_H5[nearline
z1.raw-archive.h5, ...
zn.raw-archive.h5] + SCOPE_DAT_A(jeiss scope
z1_0-0-0.dat, z1_0-0-1.dat,
z1_0-1-0.dat, z1_0-1-1.dat) --> TRANSFER_DAT + SCOPE_DAT_B(jeiss scope
zn_0-0-0.dat, zn_0-0-1.dat,
zn_0-1-0.dat, zn_0-1-1.dat) --> TRANSFER_DAT + TRANSFER_DAT{{transfer}} --> DM11_DAT_A & DM11_DAT_B + DM11_DAT_A(prfs
z1_0-0-0.dat, z1_0-0-1.dat,
z1_0-1-0.dat, z1_0-1-1.dat ) --> DAT_TO_H5 + DM11_DAT_B(prfs
zn_0-0-0.dat, zn_0-0-1.dat
zn_0-1-0.dat, zn_0-1-1.dat ) --> DAT_TO_H5 + DAT_TO_H5{{ convert and
remove prfs dats
after verification }} --> DM11_RAW_H5 & NRS_ALIGN_H5 + DM11_RAW_H5(prfs
z1.raw.h5, ...
zn.raw.h5 ) --> ARCHIVE_H5 + NRS_ALIGN_H5(nrs
z1.uint8.h5, ...
zn.uint8.h5) + ARCHIVE_H5{{archive and
remove prfs raw h5s
after verification}} --> NEARLINE_RAW_H5 + NEARLINE_RAW_H5(nearline
z1.raw-archive.h5, ...
zn.raw-archive.h5) ``` \ No newline at end of file From 0fbabd3b1693e541584e6eac2558357783da7294 Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Mon, 25 Mar 2024 15:58:06 -0400 Subject: [PATCH 05/13] Re-structure cellmap documentation --- doc/cellmap/project_management.md | 2 +- doc/cellmap/steps/1_transfer.md | 5 ++++ .../2-1_render_import.md} | 0 .../2-2_alignment_prep.md} | 0 .../2-3_review_matches.md} | 0 .../2-4_alignment_solve.md} | 2 +- doc/cellmap/steps/3_export.md | 0 doc/cellmap/summary.md | 24 +++++++++++++++++++ 8 files changed, 31 insertions(+), 2 deletions(-) create mode 100644 doc/cellmap/steps/1_transfer.md rename doc/cellmap/{recon_p01_render_import.md => steps/2-1_render_import.md} (100%) rename doc/cellmap/{recon_p02_alignment_prep.md => steps/2-2_alignment_prep.md} (100%) rename doc/cellmap/{recon_p03_review_matches.md => steps/2-3_review_matches.md} (100%) rename doc/cellmap/{recon_p04_alignment_solve.md => steps/2-4_alignment_solve.md} (97%) create mode 100644 doc/cellmap/steps/3_export.md create mode 100644 doc/cellmap/summary.md diff --git a/doc/cellmap/project_management.md b/doc/cellmap/project_management.md index c7424993..508dbcac 100644 --- a/doc/cellmap/project_management.md +++ b/doc/cellmap/project_management.md @@ -25,7 +25,7 @@ sequenceDiagram ``` ## Issues -For every new dataset, an issue is opened. What information should be included and what processes this triggers is detailed in [part 1 of the pipeline documentation](recon_p01_render_import.md). Here, we concentrate on the management of the issues. +For every new dataset, an issue is opened. What information should be included and what processes this triggers is detailed in [part 1 of the pipeline documentation](steps/2-1_render_import.md). Here, we concentrate on the management of the issues. Generally, GitHub is mainly used to track the progress of the processing and to communicate with the collaborators. This can be sped up by assigning the issue to the responsible person, by using labels, and by tagging people from which feedback is needed. diff --git a/doc/cellmap/steps/1_transfer.md b/doc/cellmap/steps/1_transfer.md new file mode 100644 index 00000000..1dafe5eb --- /dev/null +++ b/doc/cellmap/steps/1_transfer.md @@ -0,0 +1,5 @@ +Dump: +There is an issue template that has to be filled out to gather necessary data for subsequent processing. Usually some post-processing is necessary to make sure all necessary data are present and correct: + +- TODO: which are the necessary data and how to make sure they are correct? + diff --git a/doc/cellmap/recon_p01_render_import.md b/doc/cellmap/steps/2-1_render_import.md similarity index 100% rename from doc/cellmap/recon_p01_render_import.md rename to doc/cellmap/steps/2-1_render_import.md diff --git a/doc/cellmap/recon_p02_alignment_prep.md b/doc/cellmap/steps/2-2_alignment_prep.md similarity index 100% rename from doc/cellmap/recon_p02_alignment_prep.md rename to doc/cellmap/steps/2-2_alignment_prep.md diff --git a/doc/cellmap/recon_p03_review_matches.md b/doc/cellmap/steps/2-3_review_matches.md similarity index 100% rename from doc/cellmap/recon_p03_review_matches.md rename to doc/cellmap/steps/2-3_review_matches.md diff --git a/doc/cellmap/recon_p04_alignment_solve.md b/doc/cellmap/steps/2-4_alignment_solve.md similarity index 97% rename from doc/cellmap/recon_p04_alignment_solve.md rename to doc/cellmap/steps/2-4_alignment_solve.md index 7cc6aa91..83833cc8 100644 --- a/doc/cellmap/recon_p04_alignment_solve.md +++ b/doc/cellmap/steps/2-4_alignment_solve.md @@ -61,6 +61,6 @@ should be improved. ## Correct Alignment Issues If alignment issues are found, corrections are performed -using the [patch process](recon_p03_review_matches.md#patch-corrupted-tiles) +using the [patch process](2-3_review_matches.md#patch-corrupted-tiles) with an incremented patch version number. You then repeat the solve and review processes until you have a clean alignment. \ No newline at end of file diff --git a/doc/cellmap/steps/3_export.md b/doc/cellmap/steps/3_export.md new file mode 100644 index 00000000..e69de29b diff --git a/doc/cellmap/summary.md b/doc/cellmap/summary.md new file mode 100644 index 00000000..7d257ecd --- /dev/null +++ b/doc/cellmap/summary.md @@ -0,0 +1,24 @@ +# Render pipeline for Cellmap projects +This document describes the pipeline for processing FIB-SEM data for the Cellmap project. The scripts and tools mentioned in this document can be found at `src/scripts/cellmap`. Most of the information below is specific to the cellmap project, but the general ideas should be transferable. + +Tracking of the progress of the processing and communication with collaborators is detailed in the [project management](project_management.md) section. + +General info on how to run programs on Janelia’s compute cluster can be found on the [Janelia Compute Cluster Wiki page](https://wikis.janelia.org/display/SCS/Janelia+Compute+Cluster). There are also a few things to consider for running Spark on an LSF cluster; see the [corresponding Wiki page](https://wikis.janelia.org/display/ScientificComputing/Spark+on+LSF). + +## Pipeline overview + +The pipeline consists of the following steps: +1. [Transfer](steps/1_transfer.md) the data from the scope to the shared drive. +2. Process the data: + 1. [Render import](steps/2-1_render_import.md): Import the data into the render webservice. + 2. [Prepare alignment](steps/2-2_alignment_prep.md): Prepare the data for alignment. + 3. [Review](steps/2-3_review_matches.md): Review the matches. + 4. [Alignment](steps/2-4_alignment_solve.md): Actually align the data. +3. [Export](steps/3_export.md) the data to a final N5 volume. + +For running each of the processing and export steps, up to three actors are involved: +- A web-server running the render webservice (render-ws). Here, all meta-data is persisted in form of `TileSpec`s (which includes geometric information, paths to the image file, and transformations) and `PointMatch`es. +- The file system where the actual image files are stored. This is usually the lab’s shared drive. +- The machine where the actual computation is done. This is typically the cluster or a local workstation and it needs access to both of the above. + +In addition, for the transfer step, access to the scope and [Janelia's Jenkins server](https://jenkins.int.janelia.org) is needed. From 370c4e6f6168bd3b36e30c9de49350b3d7c793fb Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Mon, 25 Mar 2024 21:18:51 -0400 Subject: [PATCH 06/13] Update transfer documentation --- doc/cellmap/steps/1_transfer.md | 64 ++++++++++++++++++++++++++++++--- 1 file changed, 60 insertions(+), 4 deletions(-) diff --git a/doc/cellmap/steps/1_transfer.md b/doc/cellmap/steps/1_transfer.md index 1dafe5eb..a3fcd565 100644 --- a/doc/cellmap/steps/1_transfer.md +++ b/doc/cellmap/steps/1_transfer.md @@ -1,5 +1,61 @@ -Dump: -There is an issue template that has to be filled out to gather necessary data for subsequent processing. Usually some post-processing is necessary to make sure all necessary data are present and correct: +# Transfer data from scope to shared filesystem -- TODO: which are the necessary data and how to make sure they are correct? - + + +## Get necessary metadata +```mermaid +flowchart LR + github[(Issue on
GitHub)] + json(volume_transfer_info.<dataset>.json) + scope[(Scope)] + github-- Extract data --->json + json<-. Check data ..->scope + +``` +There is an issue template that has to be filled out by the FIB-SEM shared resource to gather necessary data for subsequent processing. This information should be converted to a file called `volume_transfer_info..json`, where `` is the dataset name as specified by the collaborator. This file should be put under `src/resources/transfer_info/cellmap` in this repository and committed. + +It is best to double-check the validity of the data from the GitHub issue. Sometimes, some post-processing is necessary to make sure all necessary data are present and correct. To this end, log into the scope as user `flyem` and navigate to the `root_keep_path` specified in the issue, e.g.: +```bash +su flyem +ssh jeiss7.hhmi.org +cd /cygdrive/d/UploadFlags +``` +Make sure that the following data are correct: +* `SampleID` and `root_dat_path`, which can be directly seen from the filenames listed in the directory; +* `columns_per_z_layer` and `rows_per_z_layer`, which can be inferred from the file suffix, e.g., `*_0-1-3.dat` being the highest suffix means that there are 2 rows and 4 columns. + +## Transfer and conversion +```mermaid +flowchart LR + jenkins[(Jenkins
server)] + scope[(Scope)] + prfs[(prfs)] + nrs[(nrs)] + nearline[(nearline)] + jenkins-.->|"Initiate transfer (30min)"|scope + scope-->|Raw .dat|prfs + jenkins-.->|"Initiate conversion (2h)"|prfs + prfs--->|16-bit HDF5|prfs + prfs--->|8-bit HDF5 for processing|nrs + prfs--->|move .dat after conversion|nearline +``` + +To set up transfer, make sure that `volume_transfer_info..json` is in `/groups/flyem/data/render/git/EM_recon_pipeline/config`. Then, execute +```bash +./00_setup_transfer.sh volume_transfer_info..json +``` +This will copy the json file into `/groups/flyem/home/flyem/bin/dat_transfer/2022/config`, where it can be found by the processes run by the Jenkins server. + +### Configuring and starting the Jenkins server for transfer +1. Log in to [the server](https://jenkins.int.janelia.org) and navigate to your scope under the FlyEM tab. +2. If the process is not enabled, enable it. +3. To test the process, go to the build steps of the configuration menu and select a short amount of time (e.g. 2min instead of 29min). Make sure that the test run doesn't overlap with a scheduled run (happening at XX:07 and XX:37 - note that the time in in GMT). +4. If the run is successful (check in run > console output), set the time back to 29min and save the configuration. + +The Jenkins process for conversion should always be running and happens every two hours. Note that everything resides under the FlyEM tab even if the current acquisition is done by the FIB-SEM shared resource for another collaborator, since the shared resources was initially founded for FlyEM, so the name has historic reasons. + +## Set up processing directory +To set up a directory for subsequent processing, execute `11_setup_volume.sh`. This will copy all relevant scripts for processing from this directory to a directory on `prfs` specified in `/groups/cellmap/cellmap/render/align/jrc_mus-liver-zon-3`. + +## Update GitHub issue +To automatically create a text with all relevant links for the GitHub issue, execute `gen_github_text.sh`. This text can be used to update the issue description. From 25638abaddd168f30a12ee3024b01a9306d3e01f Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Tue, 26 Mar 2024 09:57:31 -0400 Subject: [PATCH 07/13] Add sentence about storage quota --- doc/cellmap/steps/1_transfer.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/cellmap/steps/1_transfer.md b/doc/cellmap/steps/1_transfer.md index a3fcd565..fd38e162 100644 --- a/doc/cellmap/steps/1_transfer.md +++ b/doc/cellmap/steps/1_transfer.md @@ -24,6 +24,8 @@ Make sure that the following data are correct: * `SampleID` and `root_dat_path`, which can be directly seen from the filenames listed in the directory; * `columns_per_z_layer` and `rows_per_z_layer`, which can be inferred from the file suffix, e.g., `*_0-1-3.dat` being the highest suffix means that there are 2 rows and 4 columns. +This is also a good point to check if there is enough space on the collaborator's drives for the data and to increase the quota if necessary. + ## Transfer and conversion ```mermaid flowchart LR From 3e50722cdc61c541edc2835324d3b99e85ce864e Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Thu, 28 Mar 2024 13:40:24 -0400 Subject: [PATCH 08/13] Update transfer doc on @trautmane's feedback --- doc/cellmap/steps/1_transfer.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/doc/cellmap/steps/1_transfer.md b/doc/cellmap/steps/1_transfer.md index fd38e162..92a5ddd8 100644 --- a/doc/cellmap/steps/1_transfer.md +++ b/doc/cellmap/steps/1_transfer.md @@ -5,9 +5,9 @@ ## Get necessary metadata ```mermaid flowchart LR - github[(Issue on
GitHub)] - json(volume_transfer_info.<dataset>.json) - scope[(Scope)] + github(Issue on
GitHub) + json{{volume_transfer_info.<dataset>.json}} + scope(Scope) github-- Extract data --->json json<-. Check data ..->scope @@ -29,17 +29,17 @@ This is also a good point to check if there is enough space on the collaborator' ## Transfer and conversion ```mermaid flowchart LR - jenkins[(Jenkins
server)] - scope[(Scope)] - prfs[(prfs)] - nrs[(nrs)] - nearline[(nearline)] + jenkins(Jenkins
server) + scope(Scope) + prfs(prfs) + nrs(nrs) + nearline(nearline) jenkins-.->|"Initiate transfer (30min)"|scope scope-->|Raw .dat|prfs jenkins-.->|"Initiate conversion (2h)"|prfs prfs--->|16-bit HDF5|prfs prfs--->|8-bit HDF5 for processing|nrs - prfs--->|move .dat after conversion|nearline + prfs--->|move 16-bit HDF5 after conversion
remove corresponding .dat|nearline ``` To set up transfer, make sure that `volume_transfer_info..json` is in `/groups/flyem/data/render/git/EM_recon_pipeline/config`. Then, execute From 020b0f601adc88f755adffa175588917e9197d13 Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Fri, 29 Mar 2024 13:48:52 -0400 Subject: [PATCH 09/13] Review directories in transfer documentation --- doc/cellmap/steps/1_transfer.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/doc/cellmap/steps/1_transfer.md b/doc/cellmap/steps/1_transfer.md index 92a5ddd8..c3a77913 100644 --- a/doc/cellmap/steps/1_transfer.md +++ b/doc/cellmap/steps/1_transfer.md @@ -1,6 +1,11 @@ # Transfer data from scope to shared filesystem - +```bash +# Commonly used paths +EMRP_ROOT="/groups/flyem/data/render/git/EM_recon_pipeline" +TRANSFER_INFO_DIR="src/resources/transfer_info/cellmap" +TRANSFER_DIR="/groups/flyem/home/flyem/bin/dat_transfer/2022/config" +``` ## Get necessary metadata ```mermaid @@ -12,7 +17,7 @@ flowchart LR json<-. Check data ..->scope ``` -There is an issue template that has to be filled out by the FIB-SEM shared resource to gather necessary data for subsequent processing. This information should be converted to a file called `volume_transfer_info..json`, where `` is the dataset name as specified by the collaborator. This file should be put under `src/resources/transfer_info/cellmap` in this repository and committed. +There is an issue template that has to be filled out by the FIB-SEM shared resource to gather necessary data for subsequent processing. This information should be converted to a file called `volume_transfer_info..json`, where `` is the dataset name as specified by the collaborator. This file should be put under `${TRANSFER_INFO_DIR}` in this repository and committed. It is best to double-check the validity of the data from the GitHub issue. Sometimes, some post-processing is necessary to make sure all necessary data are present and correct. To this end, log into the scope as user `flyem` and navigate to the `root_keep_path` specified in the issue, e.g.: ```bash @@ -41,12 +46,11 @@ flowchart LR prfs--->|8-bit HDF5 for processing|nrs prfs--->|move 16-bit HDF5 after conversion
remove corresponding .dat|nearline ``` - -To set up transfer, make sure that `volume_transfer_info..json` is in `/groups/flyem/data/render/git/EM_recon_pipeline/config`. Then, execute +To set up transfer, make sure that `volume_transfer_info..json` is in `${EMPR_ROOT}/${TRANSFER_INFO_DIR}`. Then, go to that directory and execute ```bash ./00_setup_transfer.sh volume_transfer_info..json ``` -This will copy the json file into `/groups/flyem/home/flyem/bin/dat_transfer/2022/config`, where it can be found by the processes run by the Jenkins server. +This will copy the json file into `${TRANSFER_DIR}/config`, where it can be found by the processes run by the Jenkins server. ### Configuring and starting the Jenkins server for transfer 1. Log in to [the server](https://jenkins.int.janelia.org) and navigate to your scope under the FlyEM tab. @@ -57,7 +61,7 @@ This will copy the json file into `/groups/flyem/home/flyem/bin/dat_transfer/202 The Jenkins process for conversion should always be running and happens every two hours. Note that everything resides under the FlyEM tab even if the current acquisition is done by the FIB-SEM shared resource for another collaborator, since the shared resources was initially founded for FlyEM, so the name has historic reasons. ## Set up processing directory -To set up a directory for subsequent processing, execute `11_setup_volume.sh`. This will copy all relevant scripts for processing from this directory to a directory on `prfs` specified in `/groups/cellmap/cellmap/render/align/jrc_mus-liver-zon-3`. +To set up a directory for subsequent processing, execute `11_setup_volume.sh` in `${TRANSFER_DIR}`. This will copy all relevant scripts for processing from this directory to a directory on `prfs` specified in `volume_transfer_info..json`. ## Update GitHub issue -To automatically create a text with all relevant links for the GitHub issue, execute `gen_github_text.sh`. This text can be used to update the issue description. +To automatically create a text with all relevant links for the GitHub issue, execute `gen_github_text.sh` in the processing directory. This text can be used to update the issue description. From 58af52436585669257f4df2c811d7aaa49b872e8 Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Fri, 29 Mar 2024 14:42:21 -0400 Subject: [PATCH 10/13] Update render import documentation --- doc/cellmap/steps/2-1_render_import.md | 121 +++---------------------- 1 file changed, 14 insertions(+), 107 deletions(-) diff --git a/doc/cellmap/steps/2-1_render_import.md b/doc/cellmap/steps/2-1_render_import.md index 88b67fe9..3584af69 100644 --- a/doc/cellmap/steps/2-1_render_import.md +++ b/doc/cellmap/steps/2-1_render_import.md @@ -1,118 +1,25 @@ # Reconstruction Part 1: Render Import -## Common Parameters ```bash -export NEARLINE_HOST="e06u14.int.janelia.org" -export BASE_WORK_DIR="/groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m" +# Commonly used paths +RENDER_DIR="/groups/flyem/data/render/" +PROCESSING_DIR= ``` -## Setup Work Area -```bash -# Run on host with access to /nearline because that is where .dat files are stored. -ssh ${NEARLINE_HOST} - -cd ${BASE_WORK_DIR} - -# Create work area ${BASE_WORK_DIR}/${REGION}/${TAB} with copies of pipeline scripts. -# -# USAGE: ./setup.sh ( e.g. ./setup.sh VNC Sec32 ) -./setup.sh VNC Sec26 -``` -## Validate and Correct Scope File Transfers -VNC Sec26 Processing Time: 15 minutes using 1 core *(can take much longer if there are many transfer issues)* - -The Fly EM image transfer process continuously polls the EM scopes for new data -and then copies (and converts) that data to centralized network filesystems. -The process works most of the time, but occasionally files are not transferred. -It is important to ensure that all data has been properly transferred -before deriving and importing metadata into the Render web services. +We assume that the data has been successfully transferred and converted to HDF5 format, and that there is a directory set up for processing which we call `PROCESSING_DIR` (see [transfer documentation](1_transfer.md) for details). The first step for processing is to import the data into the Render database so that it can be accessed by the render clients. -### Transfer Process (for context) -![FIBSEM transfer flow diagram](fibsem-transfer-flow.png) - -### Validation Steps +## Generate TileSpecs +Go to a machine with access to `${RENDER_DIR}`, navigate to `${PROCESSING_DIR}` and run the following command: ```bash -# While still on nearline host ... -cd VNC/Sec26 # cd ${BASE_WORK_DIR}/${REGION}/${TAB} - -# Run as flyem user so that scopes can be reached and corrections can be written. -su flyem - -# Use date range from transferred files to go back to scope and get listing of -# all files acquired during that time. -./01_gen_scope_dat_file_lists.sh - -# Look for missing and/or misplaced files. -# InLens png, Transferred dat, Archived dat, Scope dat, and Logs counts should be the same. -# Nothing should be missing. -./02_check_tabs.sh - -| ==================================================== -| Tab: Z0720-07m_VNC_Sec26 -| -| InLens png Transferred dat Archived dat Scope dat Logs -| ---------- --------------- ------------ --------- --------- -| 119187 119187 119187 119208 119187 -| -| 21 missing InLens png: ['Merlin-6285_21-08-23_084156_0-0-0', 'Merlin-6285_21-08-23_084156_0-0-1', 'Merlin-6285_2 -| 21 missing dat: ['Merlin-6285_21-08-23_084156_0-0-0', 'Merlin-6285_21-08-23_084156_0-0-1', 'Merlin-6285_2 -| 0 unarchived dat: [] -| 0 non-standard dat: [] - -# In this example, 21 dat files (along with their corresponding png files) are missing -# so we need to try to transfer the dat files again. -# This script serially scp's each missing dat file and converts it to png. -./03_fetch_missing_dat.sh - -# Re-run the check: nothing should be missing now unless the source files are corrupted in some way. -./02_check_tabs.sh - -| ==================================================== -| Tab: Z0720-07m_VNC_Sec26 -| -| InLens png Transferred dat Archived dat Scope dat Logs -| ---------- --------------- ------------ --------- --------- -| 119208 119208 119208 119208 119208 -| -| 0 missing InLens png: [] -| 0 missing dat: [] -| 0 unarchived dat: [] -| 0 non-standard dat: [] - -# Verify the total dat file count and the first and last dat file names -# match what Wei thinks they should be. Wei typically emails the names and counts -# but sometimes you need to ask Wei to send the information. -# -# TODO: merge this information into the check tabs tool -./list_first_and_last_dats.sh - -# Once everything looks good, exit from the flyem shell. -exit +./07_h5_to_render.sh ``` +This will launch a local dask job that will generate TileSpecs from the HDF5 files and upload them to the render database. All necessary information is read from the `volume_transfer_info..json` file that was created in the previous step. After this step, a dynamically rendered stack can be accessed in the point match explorer and viewed in neuroglancer. -### Move Bad dat Files +## Set up preview volume: export to N5 +NOTE: this is a feature under active development and the process will likely change in the near future. For now, the following steps are necessary. +Go to the Janelia compute cluster (e.g., `ssh login1.int.janelia.org`), navigate to `${PROCESSING_DIR}` and run the following command: ```bash -# If dat files are corrupted such that png files cannot be generated for them ... -NEARLINE_TAB_DIR="/nearline/flyem2/data/Z0720-07m_${REGION}_${TAB}" -BAD_DAT_FILE="???" -mkdir ${NEARLINE_TAB_DIR}/dat_bad -mv ${NEARLINE_TAB_DIR}/dat/${BAD_DAT_FILE} ${NEARLINE_TAB_DIR}/dat_bad - -# Note that the render import process will automatically -# patch partial layers with tiles from complete adjacent layers. +./99_append_to_export.sh ``` +This will submit a couple of spark cluster jobs and set up logging directories. The number of executors should be chosen based on the size of the dataset. Currently, a single executor will occupy 10 cores on the cluster and 3 more cores for the driver are needed. The logs for scripts usually reside in `${PROCESSING_DIR}/logs`, but the spark jobs will set up additional log directories for each executor and the driver. These directories are printed to the console when executing above command. -## Import Metadata into Render Web Services -VNC Sec26 Processing Time: 2 minutes using 32 cores - -```bash -# While still on nearline host in ${BASE_WORK_DIR}/${REGION}/${TAB} and as yourself ... - -# Run script that parses dat headers, generates JSON tile specs, and imports them into render. -# Current process sets up a 32-worker dask cluster to do the import, so you may need to -# reconfigure the number of dask workers if your nearline host does not have enough cores. -./06_dat_to_render.sh - -# Once import completes everything else needs to launch from an LSF submit host, -# so exit from the nearline host. -exit -``` +Once fininshed, the location of the final N5 volume can be found either in the driver logs or by running `gen_github_text.sh` again. From 688c736a2b3f94a5fab6cf5011b4097f52a5a7a0 Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Mon, 15 Apr 2024 12:02:12 -0400 Subject: [PATCH 11/13] Review documentation based on existing dataset --- doc/cellmap/steps/1_transfer.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/cellmap/steps/1_transfer.md b/doc/cellmap/steps/1_transfer.md index c3a77913..8adba829 100644 --- a/doc/cellmap/steps/1_transfer.md +++ b/doc/cellmap/steps/1_transfer.md @@ -4,7 +4,7 @@ # Commonly used paths EMRP_ROOT="/groups/flyem/data/render/git/EM_recon_pipeline" TRANSFER_INFO_DIR="src/resources/transfer_info/cellmap" -TRANSFER_DIR="/groups/flyem/home/flyem/bin/dat_transfer/2022/config" +TRANSFER_DIR="/groups/flyem/home/flyem/bin/dat_transfer/2022" ``` ## Get necessary metadata @@ -17,7 +17,7 @@ flowchart LR json<-. Check data ..->scope ``` -There is an issue template that has to be filled out by the FIB-SEM shared resource to gather necessary data for subsequent processing. This information should be converted to a file called `volume_transfer_info..json`, where `` is the dataset name as specified by the collaborator. This file should be put under `${TRANSFER_INFO_DIR}` in this repository and committed. +There is an issue template that has to be filled out by the FIB-SEM shared resource to gather necessary data for subsequent processing. This information should be converted to a file called `volume_transfer_info..json`, where `` is the dataset name as specified by the collaborator. No dashes should be in the project name since MongoDB doesn't like them that much. This file should be put under `${TRANSFER_INFO_DIR}` in your own copy of this repository and committed. It is best to double-check the validity of the data from the GitHub issue. Sometimes, some post-processing is necessary to make sure all necessary data are present and correct. To this end, log into the scope as user `flyem` and navigate to the `root_keep_path` specified in the issue, e.g.: ```bash @@ -26,7 +26,7 @@ ssh jeiss7.hhmi.org cd /cygdrive/d/UploadFlags ``` Make sure that the following data are correct: -* `SampleID` and `root_dat_path`, which can be directly seen from the filenames listed in the directory; +* `SampleID` and `root_dat_path`, which can be directly seen from the filenames listed in the directory; the `root_dat_path` should not have any timestamp in it; * `columns_per_z_layer` and `rows_per_z_layer`, which can be inferred from the file suffix, e.g., `*_0-1-3.dat` being the highest suffix means that there are 2 rows and 4 columns. This is also a good point to check if there is enough space on the collaborator's drives for the data and to increase the quota if necessary. @@ -46,16 +46,16 @@ flowchart LR prfs--->|8-bit HDF5 for processing|nrs prfs--->|move 16-bit HDF5 after conversion
remove corresponding .dat|nearline ``` -To set up transfer, make sure that `volume_transfer_info..json` is in `${EMPR_ROOT}/${TRANSFER_INFO_DIR}`. Then, go to that directory and execute +To set up transfer, make sure that `volume_transfer_info..json` is in `${EMPR_ROOT}/${TRANSFER_INFO_DIR}`. Then, go to `${TRANSFER_DIR}` and execute ```bash -./00_setup_transfer.sh volume_transfer_info..json +./00_setup_transfer.sh cellmap/volume_transfer_info..json ``` This will copy the json file into `${TRANSFER_DIR}/config`, where it can be found by the processes run by the Jenkins server. ### Configuring and starting the Jenkins server for transfer 1. Log in to [the server](https://jenkins.int.janelia.org) and navigate to your scope under the FlyEM tab. 2. If the process is not enabled, enable it. -3. To test the process, go to the build steps of the configuration menu and select a short amount of time (e.g. 2min instead of 29min). Make sure that the test run doesn't overlap with a scheduled run (happening at XX:07 and XX:37 - note that the time in in GMT). +3. To test the process, go to the build steps of the configuration menu and select a short amount of time (e.g. 2min instead of 29min) and hit `Build now`. Make sure that the test run doesn't overlap with a scheduled run (happening every 30min - look at past runs and note that the time is in GMT). 4. If the run is successful (check in run > console output), set the time back to 29min and save the configuration. The Jenkins process for conversion should always be running and happens every two hours. Note that everything resides under the FlyEM tab even if the current acquisition is done by the FIB-SEM shared resource for another collaborator, since the shared resources was initially founded for FlyEM, so the name has historic reasons. From 7e59f3629e13947e2c057896136010b832b70375 Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Mon, 15 Apr 2024 12:14:37 -0400 Subject: [PATCH 12/13] Add sentence about processing github issue --- doc/cellmap/steps/1_transfer.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/cellmap/steps/1_transfer.md b/doc/cellmap/steps/1_transfer.md index 8adba829..482f3c45 100644 --- a/doc/cellmap/steps/1_transfer.md +++ b/doc/cellmap/steps/1_transfer.md @@ -65,3 +65,4 @@ To set up a directory for subsequent processing, execute `11_setup_volume.sh` in ## Update GitHub issue To automatically create a text with all relevant links for the GitHub issue, execute `gen_github_text.sh` in the processing directory. This text can be used to update the issue description. +Furthermore, assign the issue to the person responsible for the next steps, add the right labels, and add the issue to the project board. From d232afa1bdb73860f3083248c1980a1884ff731a Mon Sep 17 00:00:00 2001 From: Michael Innerberger Date: Tue, 14 May 2024 19:38:05 -0400 Subject: [PATCH 13/13] Add some minor details --- doc/cellmap/steps/1_transfer.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/cellmap/steps/1_transfer.md b/doc/cellmap/steps/1_transfer.md index 482f3c45..d84dc600 100644 --- a/doc/cellmap/steps/1_transfer.md +++ b/doc/cellmap/steps/1_transfer.md @@ -26,10 +26,10 @@ ssh jeiss7.hhmi.org cd /cygdrive/d/UploadFlags ``` Make sure that the following data are correct: -* `SampleID` and `root_dat_path`, which can be directly seen from the filenames listed in the directory; the `root_dat_path` should not have any timestamp in it; +* `data_set_id` and `root_dat_path`, which can be directly seen from the filenames listed in the directory; the `root_dat_path` should not have any timestamp in it; * `columns_per_z_layer` and `rows_per_z_layer`, which can be inferred from the file suffix, e.g., `*_0-1-3.dat` being the highest suffix means that there are 2 rows and 4 columns. -This is also a good point to check if there is enough space on the collaborator's drives for the data and to increase the quota if necessary. +This is also a good point to check if there is enough space on the collaborator's drives for the data and to increase the quota if necessary. The correct group for billing can be found with `lsfgroup `. ## Transfer and conversion ```mermaid