diff --git a/doc/cellmap/fibsem-transfer-flow.md b/doc/cellmap/fibsem-transfer-flow.md index ea701468..94f042de 100644 --- a/doc/cellmap/fibsem-transfer-flow.md +++ b/doc/cellmap/fibsem-transfer-flow.md @@ -1,13 +1,13 @@ ```mermaid graph TD - SCOPE_DAT_A[jeiss scope
z1_0-0-0.dat, z1_0-0-1.dat,
z1_0-1-0.dat, z1_0-1-1.dat] --> TRANSFER_DAT - SCOPE_DAT_B[jeiss scope
zn_0-0-0.dat, zn_0-0-1.dat,
zn_0-1-0.dat, zn_0-1-1.dat] --> TRANSFER_DAT - TRANSFER_DAT{transfer} --> DM11_DAT_A & DM11_DAT_B - DM11_DAT_A[dm11
z1_0-0-0.dat, z1_0-0-1.dat,
z1_0-1-0.dat, z1_0-1-1.dat ] --> DAT_TO_H5 - DM11_DAT_B[dm11
zn_0-0-0.dat, zn_0-0-1.dat
zn_0-1-0.dat, zn_0-1-1.dat ] --> DAT_TO_H5 - DAT_TO_H5{convert and
remove dm11 dats
after verification} --> DM11_RAW_H5 & NRS_ALIGN_H5 - DM11_RAW_H5[dm11
z1.raw.h5, ...
zn.raw.h5 ] --> ARCHIVE_H5 - NRS_ALIGN_H5[nrs
z1.uint8.h5, ...
zn.uint8.h5] - ARCHIVE_H5{archive and
remove dm11 raw h5s
after verification} --> NEARLINE_RAW_H5 - NEARLINE_RAW_H5[nearline
z1.raw-archive.h5, ...
zn.raw-archive.h5] + SCOPE_DAT_A(jeiss scope
z1_0-0-0.dat, z1_0-0-1.dat,
z1_0-1-0.dat, z1_0-1-1.dat) --> TRANSFER_DAT + SCOPE_DAT_B(jeiss scope
zn_0-0-0.dat, zn_0-0-1.dat,
zn_0-1-0.dat, zn_0-1-1.dat) --> TRANSFER_DAT + TRANSFER_DAT{{transfer}} --> DM11_DAT_A & DM11_DAT_B + DM11_DAT_A(prfs
z1_0-0-0.dat, z1_0-0-1.dat,
z1_0-1-0.dat, z1_0-1-1.dat ) --> DAT_TO_H5 + DM11_DAT_B(prfs
zn_0-0-0.dat, zn_0-0-1.dat
zn_0-1-0.dat, zn_0-1-1.dat ) --> DAT_TO_H5 + DAT_TO_H5{{ convert and
remove prfs dats
after verification }} --> DM11_RAW_H5 & NRS_ALIGN_H5 + DM11_RAW_H5(prfs
z1.raw.h5, ...
zn.raw.h5 ) --> ARCHIVE_H5 + NRS_ALIGN_H5(nrs
z1.uint8.h5, ...
zn.uint8.h5) + ARCHIVE_H5{{archive and
remove prfs raw h5s
after verification}} --> NEARLINE_RAW_H5 + NEARLINE_RAW_H5(nearline
z1.raw-archive.h5, ...
zn.raw-archive.h5) ``` \ No newline at end of file diff --git a/doc/cellmap/project_management.md b/doc/cellmap/project_management.md new file mode 100644 index 00000000..508dbcac --- /dev/null +++ b/doc/cellmap/project_management.md @@ -0,0 +1,49 @@ +# Project management +All projects are tracked on the [recon_fibsem repository](https://github.com/JaneliaSciComp/recon_fibsem) on GitHub. +This repository is used for documenting and exchanging the information necessary to ingest and process new datasets, tracking the progress of the processing, and reviewing the processed datasets. + +```mermaid +sequenceDiagram + participant F as FIB-SEM
shared resource + participant S as SciComp + participant G as GitHub + participant C as Collaborator + + F->>G: Create new issue + G-->>S: Notify + activate S + S->>G: Include in project board + Note over S: Ingestion and processing + loop Review + S->>G: Post updates and questions + G-->>C: Notify + C->>G: Review and feedback + G-->>S: Notify + end + S->>C: Finalize and hand off + deactivate S +``` + +## Issues +For every new dataset, an issue is opened. What information should be included and what processes this triggers is detailed in [part 1 of the pipeline documentation](steps/2-1_render_import.md). Here, we concentrate on the management of the issues. + +Generally, GitHub is mainly used to track the progress of the processing and to communicate with the collaborators. This can be sped up by assigning the issue to the responsible person, by using labels, and by tagging people from which feedback is needed. + +### Labels + +There are three different categories of labels: + +- **Workflow labels**: `00: imaging in progress`, `01: prep done`, `02: alignment done`, `03: alignment review done`, `04: z correction done`, `05: streak correction done`, `06: intensity correction done`, `07: straightening done`, `09: clean-up done` +- **Collaborator labels**: `Cell Map`, `eFIB-SEM SR` (the FIB-SEM shared resource), `Fly EM` +- **TODO labels**: `needs alignment`, `needs clean-up`, `needs cross-correlation`, `needs N5 creation`, `needs review`, `needs straightening`, `needs streak correction`, `non-rigid tile deformations`, `alignment review concerns` + +Each dataset should have the right collaborator label. Apart from that, there are no strict rules about labeling. Use the TODO labels as needed to keep track of what needs to be done. The workflow labels are used to document the processing steps that have been done. When a dataset is done processing, usually all workflow labels below 06 are removed, as they are recorded in the issue history anyway. + +### Project board + +All issues are collected on the [project board](https://github.com/JaneliaSciComp/recon_fibsem/projects/1), which has four columns: + +- **Alignment**: All new issues are automatically entered in this column and stay there during the all standard processing steps. +- **Review**: Once processing is done, the issue is moved to this column, where Preibisch and the collaborators have a final look at it. If further processing is necessary, the issue stays in this column for the time necessary to address all issues. +- **Done**: Once the dataset passes the review process, it is moved to this column to signal that we don’t do further processing on the dataset. The dataset is now ready to be handed off to the collaborator. +- **Cleaned Up**: Once the collaborator has taken over the dataset and has indicated that no further processing is necessary, the issue is moved to this column. At this stage all intermediate data that was used for processing (including 8bit images for alignment) are deleted. This clean up is only done irregularly if disk space is scarce. \ No newline at end of file diff --git a/doc/cellmap/steps/1_transfer.md b/doc/cellmap/steps/1_transfer.md new file mode 100644 index 00000000..d84dc600 --- /dev/null +++ b/doc/cellmap/steps/1_transfer.md @@ -0,0 +1,68 @@ +# Transfer data from scope to shared filesystem + +```bash +# Commonly used paths +EMRP_ROOT="/groups/flyem/data/render/git/EM_recon_pipeline" +TRANSFER_INFO_DIR="src/resources/transfer_info/cellmap" +TRANSFER_DIR="/groups/flyem/home/flyem/bin/dat_transfer/2022" +``` + +## Get necessary metadata +```mermaid +flowchart LR + github(Issue on
GitHub) + json{{volume_transfer_info.<dataset>.json}} + scope(Scope) + github-- Extract data --->json + json<-. Check data ..->scope + +``` +There is an issue template that has to be filled out by the FIB-SEM shared resource to gather necessary data for subsequent processing. This information should be converted to a file called `volume_transfer_info..json`, where `` is the dataset name as specified by the collaborator. No dashes should be in the project name since MongoDB doesn't like them that much. This file should be put under `${TRANSFER_INFO_DIR}` in your own copy of this repository and committed. + +It is best to double-check the validity of the data from the GitHub issue. Sometimes, some post-processing is necessary to make sure all necessary data are present and correct. To this end, log into the scope as user `flyem` and navigate to the `root_keep_path` specified in the issue, e.g.: +```bash +su flyem +ssh jeiss7.hhmi.org +cd /cygdrive/d/UploadFlags +``` +Make sure that the following data are correct: +* `data_set_id` and `root_dat_path`, which can be directly seen from the filenames listed in the directory; the `root_dat_path` should not have any timestamp in it; +* `columns_per_z_layer` and `rows_per_z_layer`, which can be inferred from the file suffix, e.g., `*_0-1-3.dat` being the highest suffix means that there are 2 rows and 4 columns. + +This is also a good point to check if there is enough space on the collaborator's drives for the data and to increase the quota if necessary. The correct group for billing can be found with `lsfgroup `. + +## Transfer and conversion +```mermaid +flowchart LR + jenkins(Jenkins
server) + scope(Scope) + prfs(prfs) + nrs(nrs) + nearline(nearline) + jenkins-.->|"Initiate transfer (30min)"|scope + scope-->|Raw .dat|prfs + jenkins-.->|"Initiate conversion (2h)"|prfs + prfs--->|16-bit HDF5|prfs + prfs--->|8-bit HDF5 for processing|nrs + prfs--->|move 16-bit HDF5 after conversion
remove corresponding .dat|nearline +``` +To set up transfer, make sure that `volume_transfer_info..json` is in `${EMPR_ROOT}/${TRANSFER_INFO_DIR}`. Then, go to `${TRANSFER_DIR}` and execute +```bash +./00_setup_transfer.sh cellmap/volume_transfer_info..json +``` +This will copy the json file into `${TRANSFER_DIR}/config`, where it can be found by the processes run by the Jenkins server. + +### Configuring and starting the Jenkins server for transfer +1. Log in to [the server](https://jenkins.int.janelia.org) and navigate to your scope under the FlyEM tab. +2. If the process is not enabled, enable it. +3. To test the process, go to the build steps of the configuration menu and select a short amount of time (e.g. 2min instead of 29min) and hit `Build now`. Make sure that the test run doesn't overlap with a scheduled run (happening every 30min - look at past runs and note that the time is in GMT). +4. If the run is successful (check in run > console output), set the time back to 29min and save the configuration. + +The Jenkins process for conversion should always be running and happens every two hours. Note that everything resides under the FlyEM tab even if the current acquisition is done by the FIB-SEM shared resource for another collaborator, since the shared resources was initially founded for FlyEM, so the name has historic reasons. + +## Set up processing directory +To set up a directory for subsequent processing, execute `11_setup_volume.sh` in `${TRANSFER_DIR}`. This will copy all relevant scripts for processing from this directory to a directory on `prfs` specified in `volume_transfer_info..json`. + +## Update GitHub issue +To automatically create a text with all relevant links for the GitHub issue, execute `gen_github_text.sh` in the processing directory. This text can be used to update the issue description. +Furthermore, assign the issue to the person responsible for the next steps, add the right labels, and add the issue to the project board. diff --git a/doc/cellmap/steps/2-1_render_import.md b/doc/cellmap/steps/2-1_render_import.md new file mode 100644 index 00000000..3584af69 --- /dev/null +++ b/doc/cellmap/steps/2-1_render_import.md @@ -0,0 +1,25 @@ +# Reconstruction Part 1: Render Import +```bash +# Commonly used paths +RENDER_DIR="/groups/flyem/data/render/" +PROCESSING_DIR= +``` +We assume that the data has been successfully transferred and converted to HDF5 format, and that there is a directory set up for processing which we call `PROCESSING_DIR` (see [transfer documentation](1_transfer.md) for details). The first step for processing is to import the data into the Render database so that it can be accessed by the render clients. + +## Generate TileSpecs +Go to a machine with access to `${RENDER_DIR}`, navigate to `${PROCESSING_DIR}` and run the following command: +```bash +./07_h5_to_render.sh +``` +This will launch a local dask job that will generate TileSpecs from the HDF5 files and upload them to the render database. All necessary information is read from the `volume_transfer_info..json` file that was created in the previous step. After this step, a dynamically rendered stack can be accessed in the point match explorer and viewed in neuroglancer. + +## Set up preview volume: export to N5 +NOTE: this is a feature under active development and the process will likely change in the near future. For now, the following steps are necessary. + +Go to the Janelia compute cluster (e.g., `ssh login1.int.janelia.org`), navigate to `${PROCESSING_DIR}` and run the following command: +```bash +./99_append_to_export.sh +``` +This will submit a couple of spark cluster jobs and set up logging directories. The number of executors should be chosen based on the size of the dataset. Currently, a single executor will occupy 10 cores on the cluster and 3 more cores for the driver are needed. The logs for scripts usually reside in `${PROCESSING_DIR}/logs`, but the spark jobs will set up additional log directories for each executor and the driver. These directories are printed to the console when executing above command. + +Once fininshed, the location of the final N5 volume can be found either in the driver logs or by running `gen_github_text.sh` again. diff --git a/doc/cellmap/steps/2-2_alignment_prep.md b/doc/cellmap/steps/2-2_alignment_prep.md new file mode 100644 index 00000000..51ea411c --- /dev/null +++ b/doc/cellmap/steps/2-2_alignment_prep.md @@ -0,0 +1,177 @@ +# Reconstruction Part 2: Alignment Preparation +## Common Parameters +```bash +export BSUB_HOST="login1.int.janelia.org" +export BASE_WORK_DIR="/groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m" +export WORK_DIR="${BASE_WORK_DIR}/VNC/Sec26" # more generically: ${BASE_WORK_DIR}/${REGION}/${TAB} +``` +## Validate Tile Specs +VNC Sec26 Processing Time: 1 minute using 10 cores + +```bash +# Run on LSF / bsub submit host because most work will be done using LSF cluster. +ssh ${BSUB_HOST} +cd ${WORK_DIR} + +# Setup work area and parameters for LSF array job to validate recently imported render tile specs. +./07_gen_validate_spec_run.sh + +| Retrieving z values from http://10.40.3.162:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v1_acquire/zValues +| +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_060707_37_ValidateTiles/job_specific_parameters.txt +| .......... +| +| Common parameters for array job are: +| --baseDataUrl http://10.40.3.162:8080/render-ws/v1 --owner Z0720_07m_VNC --project Sec26 --stack v1_acquire --validatorClass org.janelia.alignment.spec.validator.TemTileSpecValidator --validatorData minCoordinate:-999999,maxCoordinate:999999,minSize:1000,maxSize:99999 +| +| Created bsub array script for 10 jobs (z or z batches) in: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_060707_37_ValidateTiles/bsub-array.sh +| +| Logs will be written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_060707_37_ValidateTiles/logs + +# If common parameters look ok (they should), run submit script (full path is output above): +run_20211217_060707_37_ValidateTiles/bsub-array.sh + +| This job will be billed to flyem +| Job <113403049> is submitted to default queue . +| This job will be billed to flyem +| Job <113403050> is submitted to default queue . +| This job will be billed to flyem +| Job <113403051> is submitted to default queue . + +# The submit script launches 3 chained jobs: +# 1. The first one runs just the first of the array job inputs [1-1]. +# 2. If that succeeds, the second job runs the remaining array job inputs [2-10]. +# 3. When the second job group completes, the third job runs a log check script. + +# Output from the log check script is emailed to you (the job launcher) and looks like this: + +| Job 113403051: in cluster Done +| ... +| The output (if any) follows: +| All log files in /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_060707_37_ValidateTiles/logs look okay. +``` + +## Generate Mipmaps +VNC Sec26 Processing Time: 17 minutes using 1,036 cores + +```bash +# Setup work area and parameters for LSF array job to generate down-sampled mipmaps for all source images. +./08_gen_mipmap_run.sh + +# Output (including full path to submit script) is similar to validation example above. + +# If common parameters look ok (they should), run submit script (full path is gen script output): +run_20211217_063059_153_Mipmap/bsub-array.sh + +# Output and 3-chained-job pattern is similar to validation example above. +# As with validation, output from the log check script is emailed to you. +``` + +## Generate Tile Pairs +VNC Sec26 Processing Time: 1 minute using 1 core + +```bash +# Create tile pair metadata (JSON files) to be used as input for match. +# This java process runs directly on the submit host which technically you are not supposed to do. +# The process is relatively short (a minute or so) and has never been an issue, so I just run it on the submit host. +# If anyone ever complains, it is easy enough to use an interactive node (busb -i) and run there. +./11_gen_new_pairs.sh + +| ------------------------------------------------------------------------ +| Setting up montage pairs (log is captured so wait for results) ... +| +| MAX_PAIRS_PER_FILE was set to 50 +| +| Full pair generation output written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/tile_pairs-20211217_071324.log +| ... +| ------------------------------------------------------------------------ +| Setting up cross pairs (log is captured so wait for results) ... +| ... +| 07:14:59.399 [main] INFO [org.janelia.alignment.util.FileUtil] saveJsonFile: exit, wrote data to /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/pairs_cross/tile_pairs_cross_p11919.json.gz +| 07:14:59.399 [main] INFO [org.janelia.render.client.TilePairClient] deriveAndSaveSortedNeighborPairs: exit, saved 715143 total pairs +| 07:14:59.399 [main] INFO [org.janelia.render.client.ClientRunner] run: exit, processing completed in 0 hours, 0 minutes, 57 seconds +``` + +## Generate Match Data +VNC Sec26 Processing Time: 75 minutes using 13,787 cores + +```bash +# Setup work areas and parameters for LSF array jobs to generate same layer and cross layer matches. +./12_gen_staged_match_run.sh + +| ------------------------------------------------------------------------ +| Setting up job for run_20211217_071528_16_multi_stage_match_montage ... +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/common_parameters.txt +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/job_specific_parameters.txt +| .................. +| Created bsub array script for 1867 jobs in /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/bsub-array.sh +| Logs will be written to /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/logs +| +| ------------------------------------------------------------------------ +| Setting up job for run_20211217_071529_994_multi_stage_match_cross ... +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/common_parameters.txt +| Generating /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/job_specific_parameters.txt +| ....................................................................................................................... +| Created bsub array script for 11920 jobs in /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/bsub-array.sh +| Logs will be written to /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/logs + +# Run both montage and cross bsub-array.sh scripts (full paths are in gen script output). +run_20211217_071528_16_multi_stage_match_montage/bsub-array.sh +run_20211217_071529_994_multi_stage_match_cross/bsub-array.sh + +# As with previous array jobs, output from a log check script is emailed to you +# for both the same layer (montage) and cross layer match processes. + +# The check script provides summary statistics for each stage of ... + +# the montage multi-stage process: + +| Stage parameters loaded from /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071528_16_multi_stage_match_montage/stage_parameters.montage.json +| ......... +| [stage name]_ +| SIFT SIFT Combined Combined SIFT_s[render scale]e[epsilon]_i[inliers]_c[coverage]pct_ +| Total Total Percent SIFT Combined Poor Poor Poor Poor GEO_s[render scale]e[epsilon]_i[combined inliers]_c[combined coverage]pct_ +| Stage Processed Saved Saved Saved Saved Quantity Coverage Quantity Coverage Slug: d[delta z]_h[parameter hash] +| --------------- --------- --------- --------- --------- --------- --------- --------- --------- --------- ----------------------------------------------------------------------------------------------------------- +| montagePass1 93341 73064 78.3 73064 0 17870 2407 0 0 montagePass1_SIFT_s0.40e02_i025_c070pct_GEO_none----_----_-------_d000_h8da6a90ff657806ebc8f4b9ff86f3340 +| montagePass2 20277 608 3.0 582 26 15703 0 3852 114 montagePass2_SIFT_s0.50e02_i025_c070pct_GEO_s1.00e05_i025_c070pct_d000_h83331cf1918bd1cd1235c1093e71caeb +| montagePass3 19669 1155 5.9 1111 44 0 0 16494 2020 montagePass3_SIFT_s0.60e03_i025_c070pct_GEO_s1.00e05_i000_c060pct_d000_ha8feba4c49ed328a2d1ac7decdb756b6 +| montagePass4 18514 12 0.1 0 12 0 0 16279 2223 montagePass4_SIFT_s0.60e03_i025_c070pct_GEO_s1.00e05_i000_c060pct_d000_hf170feeae225c5014591c47b5567613b +| montagePass5 18502 1296 7.0 1128 168 0 0 16162 1044 montagePass5_SIFT_s0.60e03_i012_c050pct_GEO_s1.00e03_i000_c050pct_d000_h118513a845169760a041936c8ff80f70 +| montagePass6 17206 3190 18.5 3190 0 14016 0 0 0 montagePass6_SIFT_s1.00e05_i012_c000pct_GEO_none----_----_-------_d000_hb1cf242963c833a4b1f1e05f2c673e91 +| +| processed_pair_count: 93341 +| saved_pair_count: 79325 +| average_seconds_per_pair: 13.26 +| +| jobs: 1867 +| average_minutes_per_job: 11.05 +| max_job_minutes: 71.93 + +# and the cross multi-stage process: + +| Stage parameters loaded from /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/run_20211217_071529_994_multi_stage_match_cross/stage_parameters.cross.json +| ........................................................... +| [stage name]_ +| SIFT SIFT Combined Combined SIFT_s[render scale]e[epsilon]_i[inliers]_c[coverage]pct_ +| Total Total Percent SIFT Combined Poor Poor Poor Poor GEO_s[render scale]e[epsilon]_i[combined inliers]_c[combined coverage]pct_ +| Stage Processed Saved Saved Saved Saved Quantity Coverage Quantity Coverage Slug: d[delta z]_h[parameter hash] +| --------------- --------- --------- --------- --------- --------- --------- --------- --------- --------- ----------------------------------------------------------------------------------------------------------- +| crossPass1 715143 156530 21.9 156530 0 558613 0 0 0 crossPass1_SIFT_s0.05e00_i020_c050pct_GEO_s0.10e00_i150_c050pct_d006_h7c57eb6f14fd7006563010950b1a8c4c +| crossPass2 558613 374210 67.0 374210 0 134499 0 49904 0 crossPass2_SIFT_s0.10e01_i020_c050pct_GEO_s0.15e00_i150_c050pct_d006_h43fb6938fc19a0a7bef73756f1dd03c2 +| crossPass3 184403 8875 4.8 8875 0 103158 0 72370 0 crossPass3_SIFT_s0.15e01_i020_c050pct_GEO_s0.25e01_i150_c050pct_d006_h888d17a7cd1c7380997dd41b256b77c9 +| crossPass4 85288 31243 36.6 31243 0 38266 15779 0 0 crossPass4_SIFT_s0.25e02_i150_c030pct_GEO_none----_----_-------_d003_h9dbf8fa30fbb5f6ccf6330362cdedc20 +| crossPass5 35823 15901 44.4 15901 0 0 0 19922 0 crossPass5_SIFT_s0.25e02_i020_c050pct_GEO_s0.50e02_i075_c000pct_d002_hc9678dff8a758fa34fa42d4f78416501 +| crossPass6 9500 8070 84.9 8070 0 1430 0 0 0 crossPass6_SIFT_s0.25e02_i020_c000pct_GEO_none----_----_-------_d001_h97ce9c637dced2411d673cb2865ba58c +| +| processed_pair_count: 715143 +| saved_pair_count: 594829 +| average_seconds_per_pair: 3.07 +| +| jobs: 11920 +| average_minutes_per_job: 3.07 +| max_job_minutes: 12.92 +``` \ No newline at end of file diff --git a/doc/cellmap/steps/2-3_review_matches.md b/doc/cellmap/steps/2-3_review_matches.md new file mode 100644 index 00000000..13c4e42b --- /dev/null +++ b/doc/cellmap/steps/2-3_review_matches.md @@ -0,0 +1,234 @@ +# Reconstruction Part 3: Review Matches and Fix Connectivity Issues +## Common Parameters +```bash +export BSUB_HOST="login1.int.janelia.org" +export BASE_WORK_DIR="/groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m" +export WORK_DIR="${BASE_WORK_DIR}/VNC/Sec26" # more generically: ${BASE_WORK_DIR}/${REGION}/${TAB} +``` +## Generate Connected Tile Cluster Data +VNC Sec26 Processing Time: 5 minutes using 1 core + +```bash +# Continue on LSF / bsub submit host because most work will be done using LSF cluster. +ssh ${BSUB_HOST} +cd ${WORK_DIR} + +# Use a single node interactive session because cluster counting usually +# takes a few minutes and you don't want to cause trouble on the submit host. +bsub -n 1 -P flyem -Is /bin/bash --login + +# Extract match counts for entire volume and identify all connected tile clusters. +# Ultimately, there should be one and only one connected tile cluster. +./21_count_acquire_clusters.sh + +| Connected tile cluster counts will be written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/cluster_count.v1_acquire.log +| +| Be patient, this could take a few minutes ... +| +| Parsing /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/cluster_count.v1_acquire.log to create: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/excluded_cells.json + +# Exit interactive session and return to the submit host to review results. +exit +``` + +## Trim Unconnected Resin Tiles +VNC Sec26 Processing Time: 15 minutes using 1 core (review), 1 minute using 52 cores (create trimmed stack) + +Most volumes have some tiles with just resin in them, especially at the beginning and end of an +acquisition. If the resin isn't dirty enough, those tiles won't produce point match correspondences +with adjacent tiles creating smaller connected clusters or "islands". The alignment solve process +requires a single connected set of tiles as input, so we need to remove clusters of purely resin tiles +and fix/patch missing connections for tiles that have tissue. + +The `excluded_cells.json` file produced by `./21_count_acquire_clusters.sh` contains cell and z range +information for all connected clusters. There are usually many many tiny clusters in resin areas that +can be combined into one or just a few larger clusters for removal (trimming). Unfortunately, +combining/reducing the small resin clusters is currently a manual process. + +The `excluded_cells.json` file looks like this: +```javascript +[ + { "cellIds": [ + "0,0", + "0,0" + ], + "minZ": 1, + "maxZ": 4244 + }, + { + "cellIds": [ + "0,0", + "0,0" + ], + "minZ": 4245, + "maxZ": 4455 + }, + { + "cellIds": [ + "0,0", + "0,0" + ], + "minZ": 4456, + "maxZ": 4459 + }, ... +``` +To combine/reduce resin clusters, use CATMAID (or any tile viewer) to confirm that the cell only contains +resin tiles and then edit `excluded_cells.json` accordingly. + +![](resin-tiles-in-catmaid.png) + +For this mini-example, the combined json for cell 0,0 would look like: +```javascript +[ + { "cellIds": ["0,0"], "minZ": 1, "maxZ": 4459 }, + ... +] +``` + +After combining/reducing all resin tile cells in VNC Sec26, the final `excluded_cells.json` file +looks like this: +```javascript +[ + { "cellIds": [ "0,0" ], "minZ": 1, "maxZ": 4471 }, + { "cellIds": [ "0,0" ], "minZ": 23965, "maxZ": 24154 }, + { "cellIds": [ "0,0" ], "minZ": 25777, "maxZ": 25867 }, + { "cellIds": [ "0,1" ], "minZ": 1, "maxZ": 2900 }, + { "cellIds": [ "0,1" ], "minZ": 25245, "maxZ": 25365 }, + { "cellIds": [ "0,3" ], "minZ": 21797, "maxZ": 21911 }, + { "cellIds": [ "0,4" ], "minZ": 18750, "maxZ": 21911 } +] +``` + +Once `excluded_cells.json` is ready ... +```bash +# Generate a trimmed (clip) stack by submitting a bsub array job. +./22_gen_clip_stack.sh +``` + +## Review Connectivity for Trimmed Stack +VNC Sec26 Processing Time: 30 minutes using 1 core *(time can vary widely depending upon connection issues)* + +Once the trimmed stack has been created and completed (you should receive an email) ... +```bash +# Re-count connected tile clusters for the trimmed stack. +./23_count_trimmed_clusters.sh + +| Connected tile cluster counts will be written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/cluster_count.v1_acquire_trimmed.log +| +| Be patient, this could take a few minutes ... +| +| 20:56:16.141 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClustersForSlab: slab z 24995.0 to 25776.0 has 1 connected tile sets with sizes [1032] +| 20:56:16.141 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: found 1 connected tile set with size [108155] +| 20:56:16.291 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: 108155 tile set with first tile 21-08-19_194752_0-0-2.1.0 and last tile 21-09-14_134854_0-0-0.25776.0 +| 20:56:16.291 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: found 3 completely unconnected tiles: [21-09-06_113017_0-0-4.18749.0, 21-09-13_091915_0-0-1.23966.0, 21-09-13_091915_0-0-2.23966.0] +| 20:56:16.291 [main] INFO [org.janelia.render.client.ClientRunner] run: exit, processing completed in 0 hours, 1 minutes, 20 seconds + +# In this case, the count log output shows that there is now only 1 cluster (that's good) but +# that there are 3 completely unconnected tiles. These tiles need to be checked and fixed ... +``` + +The first unconnected tile `21-09-06_113017_0-0-4.18749.0` was orphaned by our earlier trim and can be +fixed by simply editing `excluded_cells.json` +```javascript +{ "cellIds": [ "0,4" ], "minZ": 18750, "maxZ": 21911 } -> +{ "cellIds": [ "0,4" ], "minZ": 18749, "maxZ": 21911 } +``` + +The other two unconnected tiles `21-09-13_091915_0-0-1.23966.0` and `21-09-13_091915_0-0-2.23966.0` happened to +be corrupted during acquisition: + +![](problem-tiles.png) + +To fix this kind of problem, we usually patch the corrupted tiles with tiles from an adjacent layer. + +## Patch Corrupted Tiles +VNC Sec26 Processing Time: 45 minutes using essentially 1 core + +The current patch process has several manual steps and is a little messy. + +Run `./setup_patch.sh` to create a patch subdirectory with the scripts needed for patching: + +```bash +# USAGE: ./setup_patch.sh [patch z] ... +# In this case, the z to patch is 23966 (from unconnected tile info). +./setup_patch.sh 23966 + +| Copy these patch tile id lines into +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/v2_patch/06_patch_tile_specs.py: +| +| ("21-09-13_091915_0-0-1.23966.0", -1), # patch from prior layer +| ("21-09-13_091915_0-0-2.23966.0", -1), # patch from prior layer +| +| After patching, generate match pairs by running: +| ./11_gen_new_pairs.sh 23966 + +# move into the patch subdirectory +cd v2_patch +``` + +Edit `06_patch_tile_specs.py` to insert the tile ids (copied from the setup script output) to +be patched. The setup script defaults to patching from the previous layer `(tile_id, -1)` but in +this case the following layer looks better so the data was changed to `(tile_id, 1)`. It is a good +idea to view the area around the patch in CATMAID before deciding the best way to patch it. +```python +def main(): + owner = "Z0720_07m_VNC" # TODO: update with BR or VNC + project = "Sec26" # TODO: update with Sec number + stack = "v2_acquire_trimmed" # TODO: update with patch version number + + # TODO: update tile ids + tile_ids_to_patch = [ + ("21-09-13_091915_0-0-1.23966.0", 1), # patch from next layer + ("21-09-13_091915_0-0-2.23966.0", 1), # patch from next layer + ] +``` + +Copy the `v1_acquire_trimmed` stack to `v2_acquire_trimmed` (with the updated excluded_cells.json): +```bash +# Create and immediately submit copy job: +./01_gen_new_trimmed_stack.sh + +# Once the copy job completes, patch tiles in the `v2_acquire_trimmed` stack: +./06_patch_tile_specs.py + +| submitting PUT http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/state/LOADING +| submitting DELETE http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/tile/21-09-13_091915_0-0-1.23966.0 +| submitting DELETE http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/tile/21-09-13_091915_0-0-2.23966.0 +| submitting PUT http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/resolvedTiles for 2 tile specs +| submitting PUT http://tem-services.int.janelia.org:8080/render-ws/v1/owner/Z0720_07m_VNC/project/Sec26/stack/v2_acquire_trimmed/state/COMPLETE + +# Generate matches for the patched tiles: +./11_gen_new_pairs.sh 23966 + +| ... (similar output to when this script was originally run for alignment prep) + +./12_gen_staged_match_run.sh + +| ... (similar output to when this script was originally run for alignment prep) + +# Run both montage and cross bsub-array.sh scripts (full paths are in gen script output). +run_20220206_093430_479_multi_stage_match_montage/bsub-array.sh +run_20220206_093430_511_multi_stage_match_cross/bsub-array.sh + +# The match jobs should finish quickly since they only cover a few tiles. +# After receiving email confirmation of their completion, +# re-count connected tile clusters for the v2 trimmed stack. +./23_count_trimmed_clusters.sh + +| Connected tile cluster counts will be written to: +| /groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m/VNC/Sec26/logs/cluster_count.v2_acquire_trimmed.log +| +| Be patient, this could take a few minutes ... +| +| 09:57:35.971 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClustersForSlab: slab z 24995.0 to 25776.0 has 1 connected tile sets with sizes [1032] +| 09:57:35.971 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: found 1 connected tile set with size [108102] +| 09:57:36.073 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: 108102 tile set with first tile 21-08-19_194752_0-0-2.1.0 and last tile 21-09-14_134854_0-0-0.25776.0 +| 09:57:36.073 [main] INFO [org.janelia.render.client.ClusterCountClient] findConnectedClusters: found 0 completely unconnected tiles: [] +| 09:57:36.073 [main] INFO [org.janelia.render.client.ClientRunner] run: exit, processing completed in 0 hours, 1 minutes, 9 seconds + +# The count log output now shows that there is only 1 cluster and no unconnected tiles +# We are finally ready to run the alignment solve process. +``` \ No newline at end of file diff --git a/doc/cellmap/steps/2-4_alignment_solve.md b/doc/cellmap/steps/2-4_alignment_solve.md new file mode 100644 index 00000000..83833cc8 --- /dev/null +++ b/doc/cellmap/steps/2-4_alignment_solve.md @@ -0,0 +1,66 @@ +# Reconstruction Part 4: Align Volume +## Common Parameters +```bash +export BSUB_HOST="login1.int.janelia.org" +export BASE_WORK_DIR="/groups/flyem/data/alignment/flyem-alignment-ett/Z0720-07m" +export WORK_DIR="${BASE_WORK_DIR}/VNC/Sec26" # more generically: ${BASE_WORK_DIR}/${REGION}/${TAB} +``` +## Generate Alignment Transformations (Run Solve) +VNC Sec26 Processing Time: 215 minutes using Spark with 18 eleven-core workers, 200 cores in all + +```bash +# Continue on LSF / bsub submit host because most work will be done using LSF cluster. +ssh ${BSUB_HOST} +cd ${WORK_DIR} + +# Run solve on Spark ... +# +# Notes: +# - Default block size for distributed solve is 500 z layers +# so it is easy to allocate too many workers. +# - Using 12/18/24 workers for small/medium/large tabs works well. +./41_alignment_solve.sh 18 +``` + +## Generate Thickness Correction Data +VNC Sec26 Processing Time: 112 minutes using 215 cores + +The `./41_alignment_solve.sh` script submits a job to run `./42_gen_z_corr_run.sh` +when the alignment solve job has completed - so there is no need to manually launch +thickness correction. The thickness correction data generation process involves a +distributed z layer cross correlation calculation followed by a solve. When the +thickness correction solve completes, interactive plots are generated for cross correlation +and z correction values and an email with the Zcoords.txt thickness correction data file path +and URLs for the interactive plots is sent. The email looks like this: + +![](Sec26-v2-z-corr-solve-email.png) + +## Review Thickness Correction Data +VNC Sec26 Processing Time: 30 minutes using 1 core *(time will vary, does not include patch time for problems)* + +Although we ultimately use the thickness correction data to properly render the volume, +we also use the cross correlation information to identify alignment problems. +For the Fly EM volumes, we've found that properly aligned layers typically have a +cross correlation value `>= 0.98`. + +This plot of the `v2_acquire_trimmed` alignment shows that something is wrong around `z 552`: + +![](Sec26-v2-correlation-plot.png) + +Clicking on the problem plot point, opens a CATMAID browser window for the z layer. +You can then navigate through the stack to determine what went wrong during alignment. + +![](Sec26-v2-poorly-aligned-tiles.png) + +In this case, `z 552` has a big wave going through it and the third tile (in cell 0,2) is not +connected to its neighbor. That cell remains disconnected until the `z 1200` area. To correct +the problem, we can either remove/trim the cell tiles or add point matches to better connect the tiles +and prevent them from drifting apart during the solve process. Removal is the best option +when the tiles do not contain any useful information. In all other cases, the tile connections +should be improved. + +## Correct Alignment Issues +If alignment issues are found, corrections are performed +using the [patch process](2-3_review_matches.md#patch-corrupted-tiles) +with an incremented patch version number. You then repeat the solve and review processes +until you have a clean alignment. \ No newline at end of file diff --git a/doc/cellmap/steps/3_export.md b/doc/cellmap/steps/3_export.md new file mode 100644 index 00000000..e69de29b diff --git a/doc/cellmap/summary.md b/doc/cellmap/summary.md new file mode 100644 index 00000000..7d257ecd --- /dev/null +++ b/doc/cellmap/summary.md @@ -0,0 +1,24 @@ +# Render pipeline for Cellmap projects +This document describes the pipeline for processing FIB-SEM data for the Cellmap project. The scripts and tools mentioned in this document can be found at `src/scripts/cellmap`. Most of the information below is specific to the cellmap project, but the general ideas should be transferable. + +Tracking of the progress of the processing and communication with collaborators is detailed in the [project management](project_management.md) section. + +General info on how to run programs on Janelia’s compute cluster can be found on the [Janelia Compute Cluster Wiki page](https://wikis.janelia.org/display/SCS/Janelia+Compute+Cluster). There are also a few things to consider for running Spark on an LSF cluster; see the [corresponding Wiki page](https://wikis.janelia.org/display/ScientificComputing/Spark+on+LSF). + +## Pipeline overview + +The pipeline consists of the following steps: +1. [Transfer](steps/1_transfer.md) the data from the scope to the shared drive. +2. Process the data: + 1. [Render import](steps/2-1_render_import.md): Import the data into the render webservice. + 2. [Prepare alignment](steps/2-2_alignment_prep.md): Prepare the data for alignment. + 3. [Review](steps/2-3_review_matches.md): Review the matches. + 4. [Alignment](steps/2-4_alignment_solve.md): Actually align the data. +3. [Export](steps/3_export.md) the data to a final N5 volume. + +For running each of the processing and export steps, up to three actors are involved: +- A web-server running the render webservice (render-ws). Here, all meta-data is persisted in form of `TileSpec`s (which includes geometric information, paths to the image file, and transformations) and `PointMatch`es. +- The file system where the actual image files are stored. This is usually the lab’s shared drive. +- The machine where the actual computation is done. This is typically the cluster or a local workstation and it needs access to both of the above. + +In addition, for the transfer step, access to the scope and [Janelia's Jenkins server](https://jenkins.int.janelia.org) is needed.