Skip to content

Latest commit

 

History

History
399 lines (329 loc) · 24.9 KB

README.md

File metadata and controls

399 lines (329 loc) · 24.9 KB
Raw

optimized_antsMultivariateTemplateConstruction

This repository contains a re-implementation of the ANTs template construction pipeline specifically antsMultivariateTemplateConstruction2.sh. The pipeline attempts to maintain the principles of template construction while differing substantially in some of the implementation steps.

This implementation strives to be agnostic to the input data type and has been successfully used with a variety of imaging modalities and species at widely varying physical scales.

The base template construction pipeline modelbuild.sh is also wrapped using twolevel_modelbuild.sh to implement a multi-level template construction suitable for longitudinal data.

In addition to template construction, also provided here are post-processing scripts to provide outputs for deformation-based-morphometry (DBM) for cross-sectional and longitudinal data.

Finally, support scripts are provided to take advantage of the co-registration performed by template construction for moving labels and quantitative maps to/from subject and common space.

How does this all work?

The ANTs unbiased model/template building method consists of the iterative application of two major stages:

Stage 1: all-to-target registration

In the registration stage, all inputs are registered to a target, which is either the initial target (first round) or the evolving template from the previous round. After registration, each input is resampled into the target space.

Stage 2: template-updating

In the template updating stage, several sub-steps happen:

  1. The resampled inputs are voxel-wise averaged.
  2. The resulting average has a sharpening filter applied.
  3. The affine transforms from inputs to targets are averaged.
    • (If enabled) the average affine transform is scaled.
  4. The (non-linear) transformations from inputs to target are averaged, pseudo-inverted, and scaled (multiplied by the negative of the gradient step value)
  5. The resulting inverted average affine transform and pseudo-inverted-scaled average transform is applied to the sharpened average.

This new average is used as the target for the next round of registration and the process is repeated.

Differences from antsMultivariateTemplateConstruction2.sh

Changes made to pipeline flow which should not affect final outcome

  • File-presence based state tracking to allow for pipeline resume
  • Logging of all commands run
  • Integration of qbatch for local and cluster parallelism.
  • Complete computation of work graph before job submission.
  • Nested directory hierarchy for organization of files.
  • No overwriting of intermediate files allowing for traceability of steps.

Changes which are expected to affect final outcome

  • Registration with antsRegistration_affine_SyN.sh and driven by ants_generate_iterations.py
    • Optimized registration scale pyramids based on image and voxel size
    • Constrained linear transform path via rigid->similarity->affine path
    • Stage repeats without/with masking
  • (if enabled) Integration of masking during registration
  • Interpolation of transforms using BSpline[5] where applicable
  • Staged template construction using progressively higher-order transform types (rigid, similarity, affine, nlin)
  • (optional) Affine transforms at later stages bootstrapped from prior stages to reduce computational load
  • (if enabled) Affine transforms averaged using Lie Algebra instead of averaging matrix components
  • (if enabled) Average affine transform scaled using gradient step
  • Some defaults have been changed compared to antsMultivariateTemplateConstruction2.sh
    • Affine transforms are averaged without rigid component
    • Average sharpening defaults to UnsharpMask
    • All computation defaults to double

Requirements

This pipeline is primarily written in bash, and requires ANTs for the commands, and qbatch for running commands locally or with cluster integration.

A dependency submodule of minc-toolkit-extras is automatically included with appropriate versioning.

(Optional) advanced averaging options are provided by a python scripts which require SimpleITK, NumPy, SciPy and VTK.

Missing features

The following are features missing compared to antsMultivariateTemplateConstruction2.sh

Planned

  • multispectral registration
    • modality weights
  • change non-linear transform type
  • metric control for all stages

Will not be implemented

  • preprocessing
  • 2D
  • 4D

Installation

Provided you have installed ANTs (via conda, binaries, or source), and qbatch (via pip)

$ git clone --recursive https://github.com/CoBrALab/optimized_antsMultivariateTemplateConstruction.git

You can add this directory to your PATH or refer to the scripts from any working directory, it will properly find the rest of its dependent scripts.

Basic usage

Example help, always check ./modelbuild.sh --help in case this document has not been updated

A qbatch enabled, optimal registration pyramid based re-implementaiton of antsMultivariateTemplateConstruction2.sh
Usage: ./modelbuild.sh [-h|--help] [--output-dir <arg>] [--gradient-step <arg>] [--starting-target <arg>] [--starting-target-mask <arg>] [--starting-average-resolution <arg>] [--starting-average-type <arg>] [--starting-average-prog <arg>] [--(no-)starting-average-norm] [--iterations <arg>] [--convergence <arg>] [--syn-shrink-factors <arg>] [--syn-smoothing-sigmas <arg>] [--syn-convergence <arg>] [--syn-control <arg>] [--linear-shrink-factors <arg>] [--linear-smoothing-sigmas <arg>] [--linear-convergence <arg>] [--(no-)float] [--(no-)fast] [--average-type <AVERAGE>] [--average-prog <PROG>] [--(no-)average-norm] [--(no-)nlin-shape-update] [--(no-)affine-shape-update] [--(no-)scale-affines] [--(no-)rigid-update] [--sharpen-type <SHARPEN>] [--masks <arg>] [--(no-)mask-extract] [--mask-merge-threshold <arg>] [--stages <arg>] [--(no-)reuse-affines] [--final-target <arg>] [--final-target-mask <arg>] [--walltime-short <arg>] [--walltime-linear <arg>] [--walltime-nonlinear <arg>] [--jobname-prefix <arg>] [--job-predepend <arg>] [--(no-)skip-file-checks] [--(no-)block] [--(no-)debug] [--(no-)dry-run] <inputs-1> [<inputs-2>] ... [<inputs-n>] ...
        <inputs>: Input text file, one line per input
        -h, --help: Prints help
        --output-dir: Output directory for modelbuild (default: 'output')
        --gradient-step: Gradient scaling step during template warping, can be a comma separated list same length as number of iterations (default: '0.25')
        --starting-target: Starting target, dumb average (dumb), align all inputs using their center-of-mass before averaging (com) use the first input (first), or an external file (provide path) (default: 'first')
        --starting-target-mask: Mask for starting target if a file (no default)
        --starting-average-resolution: If no starting target is provided, an average is constructed from all inputs, resample average to a target resolution MxNxO before modelbuild (no default)
        --starting-average-type: Type of averaging to apply for starting average (default: 'median')
        --starting-average-prog: Software to use for averaging starting average
                        python with SimpleITK needed for trimmed_mean, efficient_trimean, and huber (default: 'ANTs')
        --starting-average-norm, --no-starting-average-norm: Normalize images by their mean before averaging during starting average (on by default)
        --iterations: Number of iterations of model building per stage (default: '4')
        --convergence: Convergence limit during registration calls (default: '1e-9')
        --syn-shrink-factors: Shrink factors for Non-linear (SyN) stages, provide to override automatic generation, must be provided with sigmas and convergence (no default)
        --syn-smoothing-sigmas: Smoothing sigmas for Non-linear (SyN) stages, provide to override automatic generation, must be provided with shrinks and convergence (no default)
        --syn-convergence: Convergence levels for Non-linear (SyN) stages, provide to override automatic generation, must be provided with shrinks and sigmas (no default)
        --syn-control: Non-linear (SyN) gradient and regularization parameters, not checked for correctness (default: '0.1,3,0')
        --linear-shrink-factors: Shrink factors for linear stages, provide to override automatic generation, must be provided with sigmas and convergence (no default)
        --linear-smoothing-sigmas: Smoothing sigmas for linear stages, provide to override automatic generation, must be provided with shrinks and convergence (no default)
        --linear-convergence: Convergence levels for linear stages, provide to override automatic generation, must be provided with shrinks and sigmas (no default)
        --float, --no-float: Use float instead of double for calculations (reduce memory requirements) (off by default)
        --fast, --no-fast: Run SyN registration with Mattes instead of CC (off by default)
        --average-type: Type of averaging to apply during modelbuild. Can be one of: 'mean', 'median', 'trimmed_mean', 'efficient_trimean' and 'huber' (default: 'mean')
        --average-prog: Software to use for averaging images and transforms
                        python with SimpleITK needed for trimmed_mean, efficient_trimean, and huber. Can be one of: 'ANTs' and 'python' (default: 'ANTs')
        --average-norm, --no-average-norm: Normalize images by their mean before averaging (on by default)
        --nlin-shape-update, --no-nlin-shape-update: Perform nlin shape update, disable to switch to a forward-only modelbuild (on by default)
        --affine-shape-update, --no-affine-shape-update: Scale template by inverse of average affine transforms during shape update step (on by default)
        --scale-affines, --no-scale-affines: Apply gradient step scaling factor to average affine during shape update step, requires python with VTK and SimpleITK (off by default)
        --rigid-update, --no-rigid-update: Include rigid component of transform when performing shape update on template (disable if template drifts in translation or orientation) (off by default)
        --sharpen-type: Type of sharpening applied to average during modelbuild. Can be one of: 'none', 'laplacian' and 'unsharp' (default: 'unsharp')
        --masks: File containing mask filenames, one file per line (no default)
        --mask-extract, --no-mask-extract: Use masks to extract images before registration (off by default)
        --mask-merge-threshold: Threshold to combine masks during averaging (default: '0.5')
        --stages: Stages of modelbuild used (comma separated options: 'rigid' 'similarity' 'affine' 'nlin' 'nlin-only','volgenmodel-nlin'), append a number in brackets 'rigid[n]' to override global iteration setting (default: 'rigid,similarity,affine,nlin')
        --reuse-affines, --no-reuse-affines: Reuse affines from previous stage/iteration to initialize next stage (off by default)
        --final-target: Perform a final registration between the average and final target, used in postprocessing (default: 'none')
        --final-target-mask: Mask for the final target used in postprocessing (default: 'none')
        --walltime-short: Walltime for short running stages (averaging, resampling) (default: '00:30:00')
        --walltime-linear: Walltime for linear registration stages (default: '0:45:00')
        --walltime-nonlinear: Walltime for nonlinear registration stages (default: '4:30:00')
        --jobname-prefix: Prefix to add to front of job names, used by twolevel wrapper (no default)
        --job-predepend: Job name dependency pattern to prepend to all jobs, used by twolevel wrapper (no default)
        --skip-file-checks, --no-skip-file-checks: Skip preflight checking of existence of files, used by twolevel wrapper (off by default)
        --block, --no-block: For SGE, PBS and SLURM, blocks execution until jobs are finished. (off by default)
        --debug, --no-debug: Debug mode, print all commands to stdout (off by default)
        --dry-run, --no-dry-run: Dry run, don't run any commands, implies debug (off by default)

Minimal run command, assuming an input text file inputs.txt containing one line per path to an input file

$ ./modelbuild.sh input.txt

Outputs

The final output round of a modelbuild.sh run will be in ${output_dir}/final which is linked to the final stage output directory which was specified on the command line. See Output directory structure.

The final modelbuild average is ${output_dir}/final/average/template_sharpen_shapeupdate.nii.gz

Two-level wrapper

A wrapper to enable two-level modelbuild (aka longitudinal) modelling using optimized_antsMultivariateTemplateConstruction
Usage: ./twolevel_modelbuild.sh [-h|--help] [--output-dir <arg>] [--masks <arg>] [--firstlevel-starting-target <arg>] [--firstlevel-starting-target-mask <arg>] [--secondlevel-starting-target <arg>] [--secondlevel-starting-target-mask <arg>] [--secondlevel-final-target <arg>] [--secondlevel-final-target-mask <arg>] [--(no-)debug] [--(no-)dry-run] <inputs> ...
        <inputs>: Input text files, one line per subject, comma separated scans per subject
        ... : Arguments to be passed to modelbuild.sh without validation
        -h, --help: Prints help
        --output-dir: Output directory for modelbuild (default: 'output')
        --masks: File containing mask filenames, identical to inputs in structure (no default)
        --firstlevel-starting-target: First-level starting target, dumb average (dumb), align all inputs using their center-of-mass before averaging (com) use the first input (first), or an external file (provide path) (default: 'first')
        --firstlevel-starting-target-mask: First-level Starting target mask, must be combined with file based starting target (no default)
        --secondlevel-starting-target: Second-level starting target, dumb average (dumb), align all inputs using their center-of-mass before averaging (com) use the first input (first), or an external file (provide path) (default: 'first')
        --secondlevel-starting-target-mask: Second-level Starting target mask, must be combined with file based starting target (no default)
        --secondlevel-final-target: Second-level final target (no default)
        --secondlevel-final-target-mask: Second-level Starting target mask (no default)
        --debug, --no-debug: Debug mode, print all commands to stdout (off by default)
        --dry-run, --no-dry-run: Dry run, don't run any commands, implies debug (off by default)

--output-dir will contain a firstlevel/ containing scan-wise modelbuild.sh outputs, and a secondlevel/ directory, containing subject-wise modelbuild outputs. See above for details.

Deformation Based Morphometry (DBM) -- Model build must be completed first

Once a unbiased average model has been constructed, it's possible to post-process the consensus deformation fields to produce Jacobian determinants which encode the voxel-wise distance from each input scan to the consensus average.

Post processing will generate absolute (including affine components) and relative (excluding affine components, and residual affines) log Jacobian determinants (voxel > 0, voxel expands towards subject (i.e. subject voxel is larger), voxel < 0, voxel contracts towards subject (i.e. subject voxel is smaller)).

A minimal run command, assuming a complete run from modelbuild.sh, run using input.txt

$ ./dbm.sh input.txt

Complete run options

DBM post-processing for modelbuild.sh from optimized_antsMultivariateTemplateConstruction
Usage: ./dbm.sh [-h|--help] [--output-dir <arg>] [--(no-)float] [--mask <arg>] [--target-space <SPACE>] [--delin-affine-ratio <arg>] [--(no-)use-geometric] [--jacobian-smooth <arg>] [--walltime <arg>] [--(no-)block] [--(no-)debug] [--(no-)dry-run] [--jobname-prefix <arg>] <inputs-1> [<inputs-2>] ... [<inputs-n>] ...
        <inputs>: Input text files, one line per input, one file per spectra
        -h, --help: Prints help
        --output-dir: Output directory for modelbuild (default: 'output')
        --float, --no-float: Use float instead of double for calculations (reduce memory requirements, reduce precision) (off by default)
        --mask: Mask file for average to improve delin estimates (no default)
        --target-space: Target space for DBM outputs, requires --final-target be used with modelbuild.sh. Can be one of: 'unbiased' and 'final-target' (default: 'unbiased')
        --delin-affine-ratio: Ratio of voxels within mask used to estimate delin affine (default: '0.25')
        --use-geometric, --no-use-geometric: Use geometric estimate of Jacobian instead of finite-difference (on by default)
        --jacobian-smooth: Comma separated list of smoothing gaussian FWHM, append "vox" for voxels, "mm" for millimeters (default: '4vox')
        --walltime: Walltime for short running stages (averaging, resampling) (default: '00:15:00')
        --block, --no-block: For qbatch SGE, PBS and SLURM, blocks execution until jobs are finished. (off by default)
        --debug, --no-debug: Debug mode, print all commands to stdout (off by default)
        --dry-run, --no-dry-run: Dry run, don't run any commands, implies debug (off by default)
        --jobname-prefix: Prefix to add to front of job names, used by twolevel wrapper (no default)

Outputs

Single level DBM outputs are found in ${output_dir}/dbm/jacobian/{full,relative}/smooth named according to the input scan with a suffix of the smoothing option (_fwhm_4vox for example)

Classical DBM

In classical DBM, rather than building an unbiased average, a direct registration is done to a target template and the Jacobian determinants computed. This is achievable using these tools by limiting the model construction to a single iteration and ignoring the average model in favour of the initial target.

$ ./modelbuild.sh --starting-target <MNI_model> --stages nlin[1] input.txt
$ ./dbm.sh input.txt

Two-level DBM wrapper

The two-level DBM wrapper takes the outputs from a twolevel_modelbuild.sh and produces scan-wise and subject-wise jacobian determinants for statistical analysis.

DBM post-processing for twolevel_modelbuild.sh from optimized_antsMultivariateTemplateConstruction
Usage: ./twolevel_dbm.sh [-h|--help] [--output-dir <arg>] [--jacobian-smooth <arg>] [--walltime <arg>] [--(no-)debug] [--(no-)dry-run] <inputs> ...
        <inputs>: Input text files, one line per subject, comma separated scans per subject
        ... : Arguments to be passed to dbm.sh without validation
        -h, --help: Prints help
        --output-dir: Output directory for modelbuild (default: 'output')
        --jacobian-smooth: Comma separated list of smoothing gaussian FWHM, append "vox" for voxels, "mm" for millimeters (default: '4vox')
        --walltime: Walltime for short running stages (averaging, resampling) (default: '00:15:00')
        --debug, --no-debug: Debug mode, print all commands to stdout (off by default)
        --dry-run, --no-dry-run: Dry run, don't run any commands, implies debug (off by default)

A minimal run command, assuming a complete run from twolevel_modelbuild.sh, run using input.txt

$ ./twolevel_dbm.sh input.txt

Outputs

Two-level DBM processing produces two types of outputs, overall-dbm files, which encode the entire voxel-wise difference between the original input scan and the final second-level average, and resampled-dbm which encode the within-subject change, with voxel-wise correspondence at the population level. The resampled-dbm outputs are typically what is used for longitudinal analysis as they contain within-subject changes aligned at the population level.

dbm directories are created within each ${output_dir}/firstlevel/subject_${N} directory, as well as in the ${output_dir}/secondlevel directory for the within-level computation of needed intermediates. DBM outputs intended for analysis at the second level are produced at ${output_dir}/secondlevel/{overall-dbm,resampled-dbm}/jacobian/{full,relative}/smooth with naming according to the original input scans.

Helper Tools

{twolevel}_{commonspace,subjectspace}_resample.sh are support scripts which can be used to move quantitative maps and labels between the input subject space and the common template space.

These helper scripts can accept affine transforms which define the pre-alignment between the inputs to be resampled and the files used during template construction, othterwise, inputs are presumed to be already aligned in the space.

Common uses:

  1. Moving PET images to common space for voxel-wise statistics
  2. Moving MRI quantitative maps (T1maps, T2star, QSM) to to common space for voxel-wise statistics
  3. Moving individual labels of anatomy into common space to vote a consensus label set
  4. Moving labels generated in the common space to individual subject space

Output directory structure

output/
├── initialaverage
│   └── initialtarget.nii.gz # Generated if no starting target is supplied
├── jobs
│   └── <run date/time in ISO format>
│       ├── initialaverage
│       ├── rigid_{0,1,2,3}_maskaverage
│       ├── rigid_{0,1,2,3}_maskresample
│       ├── rigid_{0,1,2,3}_reg
│       ├── rigid_{0,1,2,3}_shapeupdate
│       ├── similarity_{0,1,2,3}_maskaverage
│       ├── similarity_{0,1,2,3}_maskresample
│       ├── similarity_{0,1,2,3}_reg
│       ├── similarity_{0,1,2,3}_shapeupdate
│       ├── affine_{0,1,2,3}_maskaverage
│       ├── affine_{0,1,2,3}_maskresample
│       ├── affine_{0,1,2,3}_reg
│       ├── affine_{0,1,2,3}_shapeupdate
│       ├── nlin_{0,1,2,3}_maskaverage
│       ├── nlin_{0,1,2,3}_maskresample
│       ├── nlin_{0,1,2,3}_reg
│       └── nlin_{0,1,2,3}_shapeupdate
├── dbm
│   ├── intermediate
│   │   ├── affine # jacobians from affine transforms
│   │   ├── delin # jacobians from residual affines
│   │   └── nlin # jacobians from nonlinear deformation fields
│   └── jacobian
│       ├── full
│       │   └── smooth # Final per-scan smoothed absolute jacobains
│       └── relative
│           └── smooth # Final per-scan smoothed relative jacobains
├── rigid
│   ├── {0,1,2,3}
│   │   ├── average
│   │   │   ├── template.nii.gz # Average of resampled files
│   │   │   ├── affine.mat # Average of affines
│   │   │   ├── template_sharpen.nii.gz # Template after sharpening
│   │   │   ├── nonzero.nii.gz # Nonzero mask to clip negative values from BSpline[5]
│   │   │   └── template_sharpen_shapeupdate.nii.gz # Template after shape update
│   │   ├── resample # One file per input, resampled into template space
│   │   │   └── masks # One mask file per input, resampled into template space
│   │   └── transforms # Affine transform files per input
├── similarity
│   ├── {0,1,2,3}
│   │   ├── average
│   │   │   ├── template.nii.gz
│   │   │   ├── affine.mat
│   │   │   ├── template_sharpen.nii.gz
│   │   │   ├── nonzero.nii.gz
│   │   │   └── template_sharpen_shapeupdate.nii.gz
│   │   ├── resample
│   │   │   └── masks
│   │   └── transforms
├── affine
│   ├── {0,1,2,3}
│   │   ├── average
│   │   │   ├── template.nii.gz
│   │   │   ├── affine.mat
│   │   │   ├── template_sharpen.nii.gz
│   │   │   ├── nonzero.nii.gz
│   │   │   └── template_sharpen_shapeupdate.nii.gz
│   │   ├── resample
│   │   │   └── masks
│   │   └── transforms
└── nlin
    └── {0,1,2,3}
        ├── average
        │   ├── template.nii.gz
        │   ├── affine.mat
        │   ├── template_sharpen.nii.gz
        │   ├── warp.nii.gz # Average of warp fields
        │   ├── scaled_warp.nii.gz # Scaled/pseudoinverted average warp
        │   ├── affine_scaled_warp.nii.gz # Warp transformed with inverse affine
        │   ├── nonzero.nii.gz
        │   └── template_sharpen_shapeupdate.nii.gz
        ├── resample
        │   └── masks
        └── transforms # Affine and warp transform files per input