Skip to content

Commit

Permalink
Update SVG task dataset ids (#908)
Browse files Browse the repository at this point in the history
* update dataset id

* update spatial resource scripts

* add changelog entry
  • Loading branch information
sainirmayi authored Oct 2, 2024
1 parent 3ba588d commit a01361f
Show file tree
Hide file tree
Showing 45 changed files with 365 additions and 375 deletions.
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
## Minor changes

- Add the CELLxGENE immune cell atlas dataset as a common test resource (PR #907)
- Update `dataset_id` for `tenx_visium`, `zenodo_spatial`, `zenodo_spatial_slidetags` datasets and use `mouse_brain_coronal` as a test resource in the `spatially_variable_genes` task (PR #908)

# openproblems v2.0.0

Expand All @@ -25,7 +26,7 @@ Most relevant parts of the overall structure:
- `openproblems_neurips2022_pbmc`: Fetch a dataset from the OpenProblems NeurIPS2022 competition
- `openproblems_v1`: Fetch a legacy OpenProblems v1 dataset
- `openproblems_v1_multimodal`: Fetch a legacy OpenProblems v1 multimodal dataset
- `tenx_vision`: Fetch a and convert 10x Visium dataset
- `tenx_visium`: Fetch a and convert 10x Visium dataset
- `zenodo_spatial`: Fetch and process an Anndata file containing DBiT seq, MERFISH, seqFISH, Slide-seq v2, STARmap, and Stereo-seq data from Zenodo.
- `zenodo_spatial_slidetags`: Download a compressed file containing gene expression matrix and spatial locations from zenodo.

Expand Down
2 changes: 1 addition & 1 deletion src/datasets/loaders/tenx_visium/script.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
par = {
"input_expression": "https://cf.10xgenomics.com/samples/spatial-exp/2.0.0/CytAssist_FFPE_Mouse_Brain_Rep1/CytAssist_FFPE_Mouse_Brain_Rep1_filtered_feature_bc_matrix.h5",
"input_spatial": "https://cf.10xgenomics.com/samples/spatial-exp/2.0.0/CytAssist_FFPE_Mouse_Brain_Rep1/CytAssist_FFPE_Mouse_Brain_Rep1_spatial.tar.gz",
"dataset_id": "tenx_visium/mouse_brain_coronal_section1_visium",
"dataset_id": "mouse_brain_coronal",
"dataset_name": "Mouse Brain Coronal Section 1 (FFPE)",
"dataset_url": "https://www.10xgenomics.com/datasets/mouse-brain-coronal-section-1-ffpe-2-standard",
"dataset_summary": "Gene expression library of Mouse Brain (CytAssist FFPE) using the Mouse Whole Transcriptome Probe Set",
Expand Down
2 changes: 1 addition & 1 deletion src/datasets/loaders/tenx_visium/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

input_expression ="https://cf.10xgenomics.com/samples/spatial-exp/2.0.0/CytAssist_FFPE_Mouse_Brain_Rep1/CytAssist_FFPE_Mouse_Brain_Rep1_filtered_feature_bc_matrix.h5"
input_spatial = "https://cf.10xgenomics.com/samples/spatial-exp/2.0.0/CytAssist_FFPE_Mouse_Brain_Rep1/CytAssist_FFPE_Mouse_Brain_Rep1_spatial.tar.gz"
dataset_id = "10x_visium/mouse_brain_coronal_section1"
dataset_id = "mouse_brain_coronal"
dataset_name = "Mouse Brain Coronal Section 1 (FFPE)"
dataset_url = "https://www.10xgenomics.com/datasets/mouse-brain-coronal-section-1-ffpe-2-standard"
dataset_summary = "Gene expression library of Mouse Brain (CytAssist FFPE) using the Mouse Whole Transcriptome Probe Set"
Expand Down
281 changes: 134 additions & 147 deletions src/datasets/resource_scripts/tenx_visium.sh

Large diffs are not rendered by default.

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions src/datasets/resource_scripts/zenodo_spatial_slidetags.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

cat > "/tmp/params.yaml" << 'HERE'
param_list:
- id: zenodo_spatial_slidetags/human_cortex_slidetags
- id: zenodo_spatial_slidetags/slidetags/human_cortex
input_data: "https://zenodo.org/records/12785822/files/slidetag_human_cortex.tar.gz?download=1"
dataset_name: Slide-tags - Human Cortex
dataset_url: "https://www.nature.com/articles/s41586-023-06837-4"
Expand All @@ -14,7 +14,7 @@ param_list:
gene_filter_min_spots: 50
remove_mitochondrial: true
- id: zenodo_spatial_slidetags/human_skin_melanoma_slidetags
- id: zenodo_spatial_slidetags/slidetags/human_skin_melanoma
input_data: "https://zenodo.org/records/12785822/files/slidetag_human_skin_melanoma.tar.gz?download=1"
dataset_name: Slide-tags - Human Skin Melanoma
dataset_url: "https://www.nature.com/articles/s41586-023-06837-4"
Expand All @@ -26,7 +26,7 @@ param_list:
gene_filter_min_spots: 50
remove_mitochondrial: true
- id: zenodo_spatial_slidetags/human_tonsil_slidetags
- id: zenodo_spatial_slidetags/slidetags/human_tonsil
input_data: "https://zenodo.org/records/12785822/files/slidetag_human_tonsil.tar.gz?download=1"
dataset_name: Slide-tags - Human Tonsil
dataset_url: "https://www.nature.com/articles/s41586-023-06837-4"
Expand All @@ -38,7 +38,7 @@ param_list:
gene_filter_min_spots: 50
remove_mitochondrial: true
- id: zenodo_spatial_slidetags/mouse_embryo_slidetags
- id: zenodo_spatial_slidetags/slidetags/mouse_embryo
input_data: "https://zenodo.org/records/12785822/files/slidetag_mouse_embryo.tar.gz?download=1"
dataset_name: Slide-tags - Mouse Embryo
dataset_url: "https://www.nature.com/articles/s41586-023-06837-4"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ set -e

cat > /tmp/params.yaml << 'HERE'
param_list:
- id: mouse_brain_coronal_section1
- id: mouse_brain_coronal
input_expression: "https://cf.10xgenomics.com/samples/spatial-exp/2.0.0/CytAssist_FFPE_Mouse_Brain_Rep1/CytAssist_FFPE_Mouse_Brain_Rep1_filtered_feature_bc_matrix.h5"
input_spatial: "https://cf.10xgenomics.com/samples/spatial-exp/2.0.0/CytAssist_FFPE_Mouse_Brain_Rep1/CytAssist_FFPE_Mouse_Brain_Rep1_spatial.tar.gz"
dataset_name: Mouse Brain Coronal Section 1 (FFPE)
dataset_name: 10X Visium - Mouse Brain Coronal
dataset_url: "https://www.10xgenomics.com/datasets/mouse-brain-coronal-section-1-ffpe-2-standard"
dataset_summary: Gene expression library of Mouse Brain (CytAssist FFPE) using the Mouse Whole Transcriptome Probe Set
dataset_description: "FFPE Mouse Brain tissue blocks sectioned as described in Visium CytAssist Spatial Gene Expression for FFPE - Tissue Preparation Guide Demonstrated Protocol. The H&E stained glass slide with tissue section was processed via Visium CytAssist instrument to transfer analytes to a Visium CytAssist Spatial Gene Expression slide. The probe extension and library construction steps follow the standard Visium for FFPE workflow outside of the instrument. The H&E image was acquired using Olympus VS200 Slide Scanning Microscope. Sequencing depth was 53,497 reads per spot. Sequencing configuration: 28bp read 1 (16bp Visium spatial barcode, 12bp UMI), 90bp read 2 (transcript), 10bp i7 sample barcode and 10bp i5 sample barcode. Key metrics include: 2,310 spots detected under tissue; 6,736 median genes per spot; 24,862 median UMI counts per spot."
Expand Down
10 changes: 5 additions & 5 deletions src/tasks/spatially_variable_genes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ flowchart LR
A subset of the common dataset.

Example file:
`resources_test/common/mouse_brain_coronal_section1/dataset.h5ad`
`resources_test/common//dataset.h5ad`

Format:

Expand Down Expand Up @@ -142,7 +142,7 @@ Arguments:
The dataset without spatially variable genes.

Example file:
`resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad`
`resources_test/spatially_variable_genes//dataset.h5ad`

Format:

Expand Down Expand Up @@ -177,7 +177,7 @@ Slot description:
Anndata with true spatial variability.

Example file:
`resources_test/spatially_variable_genes/mouse_brain_coronal_section1/solution.h5ad`
`resources_test/spatially_variable_genes//solution.h5ad`

Description:

Expand Down Expand Up @@ -274,7 +274,7 @@ Arguments:
Anndata with estimate spatial variability.

Example file:
`resources_test/spatially_variable_genes/mouse_brain_coronal_section1/output.h5ad`
`resources_test/spatially_variable_genes//output.h5ad`

Description:

Expand Down Expand Up @@ -309,7 +309,7 @@ Slot description:
Metric score file.

Example file:
`resources_test/spatially_variable_genes/mouse_brain_coronal_section1/score.h5ad`
`resources_test/spatially_variable_genes//score.h5ad`

Format:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,6 @@ functionality:
path: /src/common/comp_tests/check_method_config.py
- type: python_script
path: /src/common/comp_tests/run_and_check_adata.py
- path: /resources_test/spatially_variable_genes/mouse_brain_coronal_section1
dest: resources_test/spatially_variable_genes/mouse_brain_coronal_section1
- path: /resources_test/spatially_variable_genes/mouse_brain_coronal
dest: resources_test/spatially_variable_genes/mouse_brain_coronal
- path: /src/common/library.bib
4 changes: 2 additions & 2 deletions src/tasks/spatially_variable_genes/api/comp_method.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,6 @@ functionality:
path: /src/common/comp_tests/check_method_config.py
- type: python_script
path: /src/common/comp_tests/run_and_check_adata.py
- path: /resources_test/spatially_variable_genes/mouse_brain_coronal_section1
dest: resources_test/spatially_variable_genes/mouse_brain_coronal_section1
- path: /resources_test/spatially_variable_genes/mouse_brain_coronal
dest: resources_test/spatially_variable_genes/mouse_brain_coronal
- path: /src/common/library.bib
4 changes: 2 additions & 2 deletions src/tasks/spatially_variable_genes/api/comp_metric.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ functionality:
path: /src/common/comp_tests/check_metric_config.py
- type: python_script
path: /src/common/comp_tests/run_and_check_adata.py
- path: /resources_test/spatially_variable_genes/mouse_brain_coronal_section1
dest: resources_test/spatially_variable_genes/mouse_brain_coronal_section1
- path: /resources_test/spatially_variable_genes/mouse_brain_coronal
dest: resources_test/spatially_variable_genes/mouse_brain_coronal
- path: /src/common/library.bib

Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,5 @@ functionality:
test_resources:
- type: python_script
path: /src/common/comp_tests/run_and_check_adata.py
- path: /resources_test/common/mouse_brain_coronal_section1
dest: resources_test/common/mouse_brain_coronal_section1
- path: /resources_test/common/mouse_brain_coronal
dest: resources_test/common/mouse_brain_coronal
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
type: file
example: "resources_test/common/mouse_brain_coronal_section1/dataset.h5ad"
example: "resources_test/common/mouse_brain_coronal/dataset.h5ad"
info:
label: "Common Dataset"
summary: A subset of the common dataset.
Expand Down
2 changes: 1 addition & 1 deletion src/tasks/spatially_variable_genes/api/file_dataset.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
type: file
example: "resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad"
example: "resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad"
info:
label: "Dataset"
summary: The dataset without spatially variable genes.
Expand Down
2 changes: 1 addition & 1 deletion src/tasks/spatially_variable_genes/api/file_output.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
type: file
example: "resources_test/spatially_variable_genes/mouse_brain_coronal_section1/output.h5ad"
example: "resources_test/spatially_variable_genes/mouse_brain_coronal/output.h5ad"
info:
label: Output
summary: "Anndata with estimate spatial variability."
Expand Down
2 changes: 1 addition & 1 deletion src/tasks/spatially_variable_genes/api/file_score.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
type: file
example: "resources_test/spatially_variable_genes/mouse_brain_coronal_section1/score.h5ad"
example: "resources_test/spatially_variable_genes/mouse_brain_coronal/score.h5ad"
info:
label: "Score"
summary: Metric score file.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
type: file
example: "resources_test/spatially_variable_genes/mouse_brain_coronal_section1/simulated_dataset.h5ad"
example: "resources_test/spatially_variable_genes/mouse_brain_coronal/simulated_dataset.h5ad"
info:
label: "Common Dataset"
summary: A subset of the common dataset.
Expand Down
2 changes: 1 addition & 1 deletion src/tasks/spatially_variable_genes/api/file_solution.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
type: file
example: "resources_test/spatially_variable_genes/mouse_brain_coronal_section1/solution.h5ad"
example: "resources_test/spatially_variable_genes/mouse_brain_coronal/solution.h5ad"
info:
label: Solution
summary: "Anndata with true spatial variability."
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_solution': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/solution.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'input_solution': 'resources_test/spatially_variable_genes/mouse_brain_coronal/solution.h5ad',
'output': 'output.h5ad'
}
meta = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_solution': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/solution.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'input_solution': 'resources_test/spatially_variable_genes/mouse_brain_coronal/solution.h5ad',
'output': 'output.h5ad'
}
meta = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ dest <- getwd()

# VIASH START
par <- list(
"input_data" = "resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad",
"input_data" = "resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad",
"output" = "output.h5ad",
"n_iter" = 10
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad',
'n_features': 120
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad',
'coord_type_moran_i': 'generic'

Expand Down
2 changes: 1 addition & 1 deletion src/tasks/spatially_variable_genes/methods/nnsvg/script.R
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ suppressMessages(library(dplyr))

# VIASH START
par = list(
'input_data' = 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data' = 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output' = 'output.h5ad'
)
meta = list(
Expand Down
2 changes: 1 addition & 1 deletion src/tasks/spatially_variable_genes/methods/scgco/script.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad'
}
meta = {
Expand Down
2 changes: 1 addition & 1 deletion src/tasks/spatially_variable_genes/methods/sepal/script.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad',
'coord_type_sepal': 'grid',
'max_neighs_sepal': 6,
Expand Down
2 changes: 1 addition & 1 deletion src/tasks/spatially_variable_genes/methods/somde/script.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad'
}
meta = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad'
}
meta = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad'
}
meta = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad'
}
meta = {
Expand Down
2 changes: 1 addition & 1 deletion src/tasks/spatially_variable_genes/methods/spark/script.R
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ suppressMessages(library(anndata))

# VIASH START
par <- list(
"input_data" = "resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad",
"input_data" = "resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad",
"output" = "output.h5ad"
)
meta <- list(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ suppressMessages(library(anndata))

# VIASH START
par <- list(
"input_data" = "resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad",
"input_data" = "resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad",
"output" = "output.h5ad"
)
meta <- list(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad'
}
meta = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

# VIASH START
par = {
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/dataset.h5ad',
'input_data': 'resources_test/spatially_variable_genes/mouse_brain_coronal/dataset.h5ad',
'output': 'output.h5ad'
}
meta = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@

## VIASH START
par = {
'input_method': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/output.h5ad',
'input_solution': 'resources_test/spatially_variable_genes/mouse_brain_coronal_section1/solution.h5ad',
'input_method': 'resources_test/spatially_variable_genes/mouse_brain_coronal/output.h5ad',
'input_solution': 'resources_test/spatially_variable_genes/mouse_brain_coronal/solution.h5ad',
'output': 'score.h5ad'
}
meta = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,8 @@ functionality:
- type: python_script
path: script.py
test_resources:
- path: /resources_test/common/mouse_brain_coronal_section1
dest: resources_test/common/mouse_brain_coronal_section1
- path: /resources_test/common/mouse_brain_coronal
dest: resources_test/common/mouse_brain_coronal
- type: python_script
path: /src/common/comp_tests/run_and_check_adata.py
platforms:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

### VIASH START
par = {
"input": "resources_test/common/mouse_brain_coronal_section1/dataset.h5ad",
"input": "resources_test/common/mouse_brain_coronal/dataset.h5ad",
"input_layer": "normalized",
"output": "reference_dataset.h5ad",
"num_features": 50,
Expand Down
Loading

0 comments on commit a01361f

Please sign in to comment.