GWAS SumStats Tools

You can access comprehensive documentation for using gwas-sumstat-tools at this link: GWAS SumStats Tools Documentation.

Overview:

There are four commands, read, format validate and gen_meta (gen_meta function is currently only accessible to internal GWAS catalog users.)

read is for:

• Previewing a data file: no options
• Extracting the field headers: -h
• Extracting all the metadata: -M
• Extacting specific field, value pairs from the metada: -m <field name>

format is for:

• Converting sumstats data file to the standard format - gwas-ssf. This is not guaranteed to return a valid standard file, because manadatory data fields could be missing in the input.
  • Generate a configuration file, which serves as a blueprint for the formatting options.
  • Test the configuration file on the first five rows of the input file.
  • Apply the configuration file to the entire input file and generate formatted output file

  [!NOTE] It is memory efficient and will take approx. 30s per 1 million records

gen_meta is for:

• Generate metadata for a data file: -m
  • Read metadata in from existing file: --meta-in <file>
  • Create metadata from the GWAS Catalog (internal use, requires authenticated API): -g
  • Edit/add the values to the metadata: -e with --<FIELD>=<VALUE>

validate is for:

• Validating a summary statistic file using a dynamically generated schema

Requirements

• python >= 3.9 and <3.12

Installation

Local installation with pip

$ pip3 install gwas-sumstats-tools

Run with Docker

The following Docker command is the equivalent to running gwas-ssf.

$ docker run -it -v ${PWD}:/application ebispot/gwas-sumstats-tools:latest

Just append any subcommands or arguments e.g.:

$ docker run -it -v ${PWD}:/application ebispot/gwas-sumstats-tools:latest validate

Usage

$ gwas-ssf [OPTIONS] COMMAND [ARGS]...

Options:

• --help: Show this message and exit.

Commands:

• validate: Validate a sumstats file
• format: Format a sumstats file
• gen_meta: generate meta-yaml file
• read: Read a sumstats file

gwas-ssf validate

Validate a sumstats file

Usage:

$ gwas-ssf validate [OPTIONS] FILENAME

Arguments:

• FILENAME: Input sumstats file. Must be TSV (may be gzipped) [required]

Options:

• -e, --errors-out: Output erros to a csv file, .err.csv.gz
• -z, --p-zero: Force p-values of zero to be allowable. Takes precedence over inferred value (-i)
• -m, --min-rows: Minimum rows acceptable for the file [default: 100000]
• -i, --infer-from-metadata: Infer validation options from the metadata file -meta.yaml. E.g. a populated field for analysis software makes p-values of zero allowable.
• --help: Show this message and exit.

gwas-ssf read

Read (preview) a sumstats file

Usage:

$ gwas-ssf read [OPTIONS] FILENAME

Arguments:

• FILENAME: Input sumstats file [required]

Options:

• -h, --get-header: Just return the headers of the file [default: False]
• --meta-in PATH: Specify a metadata file to read in, defaulting to -meta.yaml
• -M, --get-all-metadata: Return all metadata [default: False]
• -m, --get-metadata TEXT: Get metadata for the specified fields e.g. `-m genomeAssembly -m isHarmonised
• --help: Show this message and exit.

gwas-ssf format

Format a sumstats file and creating a new one. Add/edit metadata.

Usage:

$ gwas-ssf format [OPTIONS] FILENAME

Arguments:

• FILENAME: Input sumstats file. Must be TSV or CSV and may be gzipped [required]

Options:

• Options for reading the input file
  • -d, --delimiter Text: Specify the delimiter in the file, if not specified, we can automatically detect the delimiter as whitespace if your file is *.txt, comma if your file is *.csv, or tab if your file is *.tsv.gz. Otherwise, please specify the delimiter which can help to recognise the column correctly
  • -r, --remove_comments Text: Remove the lines starts with the given character
• Options for generating configuration file
  • -g, --generate_config Boolean: To generate the configuration file for the file needed to be formatted
  • --config_out Path:Specify the configure JSON output file
• Options for applying configuration file
  • -o, --ss-out PATH: Output sumstats file
  • -a, --apply_config Boolean: Apply the given configuration file to the file
  • -t, -test_config Boolean: Test the given configuration file to the first 5 rows of the file
  • --config_in Path: Specify a configure JSON file to read in
  • -f, --analysis_software Text: Specify the analysis software used for generating the summary statistics data
  • -s, --minimal2standard: Try to convert a valid, minimally formatted file to the standard format.This assumes the file at least has p_value combined with rsid in variant_id field or chromosome and base_pair_location. Validity of the new file is not guaranteed because mandatory data could be missing from the original file. [default: False]
• Options for batch applying configuration file
  • -b, --batch_apply Boolean: Apply configuration files to a batch of summary statistics files
  • --lsf Boolean:Running the batch process via submitting jobs via LSF
  • --slurm Boolean:Running the batch process via submitting job via Slurm

gwas-ssf gen_meta

Generate a meta-yaml file for the existing sumstats file OR edit the existing meta-yaml file.

Usage:

$ gwas-ssf gen_meta [OPTIONS] FILENAME

Example:

# Generate a meta-yaml file from GWAS API (-g) with customised fields (-e --file_type=pre-gwas-ssf) for GCST90278188.tsv files
$ gwas-ssf gen_meta --meta-out GCST90278188.tsv-meta.yaml -g GCST90278188.tsv -e --file_type=pre-gwas-ssf

Arguments:

• FILENAME: Input sumstats file. Must be TSV or CSV and may be gzipped [required]

Options:

• --meta-out PATH: Specify the metadata output file
• -g, --meta-gwas: Populate metadata from GWAS Catalog [default: False]
• -e, --meta-edit: Enable metadata edit mode. Then provide params to edit in the --<FIELD>=<VALUE> format e.g. --GWASID=GCST123456 to edit/add that value [default: False]
• --help: Show this message and exit.

Development

This repository uses poetry for dependency and packaging management.

To run the tests:

1. install poetry
2. git clone https://github.com/EBISPOT/gwas-sumstats-tools.git
3. cd gwas-sumstats-tools
4. python3 -m venv env
5. pip install poetry
6. poetry install
7. poetry run pytest -s

To make a change: branch from master -> PR to master -> poetry version -> git add pyproject.toml -> git commit -> git tag -> git push origin master --tags If all the tests pass, this will publish to pypi.

A simple toolkit for reading and formatting GWAS sumstats files from the GWAS Catalog. Built with:

• Petl
• Pydantic
• Typer