Recent Updates to the Harmonisation Pipeline
1. Harmonisation Step Changes
- Z-Score Harmonisation: The pipeline now harmonises z-scores in the input file, ensuring consistency for effect size calculations.
- Automatic Conversion of neg_Log_10_P_Value: neg_log_10_p_value fields are now automatically converted to p_values for easier cross-study comparison.
2. Workflow Modifications
- Customisable Input File Names: The pipeline now accepts customised input file names.
- Consistent Naming for GCST Files: If the input file name follows the pattern GCST[0-9]+, the output files will continue to use the GCST naming convention.
3. Output Folder Structure Adjustments
- Reorganisation of Intermediate Files: The intermediate files folder is now numbered for easier navigation.
- Final Output Files: The final output folder includes only the essential files: h.tsv.gz, h.tsv.gz.tbi, h.tsv.gz-meta.yaml, and the running.log.
4. Configuration File Isolation
- Executor Configuration Isolation: The executor configuration file is now isolated, making it easier for users to adapt the pipeline for various executors.
5. Removal of Obsolete Steps
- Rsid Validation Removal: As current rsid values come directly from VCF files, the separate rsid validation step has been removed.
6. Automatic Version Logging
- Dynamic Versioning in Running Log: The hardcoded version entry in the running log has been replaced with an automated version retrieval from the pipeline parameters.
7. Error Handling Strategy
- Pipeline Termination on Error: Users may now terminate the pipeline if an error occurs. However, this is not recommended due to the impact on parallelised jobs.
8.Failed File Management
- Failed Logs in FTP: Previously, files where all variants failed harmonisation (containing only headers) were moved to an internal "failed" folder. Now, the failed log is moved to the FTP directory, enabling users to view data that did not pass harmonisation.
What's Changed
- Fix typo in palindromes by @smlmbrt in #106
- Dev 1.9 by @jiyue1214 in #111
- Updates to a new version v1.1.10 (GWAS_SSF) by @jiyue1214 in #112
New Contributors
Full Changelog: v1.1.9...v1.1.10
Contributors
smlmbrt and jiyue1214