Unable to view HGVS annotations in local install

[lecb]

Describe the bug
I am unable to view HGVS annotations for variants (I've never been able to since installing local seqr version). I've dumped psql references databases and recreated to check that wasn't the reason.

Link to page(s) where bug is occurring
All variant pages

Scope of the bug
Across all projects

Screenshots

FYI
I have upgraded to the latest version of seqr - which has fixed the issues of #2231. As an aside, panelApp isn't working. I note there is a panelapp folder within the docker seqr container. Happy to open a separate issue for this.

[hanars]

HGVS annotations come in through the pipeline, so the first thing you need to do is make sure the data is actually in your raw data. This might be something better addressed by me speaking directly with whoever on your end manages data loading

[eseaby]

I manage data loading at the moment… we are a small research group doing pretty much everything by ourselves!

[hanars]

Good to know! Okay so the first thing I need to know is if you can confirm that the hgvs data is properly added in elasticsearch. In the data in elasticsearch, each variant has a field called sortedTranscriptConsequences and for each entry in that list there should be values under hgvsc and hgvsp

[lecb]

Thanks. I'm not sure if I'm looking at the right thing, but using the following command:

curl 'http://localhost:9200/ibd2021/_search?q=*' | grep 'hgvs' I can see NULL is the result of each hgvsc and hgsvp.

[hanars]

You are looking at the right thing, and that means that the issue is actually with the data loading itself, as you are not exporting hgvs annotations. Do you know if your group has made any tweaks on your end to the loading pipeline code? Maybe running vep elsewhere? You could also try making sure you have the latest version of the piepline and then try running it on a small dataset and see if its now exporting hgvs

[lecb]

We have our own pipeline using GATK to generate our joint called vcf. We then put the joint-called vcf directly into seqr using the following command:

docker-compose exec pipeline-runner /bin/bash
python3 -m seqr_loading SeqrMTToESTask --local-scheduler \
--reference-ht-path /seqr-reference-data/GRCh38/combined_reference_data_grch38.ht \
--clinvar-ht-path /seqr-reference-data/GRCh38/clinvar.GRCh38.2020-06-15.ht \
--vep-config-json-path /vep-GRCh38.json \
--es-host elasticsearch \
--es-index-min-num-shards 3 \
--sample-type WES \
--es-index ibd2021 \
--genome-version 38 \
--source-paths /input_vcfs/ibd2021.vcf.gz \
--dest-path /vep_data/ibd2021.mt \
--dont-validate #use this as don't have space to save log files

So I'm guessing it must be at this step...

Or are you talking about a different "pipeline"...

[hanars]

That error says you aren't setting the --dest-path parameter in your command

[lecb]

Sorry, should have said this is the whole command:

python3 -m seqr_loading SeqrMTToESTask --local-scheduler \
--reference-ht-path /seqr-reference-data/GRCh38/combined_reference_data_grch38.ht \
--clinvar-ht-path /seqr-reference-data/GRCh38/clinvar.GRCh38.2020-06-15.ht \
--vep-config-json-path /vep85-GRCh38-loftee-gcloud.json
--es-host elasticsearch \
--es-index-min-num-shards 3 \
--sample-type WES \
--es-index ADupdated \
--genome-version 38 \
--source-paths /input_vcfs/AD_28_joint_called.recalibrated.vcf.gz \
--dest-path /vep_data/ADupdated.mt \
--dont-validate

gives the same issue....

[hanars]

you're missing a backslash - --vep-config-json-path /vep85-GRCh38-loftee-gcloud.json shoudl be --vep-config-json-path /vep85-GRCh38-loftee-gcloud.json \

[lecb]

Thanks.. unfortunately still throwing a bug..

/usr/local/lib/python3.7/site-packages/luigi/parameter.py:283: UserWarning: Parameter "hgmd_ht_path" with value "None" is not of type string.
  warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if SeqrMTToESTask(source_path=<luigi.parameter.Parameter object at 0x7f1942f9d1d0>, use_temp_loading_nodes=True, es_host=elasticsearch, es_port=9200, es_index=adupdated, es_username=pipeline, es_password=None, es_index_min_num_shards=3, source_paths=/input_vcfs/AD_28_joint_called.recalibrated.vcf.gz, dest_path=/vep_data/adupdated.mt, genome_version=38, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht, clinvar_ht_path=/seqr-reference-data/GRCh38/clinvar.GRCh38.2020-06-15.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=True, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep85-GRCh38-loftee-gcloud.json) is complete
DEBUG: Checking if SeqrVCFToMTTask(source_paths=/input_vcfs/AD_28_joint_called.recalibrated.vcf.gz, dest_path=/vep_data/adupdated.mt, genome_version=38, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht, clinvar_ht_path=/seqr-reference-data/GRCh38/clinvar.GRCh38.2020-06-15.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=True, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep85-GRCh38-loftee-gcloud.json) is complete
INFO: Informed scheduler that task   SeqrMTToESTask__seqr_reference__VARIANTS__vep_data_adupda_9a4d3de4a5   has status   PENDING
DEBUG: Checking if VcfFile(filename=/input_vcfs/AD_28_joint_called.recalibrated.vcf.gz) is complete
INFO: Informed scheduler that task   SeqrVCFToMTTask__seqr_reference__VARIANTS__vep_data_adupda_c1a1aa2b1a   has status   PENDING
INFO: Informed scheduler that task   VcfFile__input_vcfs_AD_2_025f3d57ac   has status   DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 2
INFO: [pid 256] Worker Worker(salt=339970122, workers=1, host=3abbba99a730, username=root, pid=256) running   SeqrVCFToMTTask(source_paths=/input_vcfs/AD_28_joint_called.recalibrated.vcf.gz, dest_path=/vep_data/adupdated.mt, genome_version=38, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht, clinvar_ht_path=/seqr-reference-data/GRCh38/clinvar.GRCh38.2020-06-15.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=True, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep85-GRCh38-loftee-gcloud.json)
ERROR: [pid 256] Worker Worker(salt=339970122, workers=1, host=3abbba99a730, username=root, pid=256) failed    SeqrVCFToMTTask(source_paths=/input_vcfs/AD_28_joint_called.recalibrated.vcf.gz, dest_path=/vep_data/adupdated.mt, genome_version=38, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht, clinvar_ht_path=/seqr-reference-data/GRCh38/clinvar.GRCh38.2020-06-15.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=True, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep85-GRCh38-loftee-gcloud.json)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/hail-elasticsearch-pipelines/luigi_pipeline/seqr_loading.py", line 53, in run
    if self.vep_config_json_path: check_if_path_exists(self.vep_config_json_path, "vep_config_json_path")
  File "/hail-elasticsearch-pipelines/luigi_pipeline/seqr_loading.py", line 18, in check_if_path_exists
    raise ValueError(f"{label} path not found: {path}")
ValueError: vep_config_json_path path not found: /vep85-GRCh38-loftee-gcloud.json
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   SeqrVCFToMTTask__seqr_reference__VARIANTS__vep_data_adupda_c1a1aa2b1a   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 2 pending tasks possibly being run by other workers
DEBUG: There are 2 pending tasks unique to this worker
DEBUG: There are 2 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=339970122, workers=1, host=3abbba99a730, username=root, pid=256) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 3 tasks of which:
* 1 complete ones were encountered:
    - 1 VcfFile(filename=/input_vcfs/AD_28_joint_called.recalibrated.vcf.gz)
* 1 failed:
    - 1 SeqrVCFToMTTask(...)
* 1 were left pending, among these:
    * 1 had failed dependencies:
        - 1 SeqrMTToESTask(...)

This progress looks :( because there were failed tasks```

[hanars]

It looks like you don't actually have that VEP config downloaded. Make sure you've downloaded all the prerequisites listed in the "annotate on prem" section of the local deployment readme: https://github.com/broadinstitute/seqr/blob/master/deploy/LOCAL_INSTALL.md#option-2-annotate-and-load-on-prem

[lecb]

Thanks. This must have changed from when we originally installed seqr?

[hanars]

These are gene specific details - I thought we were talking about variants. Gene reference data comes from a totally different source and is never ingested during the pipeline. For gene level data, run:

docker-compose exec seqr python -u manage.py update_all_reference_data --use-cached-omim

[lecb]

Thanks. The LOF and missense constraint bits are fixed, but the variant level data (hgvs) is not displaying even after re-loading the data... PanelApp doesn't appear either...

[hanars]

Hmm so honestly we have kind of hit my limits on my ability to debug issues with the pipeline. Maybe @mike-w-wilson or @bw2 has seen this issue before with VEP not adding hgvs before?

For panel app, thats also a separate command. You need to run the following if you want both PanelApp UK and PanelApp Australia, just run one of them if you only want one

docker-compose exec seqr python -u manage.py import_all_panels https://panelapp.agha.umccr.org/api/v1 --label=AU
docker-compose exec seqr python -u manage.py import_all_panels https://panelapp.genomicsengland.co.uk/api/v1 --label=UK

[lecb]

Thanks so much Hana. Will wait to see if @bw2 or @mike-w-wilson may be able to help.

[hanars]

From another user with a local installation, they found installing some additional hal utilities fixed their problems (#2264 (reply in thread)) . Maybe it is worth trying that? You would run

docker-compose exec pipeline python3 -m pip install git+https://github.com/bw2/hail-utils.git

[hanars]

Hi Ellie,

Someone was able to figure out how to solve the hgvs issue (#2695) and the latest release of the docker image should fix the problem. Note that in order to get the fix you will need to re-run the download_reference_data step and reload the data

[lecb]

This worked! Thanks!