ERROR: Analysis exception: 'MatrixTable.annotate_rows': name collision with field indexed by ['row', 'column']: 'AF' #2209

bnouyou · 2021-10-22T08:06:05Z

bnouyou
Oct 22, 2021

Hello,

I have launch the pipeline today for loading my samples, and I have obtained this error message at step 2:

[Stage 2:======================================================>(493 + 2) / 500]2021-10-22 07:45:21 Hail: ERROR: Analysis exception: 'MatrixTable.annotate_rows': name collision with field indexed by ['row', 'column']: 'AF'
ERROR: [pid 40692] Worker Worker(salt=117420454, workers=1, host=273dd4bcb41b, username=root, pid=40692) failed    SeqrVCFToMTTask(source_paths=/input_vcfs/EXOME/sample.vcf.gz, dest_path=/input_vcfs/EXOME/sample.vcf.gz.mt, genome_version=37, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh37/all_reference_data/combined_reference_data_grch37.ht, clinvar_ht_path=/seqr-reference-data/GRCh37/clinvar/clinvar.GRCh37.2020-06-15.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=True, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep-GRCh37.json)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/hail-elasticsearch-pipelines/luigi_pipeline/seqr_loading.py", line 55, in run
    self.read_vcf_write_mt()
  File "/hail-elasticsearch-pipelines/luigi_pipeline/seqr_loading.py", line 79, in read_vcf_write_mt
    overwrite=True).select_annotated_mt()
  File "/hail-elasticsearch-pipelines/luigi_pipeline/lib/model/base_mt_schema.py", line 133, in annotate_all
    getattr(self, atn_fn[0])(overwrite=overwrite)
  File "/hail-elasticsearch-pipelines/luigi_pipeline/lib/model/base_mt_schema.py", line 70, in wrapper
    self.mt = self.mt.annotate_rows(**annotation)
  File "<decorator-gen-1064>", line 2, in annotate_rows
  File "/usr/local/lib/python3.7/site-packages/hail/typecheck/check.py", line 614, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/usr/local/lib/python3.7/site-packages/hail/matrixtable.py", line 956, in annotate_rows
    check_annotate_exprs(caller, named_exprs, self._row_indices, {self._col_axis})
  File "/usr/local/lib/python3.7/site-packages/hail/utils/misc.py", line 452, in check_annotate_exprs
    check_collisions(caller, list(named_exprs), indices)
  File "/usr/local/lib/python3.7/site-packages/hail/utils/misc.py", line 347, in check_collisions
    raise ExpressionException(msg)
hail.expr.expressions.base_expression.ExpressionException: 'MatrixTable.annotate_rows': name collision with field indexed by ['row', 'column']: 'AF'
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   SeqrVCFToMTTask__seqr_reference__VARIANTS__input_vcfs_EXOM_97170af1c5   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 2 pending tasks possibly being run by other workers
DEBUG: There are 2 pending tasks unique to this worker
DEBUG: There are 2 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=117420454, workers=1, host=273dd4bcb41b, username=root, pid=40692) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====

Scheduled 3 tasks of which:
* 1 complete ones were encountered:
    - 1 VcfFile(filename=/input_vcfs/EXOME/sample.vcf.gz)
* 1 failed:
    - 1 SeqrVCFToMTTask(...)
* 1 were left pending, among these:
    * 1 had failed dependencies:
        - 1 SeqrMTToESTask(...)

This progress looks :( because there were failed tasks

I haven't had this error before, and this is the same error for all my samples.

Thank you in advance for your help.

Answered by hanars

Oct 22, 2021

Can you make sure you are running the pipeline with a clean VCF as input and you don't have any intermediate files cached from previous pipeline runs that are being used

View full answer

hanars · 2021-10-22T13:49:38Z

hanars
Oct 22, 2021
Maintainer

Can you make sure you are running the pipeline with a clean VCF as input and you don't have any intermediate files cached from previous pipeline runs that are being used

3 replies

bnouyou Oct 25, 2021
Author

How can I make sure about intermediate files ?
For testing, I have jus installed a new seqr instance on a new server, and I ran the pipeline.
On "old" VCF files (VCF files that I have already imported on another seqr instance 1 month before), it's working. But on the VCF file of my last run, I have this error message for all files.
Does it mean that the VCF file is not formated correctly ?
This VCF file is created by an analysis pipeline, maybe the person in charge of this pipeline has changed something. I am wondering if the problem could be because of the VCF file.

hanars Oct 25, 2021
Maintainer

yes, this sounds like an issue with your input VCF

bnouyou Oct 26, 2021
Author

Ok thank you, I will ask about the creation of this VCF file.

GeorginaAmbriz · 2023-03-23T18:53:37Z

GeorginaAmbriz
Mar 23, 2023

Hello,
I had the same error:
I'm trying to upload a wgs trio call set with the on-prem method. I've already copied the file to the ./data/input_vcfs/GRCh38 directory, and I succesfully uploaded the reference genome. So, I'm in step 4 of "Option #2: annotate and load on-prem". To upload the vcf I did:

FILE_PATH=GRCh38
FILENAME=trio1_hg38_test_filter.vcf.gz
BUILD_VERSION=38
SAMPLE_TYPE=WGS
INDEX_NAME=wgs_trio1

INPUT_FILE_PATH=${FILE_PATH}/${FILENAME}  
 docker-compose exec pipeline-runner load_data.sh $BUILD_VERSION $SAMPLE_TYPE $INDEX_NAME $INPUT_FILE_PATH

The output the I had is:

+ BUILD_VERSION=38
+ SAMPLE_TYPE=WGS
+ INDEX_NAME=wgs_trio1
+ INPUT_FILE_PATH=GRCh38/trio1_hg38_test_filter.vcf.gz
+ case ${BUILD_VERSION} in
+ FULL_BUILD_VERSION=GRCh38
+ SOURCE_FILE=/input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz
+ DEST_FILE=/input_vcfs/GRCh38/trio1_hg38_test_filter.mt
+ python3 -m seqr_loading SeqrMTToESTask --local-scheduler --reference-ht-path /seqr-reference-data/GRCh38/combined_reference_data_grch38.ht --clinvar-ht-path /seqr-reference-data/GRCh38/clinvar.GRCh38.ht --vep-config-json-path /vep_configs/vep-GRCh38-loftee.json --es-host elasticsearch --es-index-min-num-shards 1 --sample-type WGS --es-index wgs_trio1 --genome-version 38 --source-paths /input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz --dest-path /input_vcfs/GRCh38/trio1_hg38_test_filter.mt
/usr/local/lib/python3.7/site-packages/luigi/parameter.py:347: UserWarning: OptionalParameter "source_path" with value "<luigi.parameter.Parameter object at 0x7f3e2eda8908>" is not of type string or None.
  param_name, param_value))
/usr/local/lib/python3.7/site-packages/luigi/parameter.py:283: UserWarning: Parameter "es_password" with value "None" is not of type string.
  warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
/usr/local/lib/python3.7/site-packages/luigi/parameter.py:283: UserWarning: Parameter "interval_ref_ht_path" with value "None" is not of type string.
  warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
/usr/local/lib/python3.7/site-packages/luigi/parameter.py:283: UserWarning: Parameter "hgmd_ht_path" with value "None" is not of type string.
  warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
/usr/local/lib/python3.7/site-packages/elasticsearch/connection/base.py:190: ElasticsearchDeprecationWarning: Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security.
  warnings.warn(message, category=ElasticsearchDeprecationWarning)
DEBUG: Checking if SeqrMTToESTask(source_path=<luigi.parameter.Parameter object at 0x7f3e2eda8908>, use_temp_loading_nodes=True, es_host=elasticsearch, es_port=9200, es_index=wgs_trio1, es_username=pipeline, es_password=None, es_index_min_num_shards=1, source_paths=/input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz, dest_path=/input_vcfs/GRCh38/trio1_hg38_test_filter.mt, genome_version=38, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=None, clinvar_ht_path=/seqr-reference-data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WGS, dont_validate=False, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep_configs/vep-GRCh38-loftee.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz) is complete
DEBUG: Checking if SeqrVCFToMTTask(source_paths=/input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz, dest_path=/input_vcfs/GRCh38/trio1_hg38_test_filter.mt, genome_version=38, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=/seqr-reference-data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WGS, dont_validate=False, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep_configs/vep-GRCh38-loftee.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=) is complete
INFO: Informed scheduler that task   SeqrMTToESTask__seqr_reference__VARIANTS__input_vcfs_GRCh_86a5a3fc08   has status   PENDING
DEBUG: Checking if VcfFile(filename=/input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz) is complete
INFO: Informed scheduler that task   SeqrVCFToMTTask__seqr_reference__VARIANTS__input_vcfs_GRCh_6bd2c01217   has status   PENDING
INFO: Informed scheduler that task   VcfFile__input_vcfs_GRCh_edcfd847db   has status   DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 2
INFO: [pid 2919] Worker Worker(salt=692780041, workers=1, host=3a11103263e8, username=root, pid=2919) running   SeqrVCFToMTTask(source_paths=/input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz, dest_path=/input_vcfs/GRCh38/trio1_hg38_test_filter.mt, genome_version=38, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=/seqr-reference-data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WGS, dont_validate=False, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep_configs/vep-GRCh38-loftee.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=)
Initializing Hail with default parameters...
2023-03-23 18:01:18.945 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Running on Apache Spark version 3.1.3
SparkUI available at http://3a11103263e8:4040
Welcome to
     __  __     <>__
    / /_/ /__  __/ /
   / __  / _ `/ / /
  /_/ /_/\_,_/_/_/   version 0.2.105-acd89e80c345
LOGGING: writing to /hail-20230323-1801-0.2.105-acd89e80c345.log
{'_Task__hash': -4367733919609787394,
 'clinvar_ht_path': '/seqr-reference-data/GRCh38/clinvar.GRCh38.ht',
 'dataset_type': 'VARIANTS',
 'decrease_running_resources': <bound method TaskStatusReporter.decrease_running_resources of <luigi.worker.TaskStatusReporter object at 0x7f3e2de38780>>,
 'dest_path': '/input_vcfs/GRCh38/trio1_hg38_test_filter.mt',
 'dont_validate': False,
 'genome_version': '38',
 'grch38_to_grch37_ref_chain': 'gs://hail-common/references/grch38_to_grch37.over.chain.gz',
 'hail_temp_dir': None,
 'hgmd_ht_path': None,
 'interval_ref_ht_path': None,
 'param_kwargs': {'clinvar_ht_path': '/seqr-reference-data/GRCh38/clinvar.GRCh38.ht',
                  'dataset_type': 'VARIANTS',
                  'dest_path': '/input_vcfs/GRCh38/trio1_hg38_test_filter.mt',
                  'dont_validate': False,
                  'genome_version': '38',
                  'grch38_to_grch37_ref_chain': 'gs://hail-common/references/grch38_to_grch37.over.chain.gz',
                  'hail_temp_dir': None,
                  'hgmd_ht_path': None,
                  'interval_ref_ht_path': None,
                  'reference_ht_path': '/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht',
                  'remap_path': None,
                  'sample_type': 'WGS',
                  'source_paths': '/input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz',
                  'subset_path': None,
                  'vep_config_json_path': '/vep_configs/vep-GRCh38-loftee.json',
                  'vep_runner': 'VEP'},
 'reference_ht_path': '/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht',
 'remap_path': None,
 'sample_type': 'WGS',
 'scheduler_messages': None,
 'set_progress_percentage': <bound method TaskStatusReporter.update_progress_percentage of <luigi.worker.TaskStatusReporter object at 0x7f3e2de38780>>,
 'set_status_message': <bound method TaskStatusReporter.update_status_message of <luigi.worker.TaskStatusReporter object at 0x7f3e2de38780>>,
 'set_tracking_url': <bound method TaskStatusReporter.update_tracking_url of <luigi.worker.TaskStatusReporter object at 0x7f3e2de38780>>,
 'source_paths': ['/input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz'],
 'subset_path': None,
 'task_id': 'SeqrVCFToMTTask__seqr_reference__VARIANTS__input_vcfs_GRCh_6bd2c01217',
 'vep_config_json_path': '/vep_configs/vep-GRCh38-loftee.json',
 'vep_runner': 'VEP'}
2023-03-23 18:01:27.029 Hail: INFO: scanning VCF for sortedness...
2023-03-23 18:01:48.253 Hail: INFO: Coerced sorted VCF - no additional import work to do
2023-03-23 18:08:01.868 Hail: INFO: Coerced sorted dataset=====>(497 + 2) / 500]
2023-03-23 18:08:25.350 Hail: INFO: Coerced sorted dataset=====>(496 + 2) / 500]
MT using schema class SeqrVariantsAndGenotypesSchema already has "filters" annotation.
MT using schema class SeqrVariantsAndGenotypesSchema already has "rg37_locus" annotation.
MT using schema class SeqrVariantsAndGenotypesSchema already has "rsid" annotation.
MT using schema class SeqrVariantsAndGenotypesSchema already has "vep" annotation.
2023-03-23 18:09:49.519 Hail: ERROR: Analysis exception: 'MatrixTable.annotate_rows': name collision with field indexed by ['column', 'row']: 'AF'
ERROR: [pid 2919] Worker Worker(salt=692780041, workers=1, host=3a11103263e8, username=root, pid=2919) failed    SeqrVCFToMTTask(source_paths=/input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz, dest_path=/input_vcfs/GRCh38/trio1_hg38_test_filter.mt, genome_version=38, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=/seqr-reference-data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WGS, dont_validate=False, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep_configs/vep-GRCh38-loftee.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/seqr-loading-pipelines/luigi_pipeline/seqr_loading.py", line 92, in run
    self.read_input_write_mt()
  File "/seqr-loading-pipelines/luigi_pipeline/seqr_loading.py", line 135, in read_input_write_mt
    mt = self.SCHEMA_CLASS(mt, **kwargs).annotate_all(overwrite=True).select_annotated_mt()
  File "/seqr-loading-pipelines/luigi_pipeline/lib/model/base_mt_schema.py", line 175, in annotate_all
    self.set_mt(self.mt.annotate_rows(**annotations_to_apply))
  File "<decorator-gen-1148>", line 2, in annotate_rows
  File "/usr/local/lib/python3.7/site-packages/hail/typecheck/check.py", line 577, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/usr/local/lib/python3.7/site-packages/hail/matrixtable.py", line 971, in annotate_rows
    check_annotate_exprs(caller, named_exprs, self._row_indices, {self._col_axis})
  File "/usr/local/lib/python3.7/site-packages/hail/utils/misc.py", line 487, in check_annotate_exprs
    check_collisions(caller, list(named_exprs), indices)
  File "/usr/local/lib/python3.7/site-packages/hail/utils/misc.py", line 382, in check_collisions
    raise ExpressionException(msg)
hail.expr.expressions.base_expression.ExpressionException: 'MatrixTable.annotate_rows': name collision with field indexed by ['column', 'row']: 'AF'
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   SeqrVCFToMTTask__seqr_reference__VARIANTS__input_vcfs_GRCh_6bd2c01217   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 2 pending tasks possibly being run by other workers
DEBUG: There are 2 pending tasks unique to this worker
DEBUG: There are 2 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=692780041, workers=1, host=3a11103263e8, username=root, pid=2919) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 3 tasks of which:
* 1 complete ones were encountered:
    - 1 VcfFile(filename=/input_vcfs/GRCh38/trio1_hg38_test_filter.vcf.gz)
* 1 failed:
    - 1 SeqrVCFToMTTask(...)
* 1 were left pending, among these:
    * 1 had failed dependencies:
        - 1 SeqrMTToESTask(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

I had already filtered the alternative chromosomes from the vcf (Although maybe they are still listed in the header). I made the joint call with GATK and I used the hg38 intervals file from the Broad Institute.
If this is because of my vcf, I'd like to know what is the right way to generate a wgs joint call set for seqr? or
If I have to allocate more nodes to Elasticsearch, How should I do it?

Thank you for your help,
Georgina

4 replies

hanars Mar 27, 2023
Maintainer

It looks like this is not a clean unannotated GATK joint called VCF but rather a VCF or intermediate MT that has been previously annotated. As I answered above, make sure you are running the pipeline with a clean VCF as input and you don't have any intermediate files cached from previous pipeline runs that are being used

bnouyou Mar 27, 2023
Author

Hello,
For me the problem was because the VCF I tried to use was created by another caller than GATK. When I used the VCF create by only GATK, I had no problem anymore.

lauragails May 12, 2023

Would VQSR quality scores be an issue?

The VCF that I've been trying in all of my other issues/discussions does have the annotations: AC,AF,AN,AS_InbreedingCoeff,AS_QD,BaseQRankSum,ClippingRankSum,DB,DP,ExcessHet,FS,InbreedingCoeff,MLEAC,MLEAF,MQ,MQRankSum,NEGATIVE_TRAIN_SITE,QD,RAW_MQ,ReadPosRankSum,SOR,VQSLOD,culprit

This vcf worked fine when loaded onto the GUI, so I assumed it was OK. Also I don't see how it would be related to the error in question (docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?). I'm baffled because docker ps works fine and my containers are healthy.

To the best of my knowledge, this VCF was joint called with GATK.

hanars May 12, 2023
Maintainer

The docker issue should be unrelated to the file format. The file format issues will have errors like the ones other users reported in this discussion, which is not what you say your error looks like

GeorginaAmbriz · 2023-05-04T05:32:31Z

GeorginaAmbriz
May 4, 2023

Hello,
I found that the input used for gatk genotype call was a gvcf file created with Dragen, and Dragen adds different FORMAT and INFO fields than haplotypecaller. I decided to remove the additional fields created by Dragen (I used bcftools annotate -x for this), and I ran the gatk genotype call again. Then, I tried to upload the resulting vcf to seqr. This seemed to be working because the upload ran for 8h (I'm using "Option #2: annotate and load on-prem" method), but it stopped again with a different error message. The commands that I used to upload were:

FILE_PATH=GRCh38
FILE_PATH=GRCh38/S32020
FILENAME=S32020.wgs.gntypCall-hardFilt.vcf.gz
BUILD_VERSION=38
SAMPLE_TYPE=WGS
INDEX_NAME=alss32020
INPUT_FILE_PATH=${FILE_PATH}/${FILENAME}  
 
docker-compose exec pipeline-runner load_data.sh $BUILD_VERSION $SAMPLE_TYPE $INDEX_NAME $INPUT_FILE_PATH

And this is the final part of the output, I wasn't able to recover the output from the beginning:

2023-05-04 04:30:40.559 Hail: WARN: struct{aa_allele: str, aa_maf: float64, afr_allele: str, afr_maf: float64, allele_string: str, amr_allele: str, amr_maf: float64, clin_sig: array<str>, end: int32, eas_allele: str, eas_maf: float64, ea_allele: str, ea_maf: float64, eur_allele: str, eur_maf: float64, exac_adj_allele: str, exac_adj_maf: float64, exac_allele: str, exac_afr_allele: str, exac_afr_maf: float64, exac_amr_allele: str, exac_amr_maf: float64, exac_eas_allele: str, exac_eas_maf: float64, exac_fin_allele: str, exac_fin_maf: float64, exac_maf: float64, exac_nfe_allele: str, exac_nfe_maf: float64, exac_oth_allele: str, exac_oth_maf: float64, exac_sas_allele: str, exac_sas_maf: float64, id: str, minor_allele: str, minor_allele_freq: float64, phenotype_or_disease: int32, pubmed: array<int32>, sas_allele: str, sas_maf: float64, somatic: int32, start: int32, strand: int32} has no field AFR at <root>.colocated_variants[element] for value JString(ATAGATAGATAGATAGATAGATAGATAGAT:0.2034)
2023-05-04 04:30:40.559 Hail: WARN: struct{aa_allele: str, aa_maf: float64, afr_allele: str, afr_maf: float64, allele_string: str, amr_allele: str, amr_maf: float64, clin_sig: array<str>, end: int32, eas_allele: str, eas_maf: float64, ea_allele: str, ea_maf: float64, eur_allele: str, eur_maf: float64, exac_adj_allele: str, exac_adj_maf: float64, exac_allele: str, exac_afr_allele: str, exac_afr_maf: float64, exac_amr_allele: str, exac_amr_maf: float64, exac_eas_allele: str, exac_eas_maf: float64, exac_fin_allele: str, exac_fin_maf: float64, exac_maf: float64, exac_nfe_allele: str, exac_nfe_maf: float64, exac_oth_allele: str, exac_oth_maf: float64, exac_sas_allele: str, exac_sas_maf: float64, id: str, minor_allele: str, minor_allele_freq: float64, phenotype_or_disease: int32, pubmed: array<int32>, sas_allele: str, sas_maf: float64, somatic: int32, start: int32, strand: int32} has no field EAS at <root>.colocated_variants[element] for value JString(ATAGATAGATAGATAGATAGATAGATAGAT:0.4254)
2023-05-04 04:30:40.559 Hail: WARN: struct{aa_allele: str, aa_maf: float64, afr_allele: str, afr_maf: float64, allele_string: str, amr_allele: str, amr_maf: float64, clin_sig: array<str>, end: int32, eas_allele: str, eas_maf: float64, ea_allele: str, ea_maf: float64, eur_allele: str, eur_maf: float64, exac_adj_allele: str, exac_adj_maf: float64, exac_allele: str, exac_afr_allele: str, exac_afr_maf: float64, exac_amr_allele: str, exac_amr_maf: float64, exac_eas_allele: str, exac_eas_maf: float64, exac_fin_allele: str, exac_fin_maf: float64, exac_maf: float64, exac_nfe_allele: str, exac_nfe_maf: float64, exac_oth_allele: str, exac_oth_maf: float64, exac_sas_allele: str, exac_sas_maf: float64, id: str, minor_allele: str, minor_allele_freq: float64, phenotype_or_disease: int32, pubmed: array<int32>, sas_allele: str, sas_maf: float64, somatic: int32, start: int32, strand: int32} has no field SAS at <root>.colocated_variants[element] for value JString(ATAGATAGATAGATAGATAGATAGATAGAT:0.5056)
2023-05-04 04:30:40.679 Hail: WARN: Can't convert JSON value JArray(List(JString(A0A087X1S4))) to type str at <root>.transcript_consequences[element].trembl.
2023-05-04 04:30:40.710 Hail: WARN: struct{allele_num: int32, amino_acids: str, appris: str, biotype: str, canonical: int32, ccds: str, cdna_start: int32, cdna_end: int32, cds_end: int32, cds_start: int32, codons: str, consequence_terms: array<str>, distance: int32, domains: array<struct{db: str, name: str}>, exon: str, gene_id: str, gene_pheno: int32, gene_symbol: str, gene_symbol_source: str, hgnc_id: str, hgvsc: str, hgvsp: str, hgvs_offset: int32, impact: str, intron: str, lof: str, lof_flags: str, lof_filter: str, lof_info: str, minimised: int32, polyphen_prediction: str, polyphen_score: float64, protein_end: int32, protein_start: int32, protein_id: str, sift_prediction: str, sift_score: float64, strand: int32, swissprot: str, transcript_id: str, trembl: str, tsl: int32, uniparc: str, variant_allele: str} has no field mane at <root>.transcript_consequences[element] for value JString(NM_004650.3)
2023-05-04 04:30:40.714 Hail: WARN: struct{allele_num: int32, amino_acids: str, appris: str, biotype: str, canonical: int32, ccds: str, cdna_start: int32, cdna_end: int32, cds_end: int32, cds_start: int32, codons: str, consequence_terms: array<str>, distance: int32, domains: array<struct{db: str, name: str}>, exon: str, gene_id: str, gene_pheno: int32, gene_symbol: str, gene_symbol_source: str, hgnc_id: str, hgvsc: str, hgvsp: str, hgvs_offset: int32, impact: str, intron: str, lof: str, lof_flags: str, lof_filter: str, lof_info: str, minimised: int32, polyphen_prediction: str, polyphen_score: float64, protein_end: int32, protein_start: int32, protein_id: str, sift_prediction: str, sift_score: float64, strand: int32, swissprot: str, transcript_id: str, trembl: str, tsl: int32, uniparc: str, variant_allele: str} has no field flags at <root>.transcript_consequences[element] for value JArray(List(JString(cds_end_NF)))
ERROR: [pid 6883] Worker Worker(salt=194719803, workers=1, host=3a11103263e8, username=root, pid=6883) failed    SeqrVCFToMTTask(source_paths=/input_vcfs/GRCh38/S32020/S32020.wgs.gntypCall-hardFilt.vcf.gz, dest_path=/input_vcfs/GRCh38/S32020/S32020.mt, genome_version=38, vep_runner=VEP, reference_ht_path=/seqr-reference-data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=/seqr-reference-data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WGS, dont_validate=False, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=/vep_configs/vep-GRCh38-loftee.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()
  File "/seqr-loading-pipelines/luigi_pipeline/seqr_loading.py", line 92, in run
    self.read_input_write_mt()
  File "/seqr-loading-pipelines/luigi_pipeline/seqr_loading.py", line 140, in read_input_write_mt
    mt.write(self.output().path, stage_locally=True, overwrite=True)
  File "<decorator-gen-1200>", line 2, in write
  File "/usr/local/lib/python3.7/site-packages/hail/typecheck/check.py", line 577, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/usr/local/lib/python3.7/site-packages/hail/matrixtable.py", line 2584, in write
    Env.backend().execute(ir.MatrixWrite(self._mir, writer))
  File "/usr/local/lib/python3.7/site-packages/hail/backend/py4j_backend.py", line 104, in execute
    self._handle_fatal_error_from_backend(e, ir)
  File "/usr/local/lib/python3.7/site-packages/hail/backend/backend.py", line 189, in _handle_fatal_error_from_backend
    raise HailUserError(message_and_trace) from None
hail.utils.java.HailUserError: Error summary: HailException: array index out of bounds: index=2, length=2
------------
Hail stack trace:
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)

  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)

  File "/seqr-loading-pipelines/luigi_pipeline/seqr_loading.py", line 282, in <module>
    luigi.run() or sys.exit(1)

  File "/usr/local/lib/python3.7/site-packages/luigi/interface.py", line 194, in run
    luigi_run_result = _run(*args, **kwargs)

  File "/usr/local/lib/python3.7/site-packages/luigi/interface.py", line 211, in _run
    return _schedule_and_run([cp.get_task_obj()], worker_scheduler_factory)

  File "/usr/local/lib/python3.7/site-packages/luigi/interface.py", line 173, in _schedule_and_run
    success &= worker.run()

  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 1215, in run
    self._run_task(get_work_response.task_id)

  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 1019, in _run_task
    task_process.run()

  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 293, in run
    super(ContextManagedTaskProcess, self).run()

  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()

  File "/usr/local/lib/python3.7/site-packages/luigi/worker.py", line 141, in _run_get_new_deps
    task_gen = self.task.run()

  File "/seqr-loading-pipelines/luigi_pipeline/seqr_loading.py", line 92, in run
    self.read_input_write_mt()

  File "/seqr-loading-pipelines/luigi_pipeline/seqr_loading.py", line 120, in read_input_write_mt
    mt = self.annotate_old_and_split_multi_hts(mt)

  File "/seqr-loading-pipelines/luigi_pipeline/seqr_loading.py", line 149, in annotate_old_and_split_multi_hts
    return hl.split_multi_hts(mt.annotate_rows(locus_old=mt.locus, alleles_old=mt.alleles))

  File "/usr/local/lib/python3.7/site-packages/hail/methods/statgen.py", line 2172, in split_multi_hts
    (hl.range(0, 3).map(lambda i:

  File "/usr/local/lib/python3.7/site-packages/hail/methods/statgen.py", line 2176, in <lambda>
    ).map(lambda j: split.PL[j]))))))

  File "/usr/local/lib/python3.7/site-packages/hail/methods/statgen.py", line 2176, in <lambda>
    ).map(lambda j: split.PL[j]))))))

  File "/usr/local/lib/python3.7/site-packages/hail/expr/expressions/typed_expressions.py", line 448, in __getitem__
    return self._method("indexArray", self.dtype.element_type, item)

  File "/usr/local/lib/python3.7/site-packages/hail/expr/expressions/base_expression.py", line 695, in _method
    x = ir.Apply(name, ret_type, self._ir, *(a._ir for a in args))

  File "/usr/local/lib/python3.7/site-packages/hail/ir/ir.py", line 2669, in __init__
    self.save_error_info()

DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task   SeqrVCFToMTTask__seqr_reference__VARIANTS__input_vcfs_GRCh_b6409ec6aa   has status   FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 2 pending tasks possibly being run by other workers
DEBUG: There are 2 pending tasks unique to this worker
DEBUG: There are 2 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=194719803, workers=1, host=3a11103263e8, username=root, pid=6883) was stopped. Shutting down Keep-Alive thread
INFO: 
===== Luigi Execution Summary =====

Scheduled 3 tasks of which:
* 1 complete ones were encountered:
    - 1 VcfFile(filename=/input_vcfs/GRCh38/S32020/S32020.wgs.gntypCall-hardFilt.vcf.gz)
* 1 failed:
    - 1 SeqrVCFToMTTask(...)
* 1 were left pending, among these:
    * 1 had failed dependencies:
        - 1 SeqrMTToESTask(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

This is a different error. Is it a different issue? How can I solve it ?
If this error happened because my clean up of the initial Dragen gvcf file didn't work. Do you know a way to make the dragen gvcf file equivalent to a g.vcf from haplotypecaller. It would be really useful to find a way to turn this around because I have a big amount of samples processed with Dragen.

Thank you for your support

1 reply

hanars May 4, 2023
Maintainer

I'm sorry, but unfortunately seqr does not support Dragen VCFs at this time, and we do not have any documentation/resources available for converting Dragen VCFs to GATK VCFs, as this is outside the seqr team's expertise.

The error you are getting is occurring during the step in the pipleine that calls the hail method split_multi_hts, and in hail's method documentation it specifies what it expects the entry schema to look like, so maybe try and see if you can format your VCF so that the entry fields align with that specification

gilmorera · 2024-07-24T14:21:22Z

gilmorera
Jul 24, 2024

Hi, I'm also trying to process a DRAGEN VCF file, but I'm getting similar issues.

I removed the conflicting FORMAT/AF field:

##FORMAT=<ID=AF,Number=A,Type=Float,Description="Allele fractions for alt alleles in the order listed">
##INFO=<ID=AF,Number=A,Type=Float,Description="Allele Frequency, for each ALT allele, in the same order as listed">

I removed phasing
I normalized the VCF with bcftools (e.g. split_multi_hts)

I'm curious if DRAGEN is now supported by seqr? And if so could you point me to any pipeline changes that occurred to include this support?

2 replies

gilmorera Sep 6, 2024

We also removed alt contigs which helped us process the data with the hail pipeline. So our next steps are to see if we can do our standard variant search analysis. Are there any concerns with how this data will be represented and with how the variant search functions?

hanars Sep 10, 2024
Maintainer

DRAGEN is supported in our new v3 pipeline, which is not yet available for local installations

hanars · 2024-07-24T14:24:14Z

hanars
Jul 24, 2024
Maintainer

DRAGEN support will be added by the end of the year

4 replies

gilmorera Jul 24, 2024

THank you :)

gilmorera Sep 4, 2024

Hi @hanars are you all planning to support DRAGEN SV calls in some way? Our team is currently exploring ways to process this type of data for seqr, but the MANTA algorithm only supplies single sample SV files which are hard to merge (or optionally small cohorts < 10 samples).

gilmorera Sep 4, 2024

https://dragen-docs-public.s3.amazonaws.com/DRAGEN+User+Guide+v3.4.5.pdf

hanars Sep 10, 2024
Maintainer

SVs are not supported for local installations at all at this time, for DRAGEN or GATK. We are hopeful that we will be able to add SV support next year, and when we do we will provide documentation on which formats we will be accepting, but that is still not decided yet

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERROR: Analysis exception: 'MatrixTable.annotate_rows': name collision with field indexed by ['row', 'column']: 'AF' #2209

{{title}}

Replies: 5 comments 14 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

ERROR: Analysis exception: 'MatrixTable.annotate_rows': name collision with field indexed by ['row', 'column']: 'AF' #2209

Replies: 5 comments · 14 replies

hanars Oct 22, 2021 Maintainer

bnouyou Oct 25, 2021 Author

hanars Oct 25, 2021 Maintainer

bnouyou Oct 26, 2021 Author

hanars Mar 27, 2023 Maintainer

bnouyou Mar 27, 2023 Author

hanars May 12, 2023 Maintainer

hanars May 4, 2023 Maintainer

hanars Sep 10, 2024 Maintainer

hanars Jul 24, 2024 Maintainer

hanars Sep 10, 2024 Maintainer

Replies: 5 comments 14 replies

hanars
Oct 22, 2021
Maintainer

bnouyou Oct 25, 2021
Author

hanars Oct 25, 2021
Maintainer

bnouyou Oct 26, 2021
Author

hanars Mar 27, 2023
Maintainer

bnouyou Mar 27, 2023
Author

hanars May 12, 2023
Maintainer

hanars May 4, 2023
Maintainer

hanars Sep 10, 2024
Maintainer

hanars
Jul 24, 2024
Maintainer

hanars Sep 10, 2024
Maintainer