Cannot add dataset to project in local install

[andbanman]

For local install, we cannot add a dataset to a project. Each step through loading the data into ES succeeded, but when it comes time to add the dataset to a project it fails with error

Matches not found for ES sample ids: BH11035-1, BH11035-2, BH11035-3. Uploading a mapping file for these samples,
or select the "Ignore extra samples in callset" checkbox to ignore.

Here are the seqr logs

seqr_1             | {"timestamp": "2022-06-02 16:54:48,604", "severity": "WARNING", "message": "Not Found: /api/ui_counters/_report"}
seqr_1             | {"timestamp": "2022-06-02 16:54:51,682", "severity": "INFO", "message": "Loaded index_metadata__test from redis"}
seqr_1             | {"timestamp": "2022-06-02 16:54:51,720", "severity": "INFO", "message": "POST http://elasticsearch:9200/test/_search?size=0 [status:200 request:0.004s]"}
seqr_1             | {"timestamp": "2022-06-02 16:54:51,760", "severity": "WARNING", "httpRequest": {"requestMethod": "POST", "requestUrl": "http://seqr:9088/api/project/R0001_test/add_dataset/variants", "status": 400, "responseSize": 219, "userAgent": "Mozilla/5.0 (Mac
intosh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0", "remoteIp": "10.99.0.27", "referer": "http://seqr:9088/project/R0001_test/project_page", "protocol": "HTTP/1.1"}, "requestBody": {"datasetType": "VARIANTS", "elasticsearchIndex": "test"}, "message":
"Matches not found for ES sample ids: BH11035-1, BH11035-2, BH11035-3. Uploading a mapping file for these samples, or select the \"Ignore extra samples in callset\" checkbox to ignore.", "user": "[email protected]"}
seqr_1             | {"timestamp": "2022-06-02 16:54:51,761", "severity": "WARNING", "message": "Bad Request: /api/project/R0001_test/add_dataset/variants"}
seqr_1             | {"timestamp": "2022-06-02 16:55:35,650", "severity": "WARNING", "httpRequest": {"requestMethod": "POST", "requestUrl": "http://seqr:9088/api/ui_counters/_report", "status": 404, "responseSize": 25428, "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac
OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0", "remoteIp": "10.99.0.27", "referer": "http://seqr:9088/app/dev_tools", "protocol": "HTTP/1.1"}, "requestBody": {"report": {"reportVersion": 3, "application_usage": {"dev_tools-main": {"minutesOnScreen": 0. 05235000000
000001, "numberOfClicks": 0, "appId": "dev_tools", "viewId": "main"}}}}, "user": "[email protected]"}
seqr_1             | {"timestamp": "2022-06-02 16:55:35,651", "severity": "WARNING", "message": "Not Found: /api/ui_counters/_report"}
seqr_1             | {"timestamp": "2022-06-02 16:55:41,740", "severity": "INFO", "httpRequest": {"requestMethod": "POST", "requestUrl": "http://seqr:9088/api/console/proxy?path=_aliases&method=GET", "status": 200, "responseSize": 719, "userAgent": "Mozilla/5.0 (Macinto
sh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0", "remoteIp": "10.99.0.27", "referer": "http://seqr:9088/app/dev_tools", "protocol": "HTTP/1.1"}, "user": "[email protected]"}
seqr_1             | {"timestamp": "2022-06-02 16:55:41,745", "severity": "INFO", "httpRequest": {"requestMethod": "POST", "requestUrl": "http://seqr:9088/api/console/proxy?path=_template&method=GET", "status": 200, "responseSize": 5721, "userAgent": "Mozilla/5.0 (Macin
tosh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0", "remoteIp": "10.99.0.27", "referer": "http://seqr:9088/app/dev_tools", "protocol": "HTTP/1.1"}, "user": "[email protected]"}
seqr_1             | {"timestamp": "2022-06-02 16:55:41,747", "severity": "INFO", "httpRequest": {"requestMethod": "POST", "requestUrl": "http://seqr:9088/api/console/proxy?path=_mapping&method=GET", "status": 200, "responseSize": 11856, "userAgent": "Mozilla/5.0 (Macin
tosh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0", "remoteIp": "10.99.0.27", "referer": "http://seqr:9088/app/dev_tools", "protocol": "HTTP/1.1"}, "user": "[email protected]"}
seqr_1             | {"timestamp": "2022-06-02 16:55:55,412", "severity": "WARNING", "httpRequest": {"requestMethod": "POST", "requestUrl": "http://seqr:9088/api/ui_counters/_report", "status": 404, "responseSize": 25428, "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac
OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0", "remoteIp": "10.99.0.27", "referer": "http://seqr:9088/app/dev_tools", "protocol": "HTTP/1.1"}, "requestBody": {"report": {"reportVersion": 3, "application_usage": {"dev_tools-main": {"minutesOnScreen": 0. 05235000000
000001, "numberOfClicks": 0, "appId": "dev_tools", "viewId": "main"}}}}, "user": "[email protected]"}
seqr_1             | {"timestamp": "2022-06-02 16:55:55,414", "severity": "WARNING", "message": "Not Found: /api/ui_counters/_report"}

Meanwhile this ES query

GET /test/_search
{
  "query": {
    "match_all": {}
  }
}

yields annotations that look reasonable and have the correct sample ids

...
          "genotypes" : [
            {
              "num_alt" : 0,
              "gq" : 99,
              "ab" : 0.0,
              "dp" : 34.0,
              "sample_id" : "BH11035-1"
            },
            {
              "num_alt" : 1,
              "gq" : 99,
              "ab" : 0.59375,
              "dp" : 32.0,
              "sample_id" : "BH11035-2"
            },
            {
              "num_alt" : 0,
              "gq" : 99,
              "ab" : 0.0,
              "dp" : 33.0,
              "sample_id" : "BH11035-3"
            }
          ],
...

The upload was done with these parameters

INPUT_FILE_PATH="BH11035_trio_seqrtest.hc.g.vcf"
SAMPLE_TYPE="WGS"
BUILD_VERSION="37"
INDEX_NAME="test"

Note the input file is uncompressed.

The Hail log from the upload has many warning of these flavors:

[2022-06-01 11:28:48] 2022-06-01 18:28:48 Hail: WARN: struct{aa_allele: str, aa_maf: float64, afr_allele: str, afr_maf: float64, allele_string: str, amr_allele: str, amr_maf: float64, clin_sig: array<str>, end: int32, eas_allele: str, eas_maf: float64, ea_allele: str, ea_maf: float64, eur_allele: str, eur_maf: float64, exac_adj_allele: str, exac_adj_maf: float64, exac_allele: str, exac_afr_allele: str, exac_afr_maf: float64, exac_amr_allele: str, exac_amr_maf: float64, exac_eas_allele: str, exac_eas_maf: float64, exac_fin_allele: str, exac_fin_maf: float64, exac_maf: float64, exac_nfe_allele: str, exac_nfe_maf: float64, exac_oth_allele: str, exac_oth_maf: float64, exac_sas_allele: str, exac_sas_maf: float64, id: str, minor_allele: str, minor_allele_freq: float64, phenotype_or_disease: int32, pubmed: array<int32>, sas_allele: str, sas_maf: float64, somatic: int32, start: int32, strand: int32} has no field EUR at <root>.colocated_variants[element] for value JString(T:0.2555)

and

[2022-06-01 16:59:00] 2022-06-01 23:59:00 Hail: WARN: Can't convert JSON value JArray(List(JString(Q99218))) to type str at <root>.transcript_consequences[element].swissprot.
[2022-06-01 16:59:00] 2022-06-01 23:59:00 Hail: WARN: Can't convert JSON value JArray(List(JString(UPI000012596E))) to type str at <root>.transcript_consequences[element].uniparc.
[2022-06-01 16:59:00] 2022-06-01 23:59:00 Hail: WARN: Can't convert JSON value JArray(List(JString(J3KPK5))) to type str at <root>.transcript_consequences[element].trembl.

but no errors.

No migrations were needed

$ docker-compose exec seqr ./manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, guardian, matchmaker, panelapp, reference_data, seqr, sessions, social_django
Running migrations:
  No migrations to apply.
$ docker-compose exec seqr ./manage.py migrate --database=reference_data
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, guardian, matchmaker, panelapp, reference_data, seqr, sessions, social_django
Running migrations:
  No migrations to apply.

Images in use are

[gcr.io/seqr-project/seqr@sha256:cda900e142a038461e29dcac215f9f4732f1a38989b7eb355f9834a31116a740]
[gcr.io/seqr-project/pipeline-runner@sha256:f2604f5935b2845e5e0fb51b22e1f150f1d128e6bc46af52fb0407d6599e87a6]
[gcr.io/seqr-project/pipeline-runner@sha256:880db78b71f9294dcc465f42485b9374040a9f46abff0b117a6372c08e2d8778]
[gcr.io/seqr-project/seqr@sha256:3c290ef30268f0eeb0f75bbb37fafad542eafc3610e779bd45544709590a4877]
[gcr.io/seqr-project/redis@sha256:33ad53a44a72d8e5177b235643b4685f80e41e38f7d4708d0cfcb2151cd21f62]

This seems like a bug because the so-called missing sample IDs are present in ES. In any case, all help is much appreciated!

[hanars]

So the issue is not that the samples are missing from elasticsearch, it is that they are missing from the project. You need to upload a pedigree file using the "Update Families and Individuals" link on the project page before you add the index. Make sure the Individual IDs match the sample IDs in the VCF/elasticsearch

[andbanman]

That was the missing piece! Thank you for pointing that out. Perhaps it's obvious to add the pedigree before adding the dataset, but for a novice like me it wasn't clear. I'd suggest adding a sentence or two to the local_install instructions. Do you agree? I can submit the pull request if you like.

[hanars]

agreed that that would be helpful, please feel free to submkit a pull request