Migrating old indexes to new ES schema

[samarthaFFM]

Hi @hanars,

We would like to migrate Elastisearch indexes (annotated vcfs from older non-docker based local install of SEQR) from the old Elastisearch schema to the new schema in the current SEQR version. Is there a way to carry forward all of the annotations + indexes together to the new SEQR version?

If so, could you please help us. Thanks!

[hanars]

Hi Samarth,

There is no way to move indices directly between elasticsearch clusters, so you will have to re-run the export step of the loading pipeline to get the indices into your new elasticsearch cluster. Depending on how old the indices are and what schema they conform to you may be able to use the intermediate matrix table generated by the loading pipeline and only re-run the export step but not the annotation step. However, if you originally loaded the data on an old version of the pipeline and do not have that intermediate matrix table or the schema of that table is in a different format then this is not possible and you have to re-run the full pipeline anyways.

I think unless you have loaded the data within the past couple months, you should just start from a joint called VCF and reload all the data using the latest version of the pipeline. In addition to ensuring the schema will match up, you will also get updated reference data and annotations, which a desirable update

Let me know if you have any further questions,
Hana

[WMAPernice]

Hey Hana,

Sorry to revive this old thread but could give us a pointer on how to run just the export step of the loading pipeline? How should we e.g. modify docker/bin/load_data.sh in order to get that done?

Thanks so much for any help you can provide.
Wolf

[hanars]

To clarify, you need to run the loading pipeline all the way through the export step, following the standard directions for loading data: https://github.com/broadinstitute/seqr/blob/master/deploy/LOCAL_INSTALL.md#annotating-and-loading-vcf-callsets

The reason a I specified just the export step is that newer versions of the pipeline will checkpoint the annotation step, so if it is already completed it will automatically reuse the annotation step from the previous run and only run the export step. However, that does depend on which version of the pipeline you originally ran and what intermediate files you kept. Per my above recommendation:

unless you have loaded the data within the past couple months, you should just start from a joint called VCF and reload all the data using the latest version of the pipelin