Targeted gene sequencing panels data

[SimonLi5601]

Hi seqr team,
I have a question about hail-elasticsearch-pipelines. We have some targeted gene sequencing panels data and want to process using hail-elasticsearch-pipelines and load into seqr application. It's a very small size of VCF what only contains less than 800 variants. However, we used a very powerful cluster and took 8 hours still can't finish the first step (VCF to MT). Usually this kind of size VCF should only take less than 10 minutes without including provision time. The slow step should happen after VEP and in the stage below which is triggered by mt.write since lazy evaluation. It was in normal process speed at the beginning and become extreme slow gradually. I can even observe the last 3 variants were left not being processed in a couple of hours.

https://github.com/broadinstitute/hail-elasticsearch-pipelines/blob/master/luigi_pipeline/seqr_loading.py#L112

The gene targeted data looks very similar to the regular one except the read depth is higher. I was wondering do you see any reason why it could need so long time to process when the processed variants increased. I tried an extreme case and only kept one variant in the vcf and that was successful, so I should be able exclude any format issue. I looked into the computed fields and I don't see anything is suspicious. I pasted some snippets of variants below in case it helps. Thanks for any advice!

`chr1 9710443 rs373376511 G A 75436.2 . AC=1;AF=0.005495;AN=182;AS_BaseQRankSum=3.6;AS_FS=1.115;AS_InbreedingCoeff=-0.0055;AS_MQ=59.99;AS_MQRankSum=-0.2;AS_QD=12.23;AS_ReadPosRankSum=1.5;AS_SOR=0.83;BaseQRankSum=3.68;DB;DP=254358;ExcessHet=3.0103;FS=1.115;InbreedingCoeff=-0.0055;MLEAC=1;MLEAF=0.005495;MQ=60;MQRankSum=-0.109;QD=12.23;ReadPosRankSum=1.63;SOR=0.828 GT:AD:DP:GQ:PL 0/0:3008,0:3008:99:0,120,1800 0/0:1988,0:1988:99:0,120,1800 0/0:2809,0:2809:99:0,120,1800 0/0:2766,0:2766:99:0,120,1800 0/0:3020,0:3020:99:0,120,1800 0/0:2093,0:2093:99:0,120,1800 0/0:2486,0:2486:99:0,120,1800 0/0:2818,0:2818:99:0,120,1800 0/0:3122,0:3122:99:0,120,1800 0/0:2929,0:2929:99:0,120,1800 0/0:3290,0:3290:99:0,120,1800 0/0:2932,0:2932:99:0,120,1800 0/0:2282,0:2282:99:0,120,1800 0/0:2279,0:2279:99:0,120,1800 0/0:3303,0:3303:99:0,120,1800 0/0:3660,0:3660:99:0,120,1800 0/0:3275,0:3275:99:0,120,1800 0/0:2221,0:2221:99:0,120,1800 0/0:2653,0:2653:99:0,120,1800 0/0:3393,0:3393:99:0,120,1800 0/0:2938,0:2938:99:0,120,1800 0/0:3019,0:3019:99:0,120,1800 0/0:3441,0:3441:99:0,120,1800 0/0:2924,0:2924:99:0,120,1800 0/0:2222,0:2222:99:0,120,1800 0/0:2801,0:2801:99:0,120,1800 0/0:3738,0:3738:99:0,120,1800 0/0:3295,0:3295:99:0,120,1800 0/0:2938,0:2938:99:0,120,1800 0/0:2452,0:2452:99:0,120,1800 0/0:3502,0:3502:99:0,120,1800 0/0:3377,0:3377:99:0,120,1800 0/0:2261,0:2261:99:0,120,1800 0/0:3701,0:3701:99:0,120,1800 0/0:3127,0:3127:99:0,120,1800 0/0:2299,0:2299:99:0,120,1800 0/0:2878,0:2878:99:0,120,1800 0/0:3469,0:3469:99:0,120,1800 0/0:2339,0:2339:99:0,120,1800 0/0:2575,0:2575:99:0,120,1800 0/1:3318,2848:6175:99:75456,0,88629 0/0:2524,0:2524:99:0,120,1800 0/0:2789,0:2789:99:0,120,1800 0/0:2600,0:2600:99:0,120,1800 0/0:3178,0:3178:99:0,120,1800 0/0:2070,0:2070:99:0,120,1800 0/0:2315,0:2315:99:0,120,1800 0/0:2482,0:2482:99:0,120,1800 0/0:2008,0:2008:99:0,120,1800 0/0:2398,0:2398:99:0,120,1800 0/0:2645,0:2645:99:0,120,1800 0/0:2788,0:2788:99:0,120,1800 0/0:1885,0:1885:99:0,120,1800 0/0:776,0:776:99:0,120,1800 0/0:1764,0:1764:99:0,120,1800 0/0:3095,0:3095:99:0,120,1800 0/0:4227,0:4227:99:0,120,1800 0/0:3473,0:3473:99:0,120,1800 0/0:2399,0:2399:99:0,120,1800 0/0:3592,0:3592:99:0,120,1800 0/0:2754,0:2754:99:0,120,1800 0/0:2521,0:2521:99:0,120,1800 0/0:2885,0:2885:99:0,120,1800 0/0:2870,0:2870:99:0,120,1800 0/0:2236,0:2236:99:0,120,1800 0/0:1960,0:1960:99:0,120,1800 0/0:2899,0:2899:99:0,120,1800 0/0:2665,0:2665:99:0,120,1800 0/0:2177,0:2177:99:0,120,1800 0/0:1957,0:1957:99:0,120,1800 0/0:3125,0:3125:99:0,120,1800 0/0:2705,0:2705:99:0,120,1800 0/0:3085,0:3085:99:0,120,1800 0/0:1914,0:1914:99:0,120,1800 0/0:3049,0:3049:99:0,120,1800 0/0:2486,0:2486:99:0,120,1800 0/0:2999,0:2999:99:0,120,1800 0/0:2597,0:2597:99:0,120,1800 0/0:3163,0:3163:99:0,120,1800 0/0:2408,0:2408:99:0,120,1800 0/0:3185,0:3185:99:0,120,1800 0/0:3202,0:3202:99:0,120,1800 0/0:2209,0:2209:99:0,120,1800 0/0:3113,0:3113:99:0,120,1800 0/0:2480,0:2480:99:0,120,1800 0/0:2701,0:2701:99:0,120,1800 0/0:2685,0:2685:99:0,120,1800 0/0:2557,0:2557:99:0,120,1800 0/0:2698,0:2698:99:0,120,1800 0/0:3196,0:3196:99:0,120,1800 0/0:2962,0:2962:99:0,120,1800

chr1 9710632 rs34885574;rs112806445 CAG C 1.71247e+06 . AC=48;AF=0.264;AN=182;AS_BaseQRankSum=-0.8;AS_FS=0;AS_InbreedingCoeff=-0.3582;AS_MQ=58.21;AS_MQRankSum=-1;AS_QD=1.31;AS_ReadPosRankSum=-1.2;AS_SOR=0.702;BaseQRankSum=0.353;DB;DP=518539;ExcessHet=160;FS=0;InbreedingCoeff=-1;MLEAC=48;MLEAF=0.264;MQ=59.96;MQRankSum=-0.851;QD=4.65;ReadPosRankSum=1.35;SOR=0.689 GT:AD:DP:GQ:PL 0/0:2724,38:4247:99:26289,39374,166674 0/1:3021,654:3870:99:8012,0,92337 0/1:3631,722:4560:99:9398,0,111502 0/1:3646,715:4546:99:7718,0,114110 0/1:3776,719:4716:99:8932,0,116462 0/0:869,112:3232:99:47702,55103,133621 0/1:3197,760:4023:99:6745,0,97548 0/1:3492,696:4416:99:7924,0,108082 0/1:3848,701:4614:99:11206,0,120014 0/0:2375,23:4009:99:28663,41993,162273 0/0:2392,11:4131:99:29418,42354,168054 0/0:2290,29:3914:99:27134,39635,160715 0/1:3172,592:3996:99:6371,0,97610 0/1:3102,643:3981:99:6452,0,96198 0/1:3663,745:4657:99:9848,0,112759 0/0:2478,19:4266:99:31309,45394,180322 0/0:2861,22:4655:99:41937,55879,183037 0/1:3215,747:4024:99:7710,0,96181 0/1:3617,692:4501:99:8171,0,111081 0/0:2665,14:4387:99:33419,47761,185126 0/0:2738,14:4214:99:26411,39682,171325 0/1:3721,658:4584:99:6100,0,116212 0/0:994,270:4493:99:73160,78446,167944 0/1:3315,678:4225:99:12773,0,102317 0/0:2418,24:3747:99:20563,32521,150390 0/1:3433,964:4497:99:10105,0,105488 0/1:4007,801:5040:99:11141,0,123936 0/1:3579,749:4397:99:14217,0,110155 0/1:3770,664:4668:99:7692,0,116914 0/1:3248,715:4193:99:9311,0,99379 0/1:3720,673:4649:99:7015,0,116963 0/0:2495,14:4261:99:32592,46732,177820 0/0:2537,21:3847:99:19491,31694,154068 0/0:2825,14:4580:99:31482,45455,181048 0/0:2736,7:4413:99:28855,42506,173843 0/0:2390,27:3655:99:20142,31479,145904 0/1:3272,712:4267:99:9820,0,100474 0/0:2431,376:4438:99:29969,32559,145313 0/1:3273,677:4206:99:7206,0,103088 0/1:3372,846:4278:99:7509,0,104604 0/0:2342,10:3971:99:27683,40998,170457 0/0:2450,30:3878:99:23728,36405,158119 0/1:3357,678:4296:99:8400,0,103300 0/0:888,222:4118:99:70115,75948,154416 0/1:3811,724:4776:99:9075,0,117879 0/1:3113,668:3933:99:7827,0,95746 0/0:2431,20:3892:99:26489,38471,151338 0/1:3253,958:4292:99:10880,0,99609 0/0:2398,17:3741:99:23960,35767,150595 0/1:3198,586:3988:99:5366,0,99832 0/0:2362,23:3864:99:24395,37254,162103 0/1:3374,656:4237:99:8306,0,103675 0/0:2240,28:3450:99:19978,30719,138358 0/1:1345,293:1691:99:3627,0,40967 0/1:3086,599:3875:99:5511,0,96076 0/0:2635,7:4091:99:22607,35524,162545 0/1:4009,757:4985:99:10270,0,124761 0/1:3784,756:4768:99:10229,0,116866 0/1:3277,873:4176:99:9061,0,100183 0/1:3807,630:4689:99:7173,0,118453 0/0:2335,15:3992:99:27403,40465,167193 0/1:3390,856:4311:99:8169,0,103941 0/0:2445,12:3935:99:22041,35466,170339 0/0:1119,8:4078:99:65567,78139,174332 0/1:3287,647:4140:99:7742,0,101823 0/1:2843,573:3610:99:5326,0,88405 0/1:3490,689:4419:99:8493,0,107601 0/1:3432,667:4280:99:6353,0,108272 0/0:1863,444:3739:99:27821,26529,111081 0/0:2118,5:3467:99:22005,32304,134506 0/0:2330,50:4007:99:28493,41586,168463 0/0:2295,57:3918:99:25656,38509,161566 0/1:3608,718:4563:99:8159,0,112574 0/1:2978,628:3807:99:5783,0,92892 0/0:2419,11:4153:99:30514,43862,171493 0/0:1998,437:3821:99:23320,21967,112804 0/0:2377,347:4297:99:29339,32542,143971 0/1:3485,822:4369:99:7112,0,107318 0/0:2870,7:4509:99:29413,44002,187492 0/1:3327,615:4115:99:6872,0,102733 0/0:2451,11:4126:99:29186,42606,171206 0/1:3717,848:4627:99:7143,0,114629 0/1:3126,684:3978:99:8256,0,96113 0/0:2773,19:4482:99:28748,43073,181476 0/0:2472,11:3988:99:25643,37989,157093 0/0:2489,5:4209:99:31711,45219,174509 0/1:3352,684:4113:99:8534,0,104384 0/1:3459,643:4303:99:7017,0,106495 0/0:2515,14:4068:99:25605,38681,166676 0/0:2710,8:4482:99:29790,42984,170449 0/0:2574,4:4314:99:30709,43528,165293

chr1 9710632 rs34885574;rs112806445 CAG CAGAG 1.71247e+06 . AC=43;AF=0.236;AN=182;AS_BaseQRankSum=0.5;AS_FS=0;AS_InbreedingCoeff=-0.3094;AS_MQ=58.43;AS_MQRankSum=-1.9;AS_QD=3.97;AS_ReadPosRankSum=2.1;AS_SOR=0.709;BaseQRankSum=0.353;DB;DP=518539;ExcessHet=160;FS=0;InbreedingCoeff=-1;MLEAC=43;MLEAF=0.236;MQ=59.96;MQRankSum=-0.851;QD=4.65;ReadPosRankSum=1.35;SOR=0.689 GT:AD:DP:GQ:PL 0/1:2724,1402:4247:99:26289,0,76238 0/0:3021,16:3870:99:8012,19265,151664 0/0:3631,11:4560:99:9398,22954,182743 0/0:3646,19:4546:99:7718,21352,182189 0/0:3776,11:4716:99:8932,22651,186669 0/1:869,2199:3232:99:47702,0,22286 0/0:3197,28:4023:99:6745,18938,164930 0/0:3492,24:4416:99:7924,20786,177881 0/0:3848,15:4614:99:11206,25214,191060 0/1:2375,1496:4009:99:28663,0,68824 0/1:2392,1561:4131:99:29418,0,68647 0/1:2290,1457:3914:99:27134,0,65441 0/0:3172,9:3996:99:6371,18125,158283 0/0:3102,38:3981:99:6452,18316,158868 0/0:3663,42:4657:99:9848,22973,179311 0/1:2478,1627:4266:99:31309,0,71840 0/1:2861,1708:4655:99:41937,0,79078 0/0:3215,21:4024:99:7710,20061,162265 0/0:3617,9:4501:99:8171,21248,178683 0/1:2665,1681:4387:99:33419,0,69686 0/1:2738,1416:4214:99:26411,0,70441 0/0:3721,12:4584:99:6100,19684,181143 0/1:994,3187:4493:99:73160,0,22460 0/0:3315,23:4225:99:12773,25008,170620 0/1:2418,1241:3747:99:20563,0,64161 0/0:3433,39:4497:99:10105,22963,177053 0/0:4007,25:5040:99:11141,25851,200230 0/0:3579,17:4397:99:14217,27400,182345 0/0:3770,28:4668:99:7692,21047,182778 0/0:3248,45:4193:99:9311,21680,167989 0/0:3720,16:4649:99:7015,21209,186164 0/1:2495,1628:4261:99:32592,0,72312 0/1:2537,1211:3847:99:19491,0,67205 0/1:2825,1591:4580:99:31482,0,74278 0/1:2736,1541:4413:99:28855,0,71899 0/1:2390,1175:3655:99:20142,0,67407 0/0:3272,58:4267:99:9820,22134,167302 0/1:2431,1581:4438:99:29969,0,78540 0/0:3273,47:4206:99:7206,19801,145415 0/0:3372,9:4278:99:7509,20288,172859 0/1:2342,1467:3971:99:27683,0,68971 0/1:2450,1298:3878:99:23728,0,72373 0/0:3357,24:4296:99:8400,20348,151999 0/1:888,2972:4118:99:70115,0,19429 0/0:3811,20:4776:99:9075,22799,187792 0/0:3113,18:3933:99:7827,19354,155554 0/1:2431,1374:3892:99:26489,0,64149 0/0:3253,17:4292:99:10880,23147,169594 0/1:2398,1283:3741:99:23960,0,66034 0/0:3198,34:3988:99:5366,17428,159308 0/1:2362,1356:3864:99:24395,0,68651 0/0:3374,10:4237:99:8306,20829,172312 0/1:2240,1134:3450:99:19978,0,62602 0/0:1345,26:1691:99:3627,8851,71381 0/0:3086,16:3875:99:5511,16990,151647 0/1:2635,1365:4091:99:22607,0,69674 0/0:4009,12:4985:99:10270,24799,198433 0/0:3784,15:4768:99:10229,24084,192784 0/0:3277,8:4176:99:9061,21603,171108 0/0:3807,12:4689:99:7173,20074,167607 0/1:2335,1498:3992:99:27403,0,66248 0/0:3390,24:4311:99:8169,20889,173322 0/1:2445,1353:3935:99:22041,0,71205 0/1:1119,2893:4078:99:65567,0,20842 0/0:3287,25:4140:99:7742,19411,160074 0/0:2843,8:3610:99:5326,16268,142942 0/0:3490,34:4419:99:8493,21478,175751 0/0:3432,4:4280:99:6353,19504,171612 0/1:1863,1400:3739:99:27821,0,61099 0/1:2118,1231:3467:99:22005,0,63118 0/1:2330,1504:4007:99:28493,0,65655 0/1:2295,1417:3918:99:25656,0,67734 0/0:3608,15:4563:99:8159,21824,186025 0/0:2978,5:3807:99:5783,17620,155500 0/1:2419,1588:4153:99:30514,0,68779 0/1:1998,1338:3821:99:23320,0,63582 0/1:2377,1545:4297:99:29339,0,75904 0/0:3485,16:4369:99:7112,19916,175981 0/1:2870,1560:4509:99:29413,0,79246 0/0:3327,5:4115:99:6872,19218,169662 0/1:2451,1531:4126:99:29186,0,70317 0/0:3717,14:4627:99:7143,20815,186214 0/0:3126,5:3978:99:8256,19907,157883 0/1:2773,1540:4482:99:28748,0,73876 0/1:2472,1376:3988:99:25643,0,65183 0/1:2489,1594:4209:99:31711,0,75087 0/0:3352,16:4113:99:8534,21175,172378 0/0:3459,10:4303:99:7017,19776,174430 0/1:2515,1409:4068:99:25605,0,75712 0/1:2710,1603:4482:99:29790,0,69947 0/1:2574,1588:4314:99:30709,0,68199

chr1 9710656 rs72633865 C G 587048 . AC=13;AF=0.071;AN=182;AS_BaseQRankSum=-1.9;AS_FS=0;AS_InbreedingCoeff=-0.0769;AS_MQ=56.67;AS_MQRankSum=-3.9;AS_QD=10.52;AS_ReadPosRankSum=7.1;AS_SOR=0.634;BaseQRankSum=-2.679;DB;DP=301412;ExcessHet=5.0106;FS=0.54;InbreedingCoeff=-0.0769;MLEAC=13;MLEAF=0.071;MQ=59.77;MQRankSum=-4.144;QD=10.52;ReadPosRankSum=3.14;SOR=0.623 GT:AD:DP:GQ:PL 0/1:2441,2008:4481:99:50670,0,71148 0/0:2399,0:2399:99:0,120,1800 0/0:3246,0:3246:99:0,120,1800 0/0:3190,0:3190:99:0,120,1800 0/0:3418,0:3418:99:0,120,1800 0/1:1922,1555:3516:99:33793,0,48428 0/0:2644,0:2644:99:0,120,1800 0/0:3064,0:3064:99:0,120,1800 0/0:3556,0:3556:99:0,120,1800 0/0:3138,0:3138:99:0,120,1800 0/0:3267,0:3267:99:0,120,1800 0/0:3019,0:3019:99:0,120,1800 0/0:2691,0:2691:99:0,120,1800 0/0:2517,0:2517:99:0,120,1800 0/0:3371,0:3371:99:0,120,1800 0/0:4928,0:4928:99:0,120,1800 0/1:2672,2318:5009:99:58293,0,78537 0/0:2645,0:2645:99:0,120,1800 0/0:3129,0:3129:99:0,120,1800 0/0:3416,0:3416:99:0,120,1800 0/0:3165,0:3165:99:0,120,1800 0/0:3238,0:3238:99:0,120,1800 0/0:3631,0:3631:99:0,120,1800 0/0:3043,0:3043:99:0,120,1800 0/0:2533,0:2533:99:0,120,1800 0/0:3048,0:3048:99:0,120,1800 0/0:4013,0:4013:99:0,120,1800 0/0:4654,0:4654:99:0,120,1800 0/0:3451,0:3451:99:0,120,1800 0/0:2744,0:2744:99:0,120,1800 0/0:3585,0:3585:99:0,120,1800 0/0:3406,0:3406:99:0,120,1800 0/0:2704,0:2704:99:0,120,1800 0/0:3680,0:3680:99:0,120,1800 0/0:3334,0:3334:99:0,120,1800 0/1:2094,1670:3786:99:36474,0,57719 0/0:3084,0:3084:99:0,120,1800 0/1:2631,2223:4877:99:54473,0,77389 0/0:2788,0:2788:99:0,120,1800 0/0:2943,0:2943:99:0,120,1800 0/0:3050,0:3050:99:0,120,1800 0/1:2376,1859:4256:99:42820,0,66795 0/0:3037,0:3037:99:0,120,1800 0/0:2882,0:2882:99:0,120,1800 0/0:3561,0:3561:99:0,120,1800 0/0:2391,0:2391:99:0,120,1800 0/0:2584,0:2584:99:0,120,1800 0/0:2756,0:2756:99:0,120,1800 0/1:2178,1758:3952:99:39142,0,59479 0/0:2684,0:2684:99:0,120,1800 0/0:2935,0:2935:99:0,120,1800 0/0:3003,0:3003:99:0,120,1800 0/1:2035,1513:3563:99:34302,0,55281 0/0:1054,0:1054:99:0,120,1800 0/0:2478,0:2478:99:0,120,1800 0/0:3134,0:3134:99:0,120,1800 0/0:3972,0:3972:99:0,120,1800 0/0:3532,0:3532:99:0,120,1800 0/0:4123,0:4123:99:0,120,1800 0/0:3723,0:3723:99:0,120,1800 0/0:2984,0:2984:99:0,120,1800 0/0:2935,0:2935:99:0,120,1800 0/0:3113,0:3113:99:0,120,1800 0/0:2782,0:2782:99:0,120,1800 0/0:2739,0:2739:99:0,120,1800 0/0:2199,0:2199:99:0,120,1800 0/0:3128,0:3128:99:0,120,1800 0/0:2860,0:2860:99:0,120,1800 0/0:2486,0:2486:99:0,120,1800 0/1:2088,1607:3697:99:36067,0,57959 0/0:3179,0:3179:99:0,120,1800 0/0:2998,0:2998:99:0,120,1800 0/0:3322,0:3322:99:0,120,1800 0/0:2435,0:2435:99:0,120,1800 0/0:3302,0:3302:99:0,120,1800 0/0:2735,0:2735:99:0,120,1800 0/1:2529,2168:4713:99:52340,0,72426 0/0:2806,0:2806:99:0,120,1800 0/1:2727,2261:4995:99:53000,0,79053 0/0:2758,0:2758:99:0,120,1800 0/0:3355,0:3355:99:0,120,1800 0/0:3327,0:3327:99:0,120,1800 0/0:2590,0:2590:99:0,120,1800 0/0:3389,0:3389:99:0,120,1800 0/0:2710,0:2710:99:0,120,1800 0/1:2515,2108:4631:99:51500,0,70863 0/0:4361,0:4361:99:0,120,1800 0/0:2848,0:2848:99:0,120,1800 0/1:2588,1953:4565:99:44283,0,74602 0/0:3370,0:3370:99:0,120,1800 0/0:3052,0:3052:99:0,120,1800

chr1 9710662 rs375534148 TAGAC T 215340 . AC=2;AF=0.011;AN=182;AS_BaseQRankSum=1.8;AS_FS=0;AS_InbreedingCoeff=-0.0111;AS_MQ=59.09;AS_MQRankSum=0.2;AS_QD=8.97;AS_ReadPosRankSum=4.3;AS_SOR=0.716;BaseQRankSum=2.45;DB;DP=279436;ExcessHet=3.1563;FS=0;InbreedingCoeff=-0.0225;MLEAC=2;MLEAF=0.011;MQ=59.9;MQRankSum=0.316;QD=14.44;ReadPosRankSum=2.97;SOR=0.715 GT:AD:DP:GQ:PL 0/0:3124,0:3124:99:0,120,1800 0/0:2399,0:2399:99:0,120,1800 0/0:3246,0:3246:99:0,120,1800 0/0:3190,0:3190:99:0,120,1800 0/0:3418,0:3418:99:0,120,1800 0/0:2139,0:2139:99:0,120,1800 0/0:2644,0:2644:99:0,120,1800 0/0:3064,0:3064:99:0,120,1800 0/0:3556,0:3556:99:0,120,1800 0/0:3138,0:3138:99:0,120,1800 0/0:3267,0:3267:99:0,120,1800 0/0:3019,0:3019:99:0,120,1800 0/0:2691,0:2691:99:0,120,1800 0/0:2517,0:2517:99:0,120,1800 0/0:3371,0:3371:99:0,120,1800 0/0:2047,13:4487:99:68870,80116,187231 0/0:3572,0:3572:99:0,120,1800 0/0:2645,0:2645:99:0,120,1800 0/0:3129,0:3129:99:0,120,1800 0/0:3416,0:3416:99:0,120,1800 0/0:3165,0:3165:99:0,120,1800 0/0:3238,0:3238:99:0,120,1800 0/0:3631,0:3631:99:0,120,1800 0/0:3043,0:3043:99:0,120,1800 0/0:2533,0:2533:99:0,120,1800 0/0:3048,0:3048:99:0,120,1800 0/0:4013,0:4013:99:0,120,1800 0/1:1801,1781:3608:99:53214,0,69404 0/0:3451,0:3451:99:0,120,1800 0/0:2744,0:2744:99:0,120,1800 0/0:3585,0:3585:99:0,120,1800 0/0:3406,0:3406:99:0,120,1800 0/0:2704,0:2704:99:0,120,1800 0/0:3680,0:3680:99:0,120,1800 0/0:3334,0:3334:99:0,120,1800 0/0:2481,0:2481:99:0,120,1800 0/0:3084,0:3084:99:0,120,1800 0/0:3486,0:3486:99:0,120,1800 0/0:2788,0:2788:99:0,120,1800 0/0:2943,0:2943:99:0,120,1800 0/0:3050,0:3050:99:0,120,1800 0/0:2788,0:2788:99:0,120,1800 0/0:3037,0:3037:99:0,120,1800 0/0:2882,0:2882:99:0,120,1800 0/0:3561,0:3561:99:0,120,1800 0/0:2391,0:2391:99:0,120,1800 0/0:2584,0:2584:99:0,120,1800 0/0:2756,0:2756:99:0,120,1800 0/0:2398,0:2398:99:0,120,1800 0/0:2684,0:2684:99:0,120,1800 0/0:2935,0:2935:99:0,120,1800 0/0:3003,0:3003:99:0,120,1800 0/0:2201,0:2201:99:0,120,1800 0/0:1054,0:1054:99:0,120,1800 0/0:2478,0:2478:99:0,120,1800 0/0:3134,0:3134:99:0,120,1800 0/0:3972,0:3972:99:0,120,1800 0/0:3532,0:3532:99:0,120,1800 0/0:1538,46:3647:99:53048,60405,142690 0/0:3723,0:3723:99:0,120,1800 0/0:2984,0:2984:99:0,120,1800 0/0:2935,0:2935:99:0,120,1800 0/0:3113,0:3113:99:0,120,1800 0/0:2782,0:2782:99:0,120,1800 0/0:2739,0:2739:99:0,120,1800 0/0:2199,0:2199:99:0,120,1800 0/0:3128,0:3128:99:0,120,1800 0/0:2860,0:2860:99:0,120,1800 0/0:2486,0:2486:99:0,120,1800 0/0:2183,0:2183:99:0,120,1800 0/0:3179,0:3179:99:0,120,1800 0/0:2998,0:2998:99:0,120,1800 0/0:3322,0:3322:99:0,120,1800 0/0:2435,0:2435:99:0,120,1800 0/0:3302,0:3302:99:0,120,1800 0/0:2735,0:2735:99:0,120,1800 0/0:3212,0:3212:99:0,120,1800 0/0:2806,0:2806:99:0,120,1800 0/0:3614,0:3614:99:0,120,1800 0/0:2758,0:2758:99:0,120,1800 0/0:3355,0:3355:99:0,120,1800 0/0:3327,0:3327:99:0,120,1800 0/0:2590,0:2590:99:0,120,1800 0/0:3389,0:3389:99:0,120,1800 0/0:2710,0:2710:99:0,120,1800 0/0:3132,0:3132:99:0,120,1800 0/1:1714,1474:3211:99:40275,0,66501 0/0:2848,0:2848:99:0,120,1800 0/0:3130,0:3130:99:0,120,1800 0/0:3370,0:3370:99:0,120,1800 0/0:3052,0:3052:99:0,120,1800

chr1 9710662 rs375534148 TAGAC CAGAC 215340 . AC=2;AF=0.011;AN=182;AS_BaseQRankSum=3.2;AS_FS=0;AS_InbreedingCoeff=-0.0111;AS_MQ=53.19;AS_MQRankSum=-1.3;AS_QD=10.51;AS_ReadPosRankSum=1.2;AS_SOR=0.726;BaseQRankSum=2.45;DB;DP=279436;ExcessHet=3.1563;FS=0;InbreedingCoeff=-0.0225;MLEAC=2;MLEAF=0.011;MQ=59.9;MQRankSum=0.316;QD=14.44;ReadPosRankSum=2.97;SOR=0.715 GT:AD:DP:GQ:PL 0/0:3124,0:3124:99:0,120,1800 0/0:2399,0:2399:99:0,120,1800 0/0:3246,0:3246:99:0,120,1800 0/0:3190,0:3190:99:0,120,1800 0/0:3418,0:3418:99:0,120,1800 0/0:2139,0:2139:99:0,120,1800 0/0:2644,0:2644:99:0,120,1800 0/0:3064,0:3064:99:0,120,1800 0/0:3556,0:3556:99:0,120,1800 0/0:3138,0:3138:99:0,120,1800 0/0:3267,0:3267:99:0,120,1800 0/0:3019,0:3019:99:0,120,1800 0/0:2691,0:2691:99:0,120,1800 0/0:2517,0:2517:99:0,120,1800 0/0:3371,0:3371:99:0,120,1800 0/1:2047,2422:4487:99:68870,0,65355 0/0:3572,0:3572:99:0,120,1800 0/0:2645,0:2645:99:0,120,1800 0/0:3129,0:3129:99:0,120,1800 0/0:3416,0:3416:99:0,120,1800 0/0:3165,0:3165:99:0,120,1800 0/0:3238,0:3238:99:0,120,1800 0/0:3631,0:3631:99:0,120,1800 0/0:3043,0:3043:99:0,120,1800 0/0:2533,0:2533:99:0,120,1800 0/0:3048,0:3048:99:0,120,1800 0/0:4013,0:4013:99:0,120,1800 0/0:1801,11:3608:99:53214,61849,160030 0/0:3451,0:3451:99:0,120,1800 0/0:2744,0:2744:99:0,120,1800 0/0:3585,0:3585:99:0,120,1800 0/0:3406,0:3406:99:0,120,1800 0/0:2704,0:2704:99:0,120,1800 0/0:3680,0:3680:99:0,120,1800 0/0:3334,0:3334:99:0,120,1800 0/0:2481,0:2481:99:0,120,1800 0/0:3084,0:3084:99:0,120,1800 0/0:3486,0:3486:99:0,120,1800 0/0:2788,0:2788:99:0,120,1800 0/0:2943,0:2943:99:0,120,1800 0/0:3050,0:3050:99:0,120,1800 0/0:2788,0:2788:99:0,120,1800 0/0:3037,0:3037:99:0,120,1800 0/0:2882,0:2882:99:0,120,1800 0/0:3561,0:3561:99:0,120,1800 0/0:2391,0:2391:99:0,120,1800 0/0:2584,0:2584:99:0,120,1800 0/0:2756,0:2756:99:0,120,1800 0/0:2398,0:2398:99:0,120,1800 0/0:2684,0:2684:99:0,120,1800 0/0:2935,0:2935:99:0,120,1800 0/0:3003,0:3003:99:0,120,1800 0/0:2201,0:2201:99:0,120,1800 0/0:1054,0:1054:99:0,120,1800 0/0:2478,0:2478:99:0,120,1800 0/0:3134,0:3134:99:0,120,1800 0/0:3972,0:3972:99:0,120,1800 0/0:3532,0:3532:99:0,120,1800 0/1:1538,2059:3647:99:53048,0,48442 0/0:3723,0:3723:99:0,120,1800 0/0:2984,0:2984:99:0,120,1800 0/0:2935,0:2935:99:0,120,1800 0/0:3113,0:3113:99:0,120,1800 0/0:2782,0:2782:99:0,120,1800 0/0:2739,0:2739:99:0,120,1800 0/0:2199,0:2199:99:0,120,1800 0/0:3128,0:3128:99:0,120,1800 0/0:2860,0:2860:99:0,120,1800 0/0:2486,0:2486:99:0,120,1800 0/0:2183,0:2183:99:0,120,1800 0/0:3179,0:3179:99:0,120,1800 0/0:2998,0:2998:99:0,120,1800 0/0:3322,0:3322:99:0,120,1800 0/0:2435,0:2435:99:0,120,1800 0/0:3302,0:3302:99:0,120,1800 0/0:2735,0:2735:99:0,120,1800 0/0:3212,0:3212:99:0,120,1800 0/0:2806,0:2806:99:0,120,1800 0/0:3614,0:3614:99:0,120,1800 0/0:2758,0:2758:99:0,120,1800 0/0:3355,0:3355:99:0,120,1800 0/0:3327,0:3327:99:0,120,1800 0/0:2590,0:2590:99:0,120,1800 0/0:3389,0:3389:99:0,120,1800 0/0:2710,0:2710:99:0,120,1800 0/0:3132,0:3132:99:0,120,1800 0/0:1714,10:3211:99:40275,51462,175410 0/0:2848,0:2848:99:0,120,1800 0/0:3130,0:3130:99:0,120,1800 0/0:3370,0:3370:99:0,120,1800 0/0:3052,0:3052:99:0,120,1800

chr1 9710693 . G A 51020.2 . AC=1;AF=0.005495;AN=182;AS_BaseQRankSum=-0.6;AS_FS=0;AS_InbreedingCoeff=-0.0055;AS_MQ=59.82;AS_MQRankSum=-1.1;AS_QD=18.87;AS_ReadPosRankSum=-0.9;AS_SOR=0.664;BaseQRankSum=-0.446;DP=273734;ExcessHet=3.0103;FS=0;InbreedingCoeff=-0.0055;MLEAC=1;MLEAF=0.005495;MQ=59.98;MQRankSum=-1.03;QD=18.87;ReadPosRankSum=-0.948;SOR=0.66 GT:AD:DP:GQ:PL 0/0:3124,0:3124:99:0,120,1800 0/0:2399,0:2399:99:0,120,1800 0/0:3246,0:3246:99:0,120,1800 0/0:3190,0:3190:99:0,120,1800 0/0:3418,0:3418:99:0,120,1800 0/0:2139,0:2139:99:0,120,1800 0/0:2644,0:2644:99:0,120,1800 0/0:3064,0:3064:99:0,120,1800 0/0:3556,0:3556:99:0,120,1800 0/0:3138,0:3138:99:0,120,1800 0/0:3267,0:3267:99:0,120,1800 0/0:3019,0:3019:99:0,120,1800 0/0:2691,0:2691:99:0,120,1800 0/0:2517,0:2517:99:0,120,1800 0/0:3371,0:3371:99:0,120,1800 0/0:3650,0:3650:99:0,120,1800 0/0:3572,0:3572:99:0,120,1800 0/0:2645,0:2645:99:0,120,1800 0/0:3129,0:3129:99:0,120,1800 0/0:3416,0:3416:99:0,120,1800 0/0:3165,0:3165:99:0,120,1800 0/0:3238,0:3238:99:0,120,1800 0/0:3631,0:3631:99:0,120,1800 0/0:3043,0:3043:99:0,120,1800 0/0:2533,0:2533:99:0,120,1800 0/0:3048,0:3048:99:0,120,1800 0/0:4013,0:4013:99:0,120,1800 0/0:3447,0:3447:99:0,120,1800 0/0:3451,0:3451:99:0,120,1800 0/0:2744,0:2744:99:0,120,1800 0/0:3585,0:3585:99:0,120,1800 0/0:3406,0:3406:99:0,120,1800 0/0:2704,0:2704:99:0,120,1800 0/0:3680,0:3680:99:0,120,1800 0/0:3334,0:3334:99:0,120,1800 0/0:2481,0:2481:99:0,120,1800 0/0:3084,0:3084:99:0,120,1800 0/0:3486,0:3486:99:0,120,1800 0/0:2788,0:2788:99:0,120,1800 0/0:2943,0:2943:99:0,120,1800 0/0:3050,0:3050:99:0,120,1800 0/0:2788,0:2788:99:0,120,1800 0/0:3037,0:3037:99:0,120,1800 0/0:2882,0:2882:99:0,120,1800 0/0:3561,0:3561:99:0,120,1800 0/0:2391,0:2391:99:0,120,1800 0/0:2584,0:2584:99:0,120,1800 0/0:2756,0:2756:99:0,120,1800 0/0:2398,0:2398:99:0,120,1800 0/0:2684,0:2684:99:0,120,1800 0/0:2935,0:2935:99:0,120,1800 0/0:3003,0:3003:99:0,120,1800 0/0:2201,0:2201:99:0,120,1800 0/0:1054,0:1054:99:0,120,1800 0/0:2478,0:2478:99:0,120,1800 0/0:3134,0:3134:99:0,120,1800 0/0:3972,0:3972:99:0,120,1800 0/0:3532,0:3532:99:0,120,1800 0/0:2751,0:2751:99:0,120,1800 0/0:3723,0:3723:99:0,120,1800 0/0:2984,0:2984:99:0,120,1800 0/0:2935,0:2935:99:0,120,1800 0/0:3113,0:3113:99:0,120,1800 0/0:2782,0:2782:99:0,120,1800 0/0:2739,0:2739:99:0,120,1800 0/0:2199,0:2199:99:0,120,1800 0/0:3128,0:3128:99:0,120,1800 0/0:2860,0:2860:99:0,120,1800 0/0:2486,0:2486:99:0,120,1800 0/0:2183,0:2183:99:0,120,1800 0/0:3179,0:3179:99:0,120,1800 0/0:2998,0:2998:99:0,120,1800 0/0:3322,0:3322:99:0,120,1800 0/0:2435,0:2435:99:0,120,1800 0/0:3302,0:3302:99:0,120,1800 0/1:1373,1331:2711:99:51040,0,53022 0/0:3212,0:3212:99:0,120,1800 0/0:2806,0:2806:99:0,120,1800 0/0:3614,0:3614:99:0,120,1800 0/0:2758,0:2758:99:0,120,1800 0/0:3355,0:3355:99:0,120,1800 0/0:3327,0:3327:99:0,120,1800 0/0:2590,0:2590:99:0,120,1800 0/0:3389,0:3389:99:0,120,1800 0/0:2710,0:2710:99:0,120,1800 0/0:3132,0:3132:99:0,120,1800 0/0:3066,0:3066:99:0,120,1800 0/0:2848,0:2848:99:0,120,1800 0/0:3130,0:3130:99:0,120,1800 0/0:3370,0:3370:99:0,120,1800 0/0:3052,0:3052:99:0,120,1800`

[hanars]

Hi Simon,

We have never worked with this sort of data before so I can't really offer you any help. If I were you, before running this sort of data in the future I would reach out to the hail team to see if they have any advice for how you should best set up a cluster for handling this type of data (I'm not really sure what you mean by "very powerful cluster" but its quite possible that the power you are giving it is not the power it needs). I have to assume that small number of high read depth samples do not necessarily have the same computing needs as large numbers of lower read depth samples. The first step you are reporting issues with is a hail-intensive (and VEP intensive) step, and the runtime is heavily impacted by the type of data being loaded and the cluster it is run on. The second step (which you expect to be slow) is an export step that is rate limited by elasticsearch and has a run time proportional to the number of variants and samples regardless of cluster configuration.

In short, it makes sense to me that this different data had different processing needs for the first step but we are not the team who can help you optimize that. I'm also not really sure what you need at this point - it sounds like the first step succeeded, and we regularly take over a day to do the export step so you are still within what we consider to be normal performance there.

Let me know if there is anything I can do that I am missing here,
Hana

[SimonLi5601]

Hi Hana, thanks for you reply. I was able to narrow down to a combination of two variants that can replicate the slowness issues(or not never finish), the strange thing is that I can load individual variants separately without problem. I can't tell it is the issue in Hail or in the pipeline. I did submit an issue to Hail team before I submitted a question here, but didn't get very helpful response which I understood it is hard for them to know what's happen without the context. I don't know why those two variants combination is special. I used binary search method to narrow down to these two. I will try other combination with those two variants, but if you have any idea that what might need to calculate between two variants, please advice, thanks!
IUIS_chr3.vcf.txt

[SimonLi5601]

I can confirm this variant "chr3-128481229-C-T is causing our issue when it is loaded along with other variants.

[hanars]

Again, I'm sorry but as we are completely unfamiliar with this type of data and have never tried loading it I don't have any insight and can't really help debug this. The pipeline does no calculations "between" variants so theres no real reason why a particular combination would be slow. Maybe they have extra info fields, or extra amounts of read depth, and just each take a long time to process? If the hail team can't give you any insight into why a particular variant representation might be extra slow in hail, the next best thing I could offer you would be if you could actually find the point in the pipeline logs where the bottleneck is happening and send it to me maybe I could take a look and see if anything jumps out at me

I should also reiterate that we often run pipelines for upwards of 24 hours, so I also don't see it as a blocking issue to just run the loading pipeline for 8 hours