Is it possible to run the pipeline without error out

[yikun-chi]

Hi

I am currently running the pipleline through a very large dataset. Currently the pipeline will error out when running into certain population. I have to remove the PID from configuration file, and restart the pipeline. Is this the recommended way? I am curious is there a way to configure the pipeline to run without erroring out and simply log the PID for which the features generation failed. This way I can generate some rough preliminary data and work on more refined data cleanning and debugging latter.

[JulioV]

It is the recommended way, however, snakemake (the orchestrator we use under the hood) has the -k option that keeps running jobs that are independent to one that fails. You can experiment with that one but we have not try it so we haven't documented it. Let us what you find!