error in loading command in the pipeline-runner container: "(gcloud.dataproc.clusters.create) INVALID_ARGUMENT: Multiple validation errors"

[GeorginaAmbriz]

Hello,
I have a local installation of seqr.
I'm in step 5 of the Annotating and loading VCF callsets step. I've managed to successfully run steps 1-4 and I uploaded a call set and reference data in a google bucket.
In step 5 I ran:

GS_BUCKET=gs://exome-1 
BUILD_VERSION=37                 # can be 37 or 38
SAMPLE_TYPE=WES                  # can be WES or WGS
INDEX_NAME=trio1     # the desired index name to output. Will be used later to link the data to the corresponding seqr project
 
INPUT_FILE_PATH=/data/GRCh37/seqr.trio.vcf.gz
gcloud config set dataproc/region us-east1
docker-compose exec pipeline-runner load_data_dataproc.sh $BUILD_VERSION $SAMPLE_TYPE $INDEX_NAME $GS_BUCKET $INPUT_FILE_PATH

And I got this output:

docker-compose exec pipeline-runner load_data_dataproc.sh $BUILD_VERSION $SAMPLE_TYPE $INDEX_NAME $GS_BUCKET $INPUT_FILE_PATH
+ BUILD_VERSION=37
+ SAMPLE_TYPE=WES
+ INDEX_NAME=trio1
+ GS_BUCKET=gs://exome-1
+ INPUT_FILE_PATH=/data/GRCh37/seqr.trio.vcf.gz
+ case ${BUILD_VERSION} in
+ FULL_BUILD_VERSION=GRCh37
+ SOURCE_FILE=gs://exome-1/data/GRCh37/seqr.trio.vcf.gz
+ DEST_FILE=gs://exome-1/data/GRCh37/seqr.mt
+ REFERENCE_DATA_BUCKET=gs://exome-1/reference_data/GRCh37
+ cd /seqr-loading-pipelines/luigi_pipeline
+ hailctl dataproc start --pkgs luigi,google-api-python-client --zone us-central1-b --vep GRCh37 --max-idle 30m --num-workers 2 --num-preemptible-workers 12 seqr-loading-cluster
Pulling VEP data from bucket in us.
hailctl dataproc: Creating a cluster with workers of machine type n1-highmem-8.
  Allocating 25292 MB of memory per executor (4 cores),
  with at least 15175 MB for Hail off-heap values and 10117 MB for the JVM.  Using a maximum Hail memory reservation of 6323 MB per core.
gcloud dataproc clusters create seqr-loading-cluster \
    --image-version=2.0.44-debian10 \
    --properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=41g|||yarn:yarn.nodemanager.resource.memory-mb=50585|||yarn:yarn.scheduler.maximum-allocation-mb=25292|||spark:spark.executor.cores=4|||spark:spark.executor.memory=10117m|||spark:spark.executor.memoryOverhead=15175m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=6323 \
    --initialization-actions=gs://hail-common/hailctl/dataproc/0.2.105/init_notebook.py,gs://hail-common/hailctl/dataproc/0.2.105/vep-GRCh37.sh \
    --metadata=^|||^VEP_REPLICATE=us|||VEP_CONFIG_PATH=/vep_data/vep-gcloud.json|||VEP_CONFIG_URI=file:///vep_data/vep-gcloud.json|||WHEEL=gs://hail-common/hailctl/dataproc/0.2.105/hail-0.2.105-py3-none-any.whl|||PKGS=aiohttp>=3.8.1,<4|aiohttp_session>=2.7,<2.8|asyncinit>=0.2.4,<0.3|avro>=1.10,<1.12|azure-identity>=1.6.0,<2|azure-storage-blob>=12.11.0,<13|bokeh>1.3,<2.0|boto3>=1.17,<2.0|botocore>=1.20,<2.0|decorator<5|Deprecated>=1.2.10,<1.3|dill>=0.3.1.1,<0.4|google-auth>=1.27.0,<2|frozenlist>=1.3.1,<2|google-cloud-storage==1.25.*|humanize>=1.0.0,<2|hurry.filesize>=0.9,<1|janus>=0.6,<1.1|Jinja2==3.0.3|nest_asyncio>=1.5.4,<2|numpy<2|orjson>=3.6.4,<4|pandas>=1.3.0,<1.5.0|parsimonious<0.9|plotly>=5.5.0,<5.11|protobuf==3.20.2|PyJWT|rich==12.6.0|python-json-logger>=2.0.2,<3|requests>=2.25.1,<3|scipy>1.2,<1.10|sortedcontainers>=2.4.0,<3|tabulate>=0.8.9,<1|uvloop>=0.16.0,<1; sys_platform!='win32'|luigi|google-api-python-client \
    --master-machine-type=n1-highmem-8 \
    --master-boot-disk-size=100GB \
    --num-master-local-ssds=0 \
    --num-secondary-workers=12 \
    --num-worker-local-ssds=0 \
    --num-workers=2 \
    --secondary-worker-boot-disk-size=200GB \
    --worker-boot-disk-size=200GB \
    --worker-machine-type=n1-highmem-8 \
    --zone=us-central1-b \
    --initialization-action-timeout=20m \
    --labels=creator=georgina_jimenezambriz_mcgill_ca \
    --max-idle=30m
Starting cluster 'seqr-loading-cluster'...
ERROR: (gcloud.dataproc.clusters.create) INVALID_ARGUMENT: Multiple validation errors:
 - Insufficient 'CPUS' quota. Requested 120.0, available 24.0.
 - Insufficient 'CPUS_ALL_REGIONS' quota. Requested 120.0, available 32.0.
 - Insufficient 'IN_USE_ADDRESSES' quota. Requested 15.0, available 8.0.
 - Zone 'exome-demo-0001/us-central1-b' resides in unsupported region 'https://www.googleapis.com/compute/v1/projects/exome-demo-0001/regions/us-central1'. Supported regions: [us-east1]
Traceback (most recent call last):
  File "/usr/local/bin/hailctl", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/__main__.py", line 107, in main
    cli.main(args)
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/cli.py", line 124, in main
    jmp[args.module].main(args, pass_through_args))
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/start.py", line 433, in main
    gcloud.run(cmd[1:])
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/gcloud.py", line 9, in run
    return subprocess.check_call(["gcloud"] + command)
  File "/usr/local/lib/python3.7/subprocess.py", line 341, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['gcloud', 'dataproc', 'clusters', 'create', 'seqr-loading-cluster', '--image-version=2.0.44-debian10', '--properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=41g|||yarn:yarn.nodemanager.resource.memory-mb=50585|||yarn:yarn.scheduler.maximum-allocation-mb=25292|||spark:spark.executor.cores=4|||spark:spark.executor.memory=10117m|||spark:spark.executor.memoryOverhead=15175m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=6323', '--initialization-actions=gs://hail-common/hailctl/dataproc/0.2.105/init_notebook.py,gs://hail-common/hailctl/dataproc/0.2.105/vep-GRCh37.sh', "--metadata=^|||^VEP_REPLICATE=us|||VEP_CONFIG_PATH=/vep_data/vep-gcloud.json|||VEP_CONFIG_URI=file:///vep_data/vep-gcloud.json|||WHEEL=gs://hail-common/hailctl/dataproc/0.2.105/hail-0.2.105-py3-none-any.whl|||PKGS=aiohttp>=3.8.1,<4|aiohttp_session>=2.7,<2.8|asyncinit>=0.2.4,<0.3|avro>=1.10,<1.12|azure-identity>=1.6.0,<2|azure-storage-blob>=12.11.0,<13|bokeh>1.3,<2.0|boto3>=1.17,<2.0|botocore>=1.20,<2.0|decorator<5|Deprecated>=1.2.10,<1.3|dill>=0.3.1.1,<0.4|google-auth>=1.27.0,<2|frozenlist>=1.3.1,<2|google-cloud-storage==1.25.*|humanize>=1.0.0,<2|hurry.filesize>=0.9,<1|janus>=0.6,<1.1|Jinja2==3.0.3|nest_asyncio>=1.5.4,<2|numpy<2|orjson>=3.6.4,<4|pandas>=1.3.0,<1.5.0|parsimonious<0.9|plotly>=5.5.0,<5.11|protobuf==3.20.2|PyJWT|rich==12.6.0|python-json-logger>=2.0.2,<3|requests>=2.25.1,<3|scipy>1.2,<1.10|sortedcontainers>=2.4.0,<3|tabulate>=0.8.9,<1|uvloop>=0.16.0,<1; sys_platform!='win32'|luigi|google-api-python-client", '--master-machine-type=n1-highmem-8', '--master-boot-disk-size=100GB', '--num-master-local-ssds=0', '--num-secondary-workers=12', '--num-worker-local-ssds=0', '--num-workers=2', '--secondary-worker-boot-disk-size=200GB', '--worker-boot-disk-size=200GB', '--worker-machine-type=n1-highmem-8', '--zone=us-central1-b', '--initialization-action-timeout=20m', '--labels=creator=georgina_jimenezambriz_mcgill_ca', '--max-idle=30m']' returned non-zero exit status 1.

Is this related to my gcloud configuration?

I'll really appreciate your help in any solution you can provide.

Thank you,
Georgina Jimenez A.
Bioinformatics core
Montreal Neurological Institute-Hospital

[hanars]

The core part of the issue seems to be that you are requesting more resources than you are allowed:

 - Insufficient 'CPUS' quota. Requested 120.0, available 24.0.
 - Insufficient 'CPUS_ALL_REGIONS' quota. Requested 120.0, available 32.0.
 - Insufficient 'IN_USE_ADDRESSES' quota. Requested 15.0, available 8.0.

This is probably a problem with your GCP/ dataproc configuration. If you can't afford the default resources seqr requests, you can also try manually creating a dataproc cluster with less resources, but that will greatly increase the runtime and efficiency of the loading pipeline

[drmurdock]

Hi @GeorginaAmbriz. Were you able to resolve this issue? I'm having the exact same problem. According to google, it's going to cost >$1000 per month to increase my quota to those specifications. To me that seems very high for what I'm trying to do (annotate WES).

Thanks,
David Murdock