getting an error while spawning a dataproc cluster with hail and jgscm

[xshah]

google-cloud-dataproc-metainfo_744a0b81-c763-482b-a914-81565eae27cc_seqr-loading-cluster-m_dataproc-initialization-script-0_output.txt
google-cloud-dataproc-metainfo_744a0b81-c763-482b-a914-81565eae27cc_seqr-loading-cluster-m_dataproc-initialization-script-1_output.txt
setuptools.extern.packaging.version.InvalidVersion: Invalid version: '0.1.10-hail'
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> jgscm

[hanars]

Can you please add some context about what exactly you are doing to reach this issue?

[xshah]

thanks @hanars - I am getting the same issue.

[hanars]

can you send the full console logs from the command you ran, as well as the dataproc logs from the run on the latest pipeline?

[xshah]

Here @hanars

• BUILD_VERSION=38
• SAMPLE_TYPE=WGS
• INDEX_NAME=636-family
• GS_BUCKET=gs://seqr-v1/
• INPUT_FILE_PATH=636-family-gatk.vcf.gz
• case ${BUILD_VERSION} in
• FULL_BUILD_VERSION=GRCh38
• SOURCE_FILE=gs://seqr-v1/636-family-gatk.vcf.gz
• DEST_FILE=gs://seqr-v1/636-family-gatk.mt
• REFERENCE_DATA_BUCKET=gs://seqr-v1//reference_data/GRCh38
• cd /seqr-loading-pipelines/luigi_pipeline
• hailctl dataproc start --pkgs luigi,google-api-python-client --zone us-central1-b --vep GRCh38 --max-idle 30m --num-workers 2 --num-preemptible-workers 12 --region us-central1 --subnet=sebralab-us-central1-1 seqr-loading-cluster
  Pulling VEP data from bucket in us.
  hailctl dataproc: Creating a cluster with workers of machine type n1-highmem-8.
  Allocating 25292 MB of memory per executor (4 cores),
  with at least 15175 MB for Hail off-heap values and 10117 MB for the JVM. Using a maximum Hail memory reservation of 6323 MB per core.
  gcloud dataproc clusters create seqr-loading-cluster
  --image-version=2.0.44-debian10
  --properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=41g|||yarn:yarn.nodemanager.resource.memory-mb=50585|||yarn:yarn.scheduler.maximum-allocation-mb=25292|||spark:spark.executor.cores=4|||spark:spark.executor.memory=10117m|||spark:spark.executor.memoryOverhead=15175m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=6323
  --initialization-actions=gs://hail-common/hailctl/dataproc/0.2.105/init_notebook.py,gs://hail-common/hailctl/dataproc/0.2.105/vep-GRCh38.sh
  --metadata=^|||^VEP_REPLICATE=us|||VEP_CONFIG_PATH=/vep_data/vep-gcloud.json|||VEP_CONFIG_URI=file:///vep_data/vep-gcloud.json|||WHEEL=gs://hail-common/hailctl/dataproc/0.2.105/hail-0.2.105-py3-none-any.whl|||PKGS=aiohttp>=3.8.1,<4|aiohttp_session>=2.7,<2.8|asyncinit>=0.2.4,<0.3|avro>=1.10,<1.12|azure-identity>=1.6.0,<2|azure-storage-blob>=12.11.0,<13|bokeh>1.3,<2.0|boto3>=1.17,<2.0|botocore>=1.20,<2.0|decorator<5|Deprecated>=1.2.10,<1.3|dill>=0.3.1.1,<0.4|google-auth>=1.27.0,<2|frozenlist>=1.3.1,<2|google-cloud-storage==1.25.|humanize>=1.0.0,<2|hurry.filesize>=0.9,<1|janus>=0.6,<1.1|Jinja2==3.0.3|nest_asyncio>=1.5.4,<2|numpy<2|orjson>=3.6.4,<4|pandas>=1.3.0,<1.5.0|parsimonious<0.9|plotly>=5.5.0,<5.11|protobuf==3.20.2|PyJWT|rich==12.6.0|python-json-logger>=2.0.2,<3|requests>=2.25.1,<3|scipy>1.2,<1.10|sortedcontainers>=2.4.0,<3|tabulate>=0.8.9,<1|uvloop>=0.16.0,<1; sys_platform!='win32'|luigi|google-api-python-client
  --master-machine-type=n1-highmem-8
  --master-boot-disk-size=100GB
  --num-master-local-ssds=0
  --num-secondary-workers=12
  --num-worker-local-ssds=0
  --num-workers=2
  --secondary-worker-boot-disk-size=200GB
  --worker-boot-disk-size=200GB
  --worker-machine-type=n1-highmem-8
  --region=us-central1
  --zone=us-central1-b
  --initialization-action-timeout=20m
  --labels=creator=601858602582-compute_developer_gserviceaccount_com
  --max-idle=30m
  --subnet=sebralab-us-central1-1
  Starting cluster 'seqr-loading-cluster'...
  Waiting on operation [projects/serbalab-techdev/regions/us-central1/operations/04545505-9a02-31a1-ba93-eee1c2912f03].
  Waiting for cluster creation operation...
  WARNING: Consider using Auto Zone rather than selecting a zone manually. See https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/auto-zone
  WARNING: For PD-Standard without local SSDs, we strongly recommend provisioning 1TB or larger to ensure consistently high I/O performance. See https://cloud.google.com/compute/docs/disks/performance for information on disk I/O performance.
  WARNING: This cluster is configured to use network 'https://www.googleapis.com/compute/v1/projects/serbalab-techdev/global/networks/sebralab-vpc-prod' and its associated firewall rules '[default-allow-internal]' which contains the following potential security vulnerability: 'port 8088 is open to the internet, this may allow arbitrary code execution via the YARN REST API. Use Component Gateway for secure remote access to the YARN UI and other cluster UIs instead: https://cloud.google.com/dataproc/docs/concepts/accessing/dataproc-gateways.'
  Waiting for cluster creation operation...done.
  ERROR: (gcloud.dataproc.clusters.create) Operation [projects/serbalab-techdev/regions/us-central1/operations/04545505-9a02-31a1-ba93-eee1c2912f03] failed: Initialization action failed. Failed action 'gs://hail-common/hailctl/dataproc/0.2.105/init_notebook.py', see output in: gs://dataproc-staging-us-central1-601858602582-ha6xxo5f/google-cloud-dataproc-metainfo/2827e0e5-6ce9-4d7e-aac9-97b1260c4085/seqr-loading-cluster-m/dataproc-initialization-script-0_output.
  Traceback (most recent call last):
  File "/usr/local/bin/hailctl", line 8, in
  sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/main.py", line 107, in main
  cli.main(args)
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/cli.py", line 124, in main
  jmp[args.module].main(args, pass_through_args))
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
  return future.result()
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/start.py", line 433, in main
  gcloud.run(cmd[1:])
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/gcloud.py", line 9, in run
  return subprocess.check_call(["gcloud"] + command)
  File "/usr/local/lib/python3.7/subprocess.py", line 341, in check_call
  raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['gcloud', 'dataproc', 'clusters', 'create', 'seqr-loading-cluster', '--image-version=2.0.44-debian10', '--properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=41g|||yarn:yarn.nodemanager.resource.memory-mb=50585|||yarn:yarn.scheduler.maximum-allocation-mb=25292|||spark:spark.executor.cores=4|||spark:spark.executor.memory=10117m|||spark:spark.executor.memoryOverhead=15175m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=6323', '--initialization-actions=gs://hail-common/hailctl/dataproc/0.2.105/init_notebook.py,gs://hail-common/hailctl/dataproc/0.2.105/vep-GRCh38.sh', "--metadata=^|||^VEP_REPLICATE=us|||VEP_CONFIG_PATH=/vep_data/vep-gcloud.json|||VEP_CONFIG_URI=file:///vep_data/vep-gcloud.json|||WHEEL=gs://hail-common/hailctl/dataproc/0.2.105/hail-0.2.105-py3-none-any.whl|||PKGS=aiohttp>=3.8.1,<4|aiohttp_session>=2.7,<2.8|asyncinit>=0.2.4,<0.3|avro>=1.10,<1.12|azure-identity>=1.6.0,<2|azure-storage-blob>=12.11.0,<13|bokeh>1.3,<2.0|boto3>=1.17,<2.0|botocore>=1.20,<2.0|decorator<5|Deprecated>=1.2.10,<1.3|dill>=0.3.1.1,<0.4|google-auth>=1.27.0,<2|frozenlist>=1.3.1,<2|google-cloud-storage==1.25.|humanize>=1.0.0,<2|hurry.filesize>=0.9,<1|janus>=0.6,<1.1|Jinja2==3.0.3|nest_asyncio>=1.5.4,<2|numpy<2|orjson>=3.6.4,<4|pandas>=1.3.0,<1.5.0|parsimonious<0.9|plotly>=5.5.0,<5.11|protobuf==3.20.2|PyJWT|rich==12.6.0|python-json-logger>=2.0.2,<3|requests>=2.25.1,<3|scipy>1.2,<1.10|sortedcontainers>=2.4.0,<3|tabulate>=0.8.9,<1|uvloop>=0.16.0,<1; sys_platform!='win32'|luigi|google-api-python-client", '--master-machine-type=n1-highmem-8', '--master-boot-disk-size=100GB', '--num-master-local-ssds=0', '--num-secondary-workers=12', '--num-worker-local-ssds=0', '--num-workers=2', '--secondary-worker-boot-disk-size=200GB', '--worker-boot-disk-size=200GB', '--worker-machine-type=n1-highmem-8', '--region=us-central1', '--zone=us-central1-b', '--initialization-action-timeout=20m', '--labels=creator=601858602582-compute_developer_gserviceaccount_com', '--max-idle=30m', '--subnet=sebralab-us-central1-1']' returned non-zero exit status 1.

and the dataproc logs are attached
google-cloud-dataproc-metainfo_2827e0e5-6ce9-4d7e-aac9-97b1260c4085_seqr-loading-cluster-m_dataproc-initialization-script-0_output.txt
google-cloud-dataproc-metainfo_2827e0e5-6ce9-4d7e-aac9-97b1260c4085_seqr-loading-cluster-m_dataproc-initialization-script-1_output.txt

[hanars]

It looks like there may be an issue with the version of hail we are packaging - can you try running docker-compose exec pipeline-runner pip install hail --upgrade and then try running the loading again?

[lauragails]

I had the same error and I can confirm that this suggestion fixed the hailctl dataproc start, and successfully created the seqr-loading-cluster.

I am having a different issue now at the next step, which I will make a different ticket for, or post below, in the morning. Thanks again!

edit I do now get a warning in a hail long that:

2023-04-28 18:19:48.961 Hail: WARN: This Hail JAR was compiled for Spark 3.3.0, running with Spark 3.3.2.
  Compatibility is not guaranteed.
2023-04-28 18:19:50.937 Hail: INFO: SparkUI: http://0522db40d9c8:4040
2023-04-28 18:19:51.305 Hail: INFO: Running Hail version 0.2.114-cc8d36408b36
2023-04-28 18:24:56.708 Hail: WARN: This Hail JAR was compiled for Spark 3.3.0, running with Spark 3.3.2.

[lauragails]

When running the code below, I have the following error (in jpeg). Happy to send full log files, but would prefer to do that over email.

Note that I ran these after going into the container with this command:
sudo docker-compose exec pipeline-runner /bin/bash # to enter container

(I also had to previously edit the hailctl dataproc start command to include a --region and --project flag, but the dataproc creation was successful!)

hailctl dataproc submit seqr-loading-cluster \
    seqr_loading.py --pyfiles "lib,../hail_scripts" \
    SeqrVCFToMTTask --local-scheduler \
         --source-paths "${SOURCE_FILE}" \
         --dest-path "${DEST_FILE}" \
         --genome-version "${BUILD_VERSION}" \
         --sample-type "${SAMPLE_TYPE}" \
         --vep-config-json-path "${REFERENCE_DATA_BUCKET}/vep-${FULL_BUILD_VERSION}-loftee-dataproc.json" \
         --reference-ht-path  "${REFERENCE_DATA_BUCKET}/combined_reference_data_grch${BUILD_VERSION}.ht" \
         --clinvar-ht-path "${REFERENCE_DATA_BUCKET}/clinvar.${FULL_BUILD_VERSION}.ht"

[hanars]

Its hard to say for sure since you are not sharing what variables you are running the command with, but it looks like you have an extra slash somewhere in your arguments. The paths are all being formatted as gs://gala-mssm/seqr_docker//data with that extra/ between seqr_docker and data

[lauragails]

Understood - that's not it, I'm not sure why both are needed, but gsutils does list that file as the path.

The full command is:

GS_FILE_PATH=data/GRCh38         # the desired file path. Good to include build version and/ or sample type to directory structure
FILENAME=basename.vcf.gz            # basename changed 


BUILD_VERSION=38
SAMPLE_TYPE=WES                  # can be WES or WGS
INDEX_NAME=basename_test
GS_BUCKET=gs://gala-mssm/seqr_docker/
INPUT_FILE_PATH=/${GS_FILE_PATH}/${FILENAME}

hailctl dataproc submit seqr-loading-cluster \
    seqr_loading.py --pyfiles "lib,../hail_scripts" \
    SeqrVCFToMTTask --local-scheduler \
         --source-paths "${SOURCE_FILE}" \
         --dest-path "${DEST_FILE}" \
         --genome-version "${BUILD_VERSION}" \
         --sample-type "${SAMPLE_TYPE}" \
         --vep-config-json-path "${REFERENCE_DATA_BUCKET}/vep-${FULL_BUILD_VERSION}-loftee-dataproc.json" \
         --reference-ht-path  "${REFERENCE_DATA_BUCKET}/combined_reference_data_grch${BUILD_VERSION}.ht" \
         --clinvar-ht-path "${REFERENCE_DATA_BUCKET}/clinvar.${FULL_BUILD_VERSION}.ht"

[hanars]

I think you have a trailing slash on GS_BUCKET - it should be GS_BUCKET=gs://gala-mssm/seqr_docker

[lauragails]

Unfortunately I've carried it throughout - which has definitely added some extra clicks. But fortunately I was consistent, and the paths below exist. I added a dry-run flag that lists the command submitted from the above command

In the meantime, I'm continuing to look. Thank you!

[lauragails]

This log may be easier to read - I put a script adding the flag(s) I needed to make the dataproc into the bucket and executed it just like the tutorial. Here is my output now for [my slightly modified load_data_dataproc script] docker-compose exec pipeline-runner load_data_dataproc.sh $BUILD_VERSION $SAMPLE_TYPE $INDEX_NAME $GS_BUCKET $INPUT_FILE_PATH:

SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
SLF4J: Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
SLF4J: Ignoring binding found at [jar:file:/usr/lib/spark/jars/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
/opt/conda/default/lib/python3.10/site-packages/luigi/parameter.py:296: UserWarning: Parameter "hgmd_ht_path" with value "None" is not of type string.
warnings.warn('Parameter "{}" with value "{}" is not of type string.'.format(param_name, param_value))
DEBUG: Checking if SeqrVCFToMTTask(source_paths=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz, dest_path=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt, genome_version=38, vep_runner=VEP, reference_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=False, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=) is complete
DEBUG: Checking if VcfFile(filename=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz) is complete
INFO: Informed scheduler that task SeqrVCFToMTTask_gs___gala_mssm_s_VARIANTS_gs___gala_mssm_s_ed6d229b14 has status PENDING
INFO: Informed scheduler that task VcfFile_gs___gala_mssm_s_b6c539a3a5 has status DONE
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 14934] Worker Worker(salt=3378674936, workers=1, host=seqr-loading-cluster-m, username=root, pid=14934) running SeqrVCFToMTTask(source_paths=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz, dest_path=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt, genome_version=38, vep_runner=VEP, reference_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=False, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=)
Initializing Hail with default parameters...
Running on Apache Spark version 3.3.0
SparkUI available at http://seqr-loading-cluster-m.c.seaver-wes.internal:35027
Welcome to
__ __ <>__
/ // /__ __/ /
/ __ / _ `/ / /
// //_,/// version 0.2.114-cc8d36408b36
LOGGING: writing to /home/hail/hail-20230426-2046-0.2.114-cc8d36408b36.log
ERROR: [pid 14934] Worker Worker(salt=3378674936, workers=1, host=seqr-loading-cluster-m, username=root, pid=14934) failed SeqrVCFToMTTask(source_paths=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz, dest_path=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.mt, genome_version=38, vep_runner=VEP, reference_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/combined_reference_data_grch38.ht, interval_ref_ht_path=, clinvar_ht_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/clinvar.GRCh38.ht, hgmd_ht_path=None, sample_type=WES, dont_validate=False, dataset_type=VARIANTS, remap_path=, subset_path=, vep_config_json_path=gs://gala-mssm/seqr_docker//reference_data/GRCh38/vep-GRCh38-loftee-dataproc.json, grch38_to_grch37_ref_chain=gs://hail-common/references/grch38_to_grch37.over.chain.gz, hail_temp_dir=)
Traceback (most recent call last):
File "/opt/conda/default/lib/python3.10/site-packages/luigi/worker.py", line 203, in run
new_deps = self._run_get_new_deps()
File "/opt/conda/default/lib/python3.10/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
task_gen = self.task.run()
File "/tmp/1e7bcd435d634dd2bb10010f7a7111ff/seqr_loading.py", line 81, in run
check_if_path_exists(source_path, "source_path")
File "/tmp/1e7bcd435d634dd2bb10010f7a7111ff/seqr_loading.py", line 23, in check_if_path_exists
raise ValueError(f"{label} path not found: {path}")
ValueError: source_path path not found: gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz
DEBUG: 1 running tasks, waiting for next task to finish
INFO: Informed scheduler that task SeqrVCFToMTTask_gs___gala_mssm_s_VARIANTS_gs___gala_mssm_s_ed6d229b14 has status FAILED
DEBUG: Asking scheduler for work...
DEBUG: Done
DEBUG: There are no more tasks to run at this time
DEBUG: There are 1 pending tasks possibly being run by other workers
DEBUG: There are 1 pending tasks unique to this worker
DEBUG: There are 1 pending tasks last scheduled by this worker
INFO: Worker Worker(salt=3378674936, workers=1, host=seqr-loading-cluster-m, username=root, pid=14934) was stopped. Shutting down Keep-Alive thread
INFO:
===== Luigi Execution Summary =====

Scheduled 2 tasks of which:

• 1 complete ones were encountered:
  • 1 VcfFile(filename=gs://gala-mssm/seqr_docker//data/GRCh38/AnVIL_ccdg_asc_ndd_daly_talkowski_mayo_asd_exome_b19.vcf.gz)
• 1 failed:
  • 1 SeqrVCFToMTTask(...)

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

[hanars]

It seems you created a duplicate issue for this: #3356. I have converted that issue back to a discussion, as this is still a discussion about issues with how you are running the command and not an actual actionable code change for the seqr dev team. I responded their as well, but this really does seem to be an issue that you have a trailing slash in your GS_BUCKET variable, which is something you yourself are setting and can change