dagster-gcp - Dataproc: retrieve job logs from GCS

[natekupp]

When a dataproc job fails, we get an error like:

DagsterEventType.STEP_FAILURE for step dataproc_solid.compute dagster_gcp.dataproc.types.DataprocError: Job error: {'state': 'ERROR', 'details': 'Job failed with message [Exception in thread "main" scala.reflect.internal.FatalError: Incorrect options]. Additional details can be found in 'gs://dataproc-.../driveroutput'.

Inspecting those error logs shows the real problem:

gsutil cat "gs://dataproc-.../driveroutput*"

Error: Unknown option --date 2019-05-28
Error: Missing option --date
Usage: EventPipeline [options]

  --s3-bucket <value>      S3 bucket to read
  --s3-prefix <value>      S3 prefix to read
  --gcs-input-bucket <value>
                           GCS input bucket to read
  --gcs-output-bucket <value>
                           GCS output bucket to write
  --local-path <value>     Local path prefix
  --date <value>
Exception in thread "main" scala.reflect.internal.FatalError: Incorrect options
	at io.dagster.events.EventPipelineConfig$$anonfun$parse$1.apply(EventPipeline.scala:150)
	at io.dagster.events.EventPipelineConfig$$anonfun$parse$1.apply(EventPipeline.scala:150)
	at scala.Option.getOrElse(Option.scala:121)
	at io.dagster.events.EventPipelineConfig$.parse(EventPipeline.scala:149)
	at io.dagster.events.EventPipeline$.run(EventPipeline.scala:73)
	at io.dagster.events.models.SparkJob.main(SparkJob.scala:17)
	at io.dagster.events.EventPipeline.main(EventPipeline.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)