Show a dataproc link on FlyteConsole

[pingsutw]

Tracking issue

flyteorg/flyte#3936

Why are the changes needed?

It will show the log link on FlyteConsole

What changes were proposed in this pull request?

Get the log link in agent and return to Propeller

How was this patch tested?

from datetime import datetime, timedelta

from airflow.operators.bash import BashOperator
from airflow.providers.google.cloud.operators.dataproc import DataprocDeleteClusterOperator, \
    DataprocSubmitSparkJobOperator, DataprocCreateClusterOperator
from airflow.utils import trigger_rule
from pytz import UTC
from flytekit import workflow

x = (datetime.now(tz=UTC) + timedelta(seconds=21)).time()
cluster_name = "flyte-dataproc-demo"


@workflow
def airflow_wf():
    # TimeSensor(task_id="time_sensor", target_time=x)
    BashOperator(task_id="airflow_bash_operator", bash_command="echo hello")

    create_cluster = DataprocCreateClusterOperator(
        task_id="create_dataproc_cluster1",
        image_version="2.0.27-debian10",
        storage_bucket="opta-gcp-dogfood-gcp",
        service_account="dogfoodgcp-userflyterol-odkb@dogfood-gcp-dataplane.iam.gserviceaccount.com",
        # service_account_scopes=["https://www.googleapis.com/auth/cloud-platform"],
        master_machine_type="n1-highmem-32",
        master_disk_size=1024,
        num_workers=2,
        worker_machine_type="n1-highmem-64",
        worker_disk_size=1024,
        region="us-west1",
        cluster_name=cluster_name,
        project_id="dogfood-gcp-dataplane",
    )

    spark_on_dataproc = DataprocSubmitSparkJobOperator(
        job_name="spark_pi",
        task_id="run_spark",
        dataproc_jars=["file:///usr/lib/spark/examples/jars/spark-examples.jar"],
        main_class="org.apache.spark.examples.JavaWordCount",
        arguments=["gs://opta-gcp-dogfood-gcp/spark/file.txt"],
        cluster_name=cluster_name,
        region="us-west1",
        # project_id="dogfood-gcp-dataplane",
    )

    delete_cluster = DataprocDeleteClusterOperator(
        task_id="delete_dataproc_cluster1",
        project_id="dogfood-gcp-dataplane",
        cluster_name=cluster_name,
        region="us-west1",
        retries=3,
        retry_delay=timedelta(minutes=5),
        email_on_failure=True,
        trigger_rule=trigger_rule.TriggerRule.ALL_DONE,
    )

    create_cluster >> spark_on_dataproc >> delete_cluster

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

NA

Docs link

NA

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.05%. Comparing base (f4d894a) to head (12c3cc5).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2333      +/-   ##
==========================================
- Coverage   83.09%   83.05%   -0.05%     
==========================================
  Files         324      324              
  Lines       24860    24860              
  Branches     3761     3547     -214     
==========================================
- Hits        20657    20647      -10     
- Misses       3582     3590       +8     
- Partials      621      623       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Future-Outlier]

LGTM.
Let's merge it.