[SPARK-52470][ML][CONNECT] Support model summary offloading

[WeichenXu123]

What changes were proposed in this pull request?

This PR makes Spark Connect ML supporting model summary offloading.

Model summary offloading is hard to support because it contains a Spark dataset which can't be easily serialized in Spark driver (NOTE: we can't java serializer to serialize the Spark dataset logical plan otherwise it is a RCE vulnerability),

to address the issue, when saving Summary to disk, it only saves the necessary data fields,
when loading Summary back, the client needs to send the dataset to Spark driver again,
to achieve it, 2 new proto messages are introduced:

1. CreateSummary in MlCommand

  // This is for re-creating the model summary when the model summary is lost
  // (model summary is lost when the model is offloaded and then loaded back)
  message CreateSummary {
    ObjectRef model_ref = 1;
    Relation dataset = 2;
  }

2: model_summary_dataset in MlRelation

  // (Optional) the dataset for restoring the model summary
  optional Relation model_summary_dataset = 3;

Why are the changes needed?

Support model summary offloading.
Without this, the model summary will be evicted from Spark driver memory after default 15min timeout, results in model.summary API unavailability.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[WeichenXu123]

The CI failure https://github.com/WeichenXu123/spark/actions/runs/15703082903/job/44242454735#step:7:3170 is not related to my PR.

[WeichenXu123]

merged to master

[LuciferYang]

@WeichenXu123 The test case "pyspark.ml.tests.connect.test_parity_pipeline" experienced timeout issues twice in the recent commit pipelines. And I don't think I've encountered this issue before. Could you please help confirm whether this is related to the current pr? Thanks ~

• https://github.com/apache/spark/actions/runs/15722059159/job/44304774885
• https://github.com/apache/spark/actions/runs/15723529338/job/44309009098

also cc @zhengruifeng

[zhengruifeng]

it may also hang in other ml connect tests:
pyspark.ml.tests.connect.test_parity_clustering in https://github.com/apache/spark/actions/runs/15742416356/job/44370943991

pyspark.ml.tests.connect.test_parity_classification in https://github.com/apache/spark/actions/runs/15741346719/job/44367299872