[SPARK-53779][SQL][CONNECT] Implement transform() in Column API

[Yicong-Huang]

What changes were proposed in this pull request?

Add transform() API in Columns API, similar to Dataset.transform():

def transform(f: Column => Column): Column

Why are the changes needed?

We want to give users a way to chain their methods, such as

df.select($"fruit"
  .transform(addPrefix)
  .transform(uppercase)
)

This pattern is also easier for AI agents to learn and write.

Does this PR introduce any user-facing change?

Yes. New API is introduced.

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

Tests generated by Copilot.

[ueshin]

LGTM.
We need to add this to PySpark, but it's ok to have a separate PR.

[Yicong-Huang]

LGTM. We need to add this to PySpark, but it's ok to have a separate PR.

opened a new ticket for PySpark. https://issues.apache.org/jira/browse/SPARK-53841

[zhengruifeng]

what about adding a test for nested transform? e.g. a.transform(_.transform(fn.upper))

[Yicong-Huang]

added!

[cloud-fan]

.transform(trim).transform(upper) is also chaining. Let's name them: Column.transform: built-in functions and Column.transform: lambda functions.

[Yicong-Huang]

changed!

[cloud-fan]

thanks, merging to master!