feat: Speed up struct and named_struct using invoke_with_args

[pepijnve]

By implementing invoke_with_args the fields derived in return_type(_from_args) can be reused (performance) and the duplicate derivation logic can be removed.

Which issue does this PR close?

• Closes Avoid deriving Fields for each invocation of struct and named_struct #14275.

Rationale for this change

Reusing the already derived fields results in a small, but measurable, performance improvement according to the added micro benchmark.

What changes are included in this PR?

Replaces the invoke_batch implementations with invoke_with_args implementations.

Are these changes tested?

Covered by the existing dataframe/dataframe_functions tests

Are there any user-facing changes?

Not at the SQL level. At the library level calls to invoke_batch for these two functions will result in a runtime error.

[lichuang]

If delete invoke_batch function, the default invoke_batch function will call invoke when args is not empty, but invoke function is left impl by Specific type, is it ok?

[pepijnve]

I wasn't sure what the intention is for the deprecated functions from an API user point of view. I had interpreted ScalarUDF(Impl) as an extension point rather than as public API that library users would call directly. If that assumption is correct, then it doesn't matter if we remove the invoke_batch implementation since ScalarFunctionExpr never calls this method. The only usages of invoke_batch are in benchmark and test code.
Perhaps @alamb, who logged #13515, can provide some insight here.

[alamb]

THe reason we kept around deprecated functions is to give downstream users of DataFusion time to adjust their code -- especially on upgrade having a deprecated function with guidance of what to change has been helpful

You can read more about this strategy here:

• https://datafusion.apache.org/library-user-guide/api-health.html

[pepijnve]

I'm coming from the Java world so I'll use the terminology from there; not sure what the equivalent is in Rust lingo. It's not explicitly stated which parts of the library are Service Provider Interface vs Application Programming Interface. My assumption was that ScalarUDFImpl is SPI and invoke_batch was kept around to not break all existing implementations and ScalarFunctionExpr is the API side of things which doesn't expose invoke_batch. Is that a correct interpretation?

[alamb]

My assumption was that ScalarUDFImpl is SPI

Yes, that is accurate in my understanding

and invoke_batch was kept around to not break all existing implementations

Yes that is also my understanding

and ScalarFunctionExpr is the API side of things which doesn't expose invoke_batch. Is that a correct interpretation?

I would say ScalarFunctionExpr is an implementation detail of how functions are invoked in the ExecutionPlan (aka the physical execution)

The split between logical/physical plans is explained a bit in the API docs / intro videos in case you are interested:
https://docs.rs/datafusion/latest/datafusion/index.html#query-planning-and-execution-overview

[alamb]

Thank you @pepijnve

This is a really nice cleanup and a good example of using the ScalarUDFImpl API👌

[alamb]

😍

[alamb]

I also took the liberty of merging up from main to get the CI to run again.

[alamb]

Update here is we are very close to cutting the 45 release branch. See

• More details on Release DataFusion 45.0.0 #14008 (comment)

I plan to merge this PR once that is done

[alamb]

We have a release branch for 45 now -- see #14008 (comment)

Let's keep the main moving!

[alamb]

Thanks again @pepijnve