feat: support Ibis 6.x UDF annotations

[omriel1]

What happened?

Hi!
I've been trying to create a substrait plan with a udf using the new annotations (udf.scalar.python, starting of ibis 6.x). For some reason it fails. This is what I tried (this udf is taken from the ibis blog: https://ibis-project.org/blog/rendered/ibis-version-6.0.0-release/#udfs) :

import ibis
from ibis import _

from ibis import udf
from ibis_substrait.compiler.core import SubstraitCompiler

t = ibis.examples.penguins.fetch()

@udf.scalar.python
def num_vowels(s: str) -> int:
    return sum(map(s.lower().count, "aeiou"))

new_t = t.mutate(num_vowels=lambda t: num_vowels(t.species))
query = new_t.filter(_.num_vowels>5)

compiler = SubstraitCompiler(
    udf_uri="urn:arrow:substrait_simple_extension_function"
)
plan = compiler.compile(query)

I'd expect it to be compiled into a Substrait plan, which is similar to those created in your tests (https://github.com/ibis-project/ibis-substrait/blob/main/ibis_substrait/tests/integration/test_pyarrow.py)

I'll also mention that my end goal is just create an arbitrary udf using Ibis, create a query and compile it into a Substrait plan. Then register the udf in pyarrow/DuckDB and consume the plan. While it works fine for me with the legacy annotations, I wonder how should it be done now.
Thanks!

What version of ibis-substrait are you using?

ibis_substrait-2.29.1

What substrait consumer(s) are you using, if any?

Currently none, Ideally Acero and duckDB

Relevant log output

args = (<ibis.expr.operations.relations.DatabaseTable object at 0x14ff56b60>,)
kwargs = {'child_rel_field_offsets': {}, 'compiler': <ibis_substrait.compiler.core.SubstraitCompiler object at 0x13fb12e50>}

    @functools.singledispatch
    def translate(*args: Any, **kwargs: Any) -> Any:
>       raise NotImplementedError(*args)
E       NotImplementedError: <ibis.expr.operations.relations.DatabaseTable object at 0x14ff56b60>

[gforsyth]

Thanks for raising this, @OmriLevyTau -- there are a few things going on here:

1. Explicit Ibis 6.x support was just released so you should upgrade to ibis_substrait 3.0
2. ibis_substrait expects unbound tables, e.g. not tied to an existing backend. The example datasets are loaded in DuckDB -- that's the source of the error you are seeing -- it wants an UnboundTable not a DatabaseTable. There's an easy fix for this, which is to call unbind() on the expression before you try to compile, e.g. plan = compiler.compile(query.unbind())
3. Once you do this, it still won't work, because I haven't yet implemented support for the new Ibis UDF annotations, so I'll mark this issue as a feature request for that support. In the interim, if you use the old annotation style (now located in ibis.legacy.udf) that should work.

[omriel1]

Thank you! it makes now more sense.