spark connect fails when performing a .show()

[universalmind303]

Describe the bug

# %%
import daft
from daft.daft import connect_start
from pyspark.sql import SparkSession


server = connect_start()
url = f"sc://localhost:{server.port()}"
session = SparkSession.builder.appName("DaftConfigTest").remote(url).getOrCreate()


session.createDataFrame([("cory", 100)], ["name", "age"]).show()

results in "Error in Daft server: Unsupported relation type: ShowString"

To Reproduce

No response

Expected behavior

No response

Component(s)

Other

Additional context

It appears that currently our show logic exists purely in python. As a prerequisite, we'll need to refactor that logic into rust so that it can be used from spark connect.

[andrewgazelka]

Describe the bug

# %%
import daft
from daft.daft import connect_start
from pyspark.sql import SparkSession


server = connect_start()
url = f"sc://localhost:{server.port()}"
session = SparkSession.builder.appName("DaftConfigTest").remote(url).getOrCreate()


session.createDataFrame([("cory", 100)], ["name", "age"]).show()

results in "Error in Daft server: Unsupported relation type: ShowString"

To Reproduce

No response

Expected behavior

No response

Component(s)

Other

Additional context

It appears that currently our show logic exists purely in python. As a prerequisite, we'll need to refactor that logic into rust so that it can be used from spark connect.

also note createDataFrame with strings is currently bugged (I am working on fixing it)

Can we just use Display rust impl for now?

[universalmind303]

Can we just use Display rust impl for now?

no, the logical plan display is equivalent to df.explain().

df.show() shows a small sample of the materialized dataset similar to df.limit(10).collect()

[universalmind303]

@andrewgazelka I can take on this one.