feat(sql): fuse distinct with other select nodes when possible

[jcrist]

This generates more concise SQL when chaining relational operations with .distinct().

Fixes #9905.

[jcrist]

Hmmm, this has turned up some SQL analysis bugs (err, I think they're bugs) in spark & risingwave. In both cases the presence of a column alias inside a SELECT DISTINCT is causing the original column name to not be in scope in the ORDER BY part of the query (it is still in scope in other parts like WHERE).

--- Order by the alias works
SELECT DISTINCT x AS z FROM test ORDER BY z
--- Order by the original name fails, saying no column named `z` found
SELECT DISTINCT x AS z FROM test ORDER BY x
--- Order by the original name works if you remove the DISTINCT
SELECT x AS z FROM test ORDER BY x

• Is this a bug in our SQL generated that most backends just happen to support, or is this a bug in pyspark/risingwave?
• Is using the alias instead of the original name everywhere a valid fix?

[jcrist]

Ok, I have a fix for this (I think), and now know more dumb things about SQL dialects than I did before.

[jcrist]

Ok, provided tests pass (I didn't check the cloud backends myself), I think this is ready for review. Due to very dumb backend-specific reasons this was more work than initially thought. I've marked a few code spots below for review.

[jcrist]

There are dragons here.

[cpcloud]

Missed opportunity to call this supplant_merged_with_ordered_distinct 😂

[jcrist]

There are also dragons here.

[cpcloud]

I'll run the clouds.

[cpcloud]

Snowflake is passing, but BigQuery is raising this exception:

E               google.api_core.exceptions.BadRequest: 400 ORDER BY clause expression references t0.id which is not visible after SELECT DISTINCT at [11:7]; reason: invalidQuery, location: query, message: ORDER BY clause expression references t0.id which is not visible after SELECT DISTINCT at [11:7]
E
E               Location: US
E               Job ID: d7ab4e0d-b30f-4afc-8ed7-286b71a895a5

@jcrist I'm guessing BigQuery needs the new rewrite rule applied?

[jcrist]

Huh, thought I'd run the cloud tests in this PR already, will fix up.

[jcrist]

Cloud tests are passing, should be good-to-go.

[cpcloud]

LGTM, thanks for chugging through the muck!

[cpcloud]

Does this cover the alias-in-outer-project case?

SELECT a, b AS c
FROM (
  SELECT DISTINCT
    a, b
  FROM t
)

would become

SELECT DISTINCT
  a, b AS c
FROM t

If not, fine to either handle later or perhaps never if it doesn't come up.

[jcrist]

This branch is for both being distinct, so:

SELECT DISTINCT a, b as c
FROM (SELECT DISTINCT a, b FROM t)

In this case, yes, the aliases are properly handled.

The branch on line 270 handles the SELECT ... FROM (SELECT DISTINCT ...) case. Right now it only works with SELECT * cases, but we might be able to make it work with outer queries that rename columns but otherwise select all of them. Right now I don't think that's worth it.

[cpcloud]

Missed opportunity to call this supplant_merged_with_ordered_distinct 😂