Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

For each agg function with input param as <T>, Add an equivalent agg function with input param as array<T> #23034

Open
cploonker opened this issue Jun 19, 2024 · 1 comment

Comments

@cploonker
Copy link

cploonker commented Jun 19, 2024

Proposal

For each aggregate function with input param as (T), Add an equivalent agg function with input param as array(T)

Why NOT use CROSS JOIN UNNEST and then do a SUM?

Existing work around in SQL, make the query very inefficient for large tables.
CROSS JOIN UNNEST can only be done with one array column at a time. What if i want to SUM two such columns in a single pass.

Expected Behavior or Use Case

Let us take the example of agg function APPROX_DISTINCT(column)
Add an overloaded agg function APPROX_DISTINCT(Array_column).
Meaning I should be able to run a APPROX_DISTINCT on a column of type ARRAY(T)

It does not work but if it worked something on the lines of APPROX_DISTINCT(UNNEST(array_column)).
i.e. COUNT(UNNEST(array_column)) would be same as SUM(CARDINALITY(array_column))

Apply the same to every single agg function e.g. AVG, SUM....

Presto Component, Service, or Connector

Presto inbuilt Aggregate functions

Possible Implementation

Overloaded versions of the function with a different parameter type

Example Screenshots (if appropriate):

Context

Related thread
https://stackoverflow.com/questions/64563863/presto-aggregate-the-arrays-for-all-columns
trinodb/trino#6292

trinodb/trino#22445

@rschlussel
Copy link
Contributor

I think this would be semantically very problematic. Some aggregation functions can take arrays as input and would aggregate on the array in the usual way. If I sawapprox_distinct(array_column), I would expect it to give me the number of distinct arrays, and not the number of distinct elements across all arrays.

CROSS JOIN UNNEST can only be done with one array column at a time. What if i want to SUM two such columns in a single pass.
You can unnest multiple array columns at a time. see https://prestodb.io/docs/current/sql/select.html#unnest.

For some of the cases you described above, if you don't want to unnest, you can consider first using an array_function that does part of the aggregation for you on each array row, and then doing an aggregation on top of it. For example:
If you are trying to get the number of distinct elements across all arrays, you could unnest and do an approx_distinct as you described. Alternatively, you could do cardinality(set_union(array_column)).

If you are trying to get the sum across all arrays, you could unnest and sum as you described, or you could do sum(array_sum(array_column)

also cc: @kaikalur

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants