-
Notifications
You must be signed in to change notification settings - Fork 80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat!: Add multi-column support to AggFormula #6206
Conversation
engine/table/src/test/java/io/deephaven/engine/table/impl/TestAggBy.java
Outdated
Show resolved
Hide resolved
table-api/src/main/java/io/deephaven/api/agg/AggregationOutputs.java
Outdated
Show resolved
Hide resolved
table-api/src/main/java/io/deephaven/api/agg/ColumnAggregation.java
Outdated
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
Outdated
Show resolved
Hide resolved
server/src/main/java/io/deephaven/server/table/ops/AggregationAdapter.java
Outdated
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaChunkedOperator.java
Outdated
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
Outdated
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
Outdated
Show resolved
Hide resolved
py/server/deephaven/agg.py
Outdated
cols (Union[str, List[str]]): If provided, supplies the column(s) to aggregate on, can be renaming expressions, | ||
i.e. "new_col = col". Default is None, which can be valid when the `formula` argument supplies the input | ||
column names or when used in Table agg_all_by operation |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Supporting cols
as a List[str]
seems to suggest that we might want to support a list of formulas.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure
formula(['output_col1=(input_col1 + input_col2) * input_col3', 'output_col2=(input_col21 + input_col22) * input_col23'])
is better than
[formula('output_col1=(input_col1 + input_col2) * input_col3'), formula('output_col2=(input_col21 + input_col22) * input_col23')]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Our other methods do the second. I'm pointing out the inconsistency.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a technical reason why we can't do that immediately for Java (already using String...
var args to support Pair
args). We could do it for Python but then we'd have a mismatch.
When/if we completely sunset the parameterized version, we can implement this without any conflict.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I created this ticket to capture this request for multiple formula per call. #6392
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Outdated
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Outdated
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Outdated
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Show resolved
Hide resolved
...e/src/main/java/io/deephaven/engine/table/impl/rangejoin/SupportedRangeJoinAggregations.java
Outdated
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
Outdated
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Outdated
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/FormulaChunkedOperator.java
Outdated
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's worthwhile bringing up the gRPC structuring layer given work that Charles want to do there too...
proto/proto-backplane-grpc/src/main/proto/deephaven/proto/table.proto
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
.
server/src/main/java/io/deephaven/server/table/ops/AggregationAdapter.java
Outdated
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
Outdated
Show resolved
Hide resolved
engine/table/src/main/java/io/deephaven/engine/table/impl/by/AggregationProcessor.java
Outdated
Show resolved
Hide resolved
...table/src/main/java/io/deephaven/engine/table/impl/by/FormulaMultiColumnChunkedOperator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think all we're missing is proper Java-side testing in QueryTableAggregationTest
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was wrong, TestAggBy coverage looks adequate to me.
Labels indicate documentation is required. Issues for documentation have been opened: Community: deephaven/deephaven-docs-community#363 |
NOTE: this operator allows you to use key columns (group by columns) as input to the formula. In this case, the key column value is presented to the formula as a scalar and not a vector.
Python examples:
Groovy examples: