Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transform functions not able to process virtual columns #14591

Open
lnbest0707-uber opened this issue Dec 4, 2024 · 0 comments
Open

Transform functions not able to process virtual columns #14591

lnbest0707-uber opened this issue Dec 4, 2024 · 0 comments

Comments

@lnbest0707-uber
Copy link
Contributor

Pinot transform functions is very useful to extract the pluggable information by ad-hoc usages. For example, users could use it to extract the timestamp related info from the kafka metadata.
Meanwhile, there are some use cases that user want to extract info from virtual columns. E.g.
{ "columnName": "pinotPartitionNumber", "transformFunction": "splitPart($segmentName, '__', 1)" }
could extract the partition information from the message. (Though we can use same function in query, it is very slow if the table becomes large.)
However, that is not possible currently due to transform function's fetching data limitation. It could only retrieve data from the current row data Map<String, Object> but not able to get those segment level virtual column data. It would be great to remove such limitations and make transform functions more flexible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants