Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Not fallback for function spark_partition_id and monotonically_increasing_id #5046

Closed

Conversation

gaoyangxiaozhu
Copy link
Contributor

Not fallback for function spark_partition_id and monotonically_increasing_id when input scheam is empty

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

(Fixes: #ISSUE-ID)

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@gaoyangxiaozhu
Copy link
Contributor Author

@zhli1142015 current monotonically_increasing_id implement causes inconsistent behavior vs vinalla spark.

looks in velox we always use one function instance when query has duplicate same function expr call

image

simple reproduce:

val df = sc.parallelize(Seq[Int](), 2).mapPartitions { _ => Iterator(Tuple1(1), Tuple1(2)) }.toDF("a")
val df2 = df.select(monotonically_increasing_id(), expr("monotonically_increasing_id()"))

@zhouyuan zhouyuan changed the title [VL ] Not fallback for function spark_partition_id and monotonically_increasing_id [VL] Not fallback for function spark_partition_id and monotonically_increasing_id Mar 20, 2024
@zhli1142015
Copy link
Contributor

Acked, let me check.

@FelixYBW
Copy link
Contributor

Can you also help to add input_file_name() function? It's very useful for debug purpose.

@gaoyangxiaozhu
Copy link
Contributor Author

Can you also help to add input_file_name() function? It's very useful for debug purpose.

acked

Copy link

github-actions bot commented May 6, 2024

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label May 6, 2024
Copy link

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.

@github-actions github-actions bot closed this May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale stale
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants