Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI][Java][Python] Integration spark jobs are failing due to missing java folder #44980

Closed
raulcd opened this issue Dec 9, 2024 · 3 comments
Closed

[CI][Java][Python] Integration spark jobs are failing due to missing java folder #44980

raulcd opened this issue Dec 9, 2024 · 3 comments
Assignees
@raulcd
Labels
Component: Continuous Integration Component: Java Component: Python Type: bug
Milestone
19.0.0

Comments

@raulcd
Copy link
Member

raulcd commented Dec 9, 2024

Describe the bug, including details regarding any error messages, version, and platform.

The Spark integration jobs were using the Java implementation to build spark against. They are currently failing with:

 Removed build tracker: '/tmp/pip-build-tracker-_6u01r9l'
+ popd
+ '[' OFF == ON ']'
/
/arrow/ci/scripts/integration_spark.sh: line 36: pushd: /arrow/java: No such file or directory

This should be investigated and fixed. I am unsure whether this should be moved to the arrow-java implementation.

Component(s)

Continuous Integration, Java, Python
@raulcd
Copy link
Member Author

raulcd commented Dec 9, 2024

@kou it seems we removed spark being built with latest Java arrow here: #44946 but we missed to remove the version retrieval.
Do we want to add a job to build spark against Arrow Java main? This job should potentially live in arrow-java

raulcd added a commit to raulcd/arrow that referenced this issue Dec 9, 2024
@raulcd
apacheGH-44980: [CI] Remove unnecessary retrieval of arrow version fr…
c410de9 
…om Java on Spark integration
@github-actions github-actions bot mentioned this issue Dec 9, 2024
Merged
@kou
Copy link
Member

kou commented Dec 9, 2024

Oh, sorry.
I missed this case.

I've opened an issue on apache/arrow-java: apache/arrow-java#451

@assignUser assignUser mentioned this issue Dec 11, 2024
Open
raulcd added a commit that referenced this issue Dec 11, 2024
@raulcd
GH-44980: [CI] Remove retrieval of Arrow version from Java on Spark i…
5db2843 
…ntegration and update test structure for PySpark (#44981)

### Rationale for this change

The job is currently failing.

### What changes are included in this PR?

Remove unnecessary check on Java code and refactor pyspark test modules to follow new test structure: apache/spark#49104

### Are these changes tested?

Via archery

### Are there any user-facing changes?

No
* GitHub Issue: #44980

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
@raulcd raulcd added this to the 19.0.0 milestone Dec 11, 2024
@raulcd
Copy link
Member Author

raulcd commented Dec 11, 2024

Issue resolved by pull request 44981
#44981

@raulcd raulcd closed this as completed Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raulcd raulcd

Labels
Component: Continuous Integration Component: Java Component: Python Type: bug
Projects
None yet
Milestone
19.0.0
Development

No branches or pull requests
2 participants
@kou @raulcd