Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-3068: Remove parquet-scala #3063

Merged
merged 1 commit into from
Nov 20, 2024
Merged

GH-3068: Remove parquet-scala #3063

merged 1 commit into from
Nov 20, 2024

Conversation

Fokko
Copy link
Contributor

@Fokko Fokko commented Nov 15, 2024 •
edited
Loading

Rationale for this change

Based on the [DISCUSS] thread: https://lists.apache.org/thread/scdq9t2gvvs4glhq0qx4qcvfp62j793s

Closes #3068

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@Fokko Fokko mentioned this pull request Nov 15, 2024
Closed
@pan3793
Copy link
Member

pan3793 commented Nov 15, 2024

Can we leave some words on README or somewhere to mention this removal?

@Fokko
Copy link
Contributor Author

Fokko commented Nov 15, 2024

@pan3793 I noticed that the Scala DSL wasn't mentioned to the list, I've added it 👍

@wgtmac
Copy link
Member

wgtmac commented Nov 18, 2024

It seems that recent builds all fail with the following error:

[INFO] ---------------< org.apache.parquet:parquet-scala_2.12 >----------------
[INFO] Building Apache Parquet Scala 1.15.0-SNAPSHOT                    [16/18]
[INFO]   from parquet-scala/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] Downloading from scala-tools.org: https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom
Warning:  Checksum validation failed, expected '<!DOCTYPE' (REMOTE_EXTERNAL) but is actually '3ee5a4fdb85dc0b9c676a21a40b6f1cc2d748a03' from scala-tools.org for https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom
Warning:  Could not validate integrity of download from https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom
org.eclipse.aether.transfer.ChecksumFailureException: Checksum validation failed, expected '<!DOCTYPE' (REMOTE_EXTERNAL) but is actually 'e0509cfe82a6ffad07497a39b82d6e0a82a5b1ac'
    at org.eclipse.aether.connector.basic.ChecksumValidator.validateExternalChecksums (ChecksumValidator.java:179)
    at org.eclipse.aether.connector.basic.ChecksumValidator.validate (ChecksumValidator.java:109)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask (BasicRepositoryConnector.java:484)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:414)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get (BasicRepositoryConnector.java:260)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads (DefaultArtifactResolver.java:537)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:449)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:261)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact (DefaultArtifactResolver.java:243)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:241)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:182)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor (DefaultRepositorySystem.java:269)
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:107)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.lambda$getPluginDescriptor$0 (DefaultMavenPluginManager.java:183)
    at org.apache.maven.plugin.DefaultPluginDescriptorCache.lambda$get$0 (DefaultPluginDescriptorCache.java:72)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent (ConcurrentHashMap.java:1660)
    at org.apache.maven.plugin.DefaultPluginDescriptorCache.get (DefaultPluginDescriptorCache.java:70)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getPluginDescriptor (DefaultMavenPluginManager.java:181)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getMojoDescriptor (DefaultMavenPluginManager.java:271)
    at org.apache.maven.plugin.DefaultBuildPluginManager.getMojoDescriptor (DefaultBuildPluginManager.java:214)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleMappingDelegate.calculateLifecycleMappings (DefaultLifecycleMappingDelegate.java:101)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateLifecycleMappings (DefaultLifecycleExecutionPlanCalculator.java:239)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateMojoExecutions (DefaultLifecycleExecutionPlanCalculator.java:197)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan (DefaultLifecycleExecutionPlanCalculator.java:119)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan (DefaultLifecycleExecutionPlanCalculator.java:135)
    at org.apache.maven.lifecycle.internal.builder.BuilderCommon.resolveBuildPlan (BuilderCommon.java:93)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:100)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:903)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:280)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:203)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
Warning:  Checksum validation failed, expected '<!DOCTYPE' (REMOTE_EXTERNAL) but is actually 'e0509cfe82a6ffad07497a39b82d6e0a82a5b1ac' from scala-tools.org for https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom
[INFO] Downloaded from scala-tools.org: https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom (23 kB at 30 kB/s)
Warning:  The POM for net.alchim31.maven:scala-maven-plugin:jar:4.9.1 is invalid, transitive dependencies (if any) will not be available: 1 problem was encountered while building the effective model
[FATAL] Non-parseable POM /home/runner/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom: attribute value must start with quotation or apostrophe not M (position: START_DOCUMENT seen <!DOCTYPE html><html lang="en"  data-adblockkey=M... @1:50)  @ line 1, column 50

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Parquet Java 1.15.0-SNAPSHOT:
[INFO] 
[INFO] Apache Parquet Java ................................ SUCCESS [  9.926 s]
[INFO] Apache Parquet Format Structures ................... SUCCESS [  7.546 s]
[INFO] Apache Parquet Generator ........................... SUCCESS [  1.785 s]
[INFO] Apache Parquet Common .............................. SUCCESS [  3.653 s]
[INFO] Apache Parquet Encodings ........................... SUCCESS [  9.611 s]
[INFO] Apache Parquet Column .............................. SUCCESS [ 11.506 s]
[INFO] Apache Parquet Arrow ............................... SUCCESS [  2.374 s]
[INFO] Apache Parquet Jackson ............................. SUCCESS [  1.731 s]
[INFO] Apache Parquet Hadoop .............................. SUCCESS [ 15.833 s]
[INFO] Apache Parquet Avro ................................ SUCCESS [  9.508 s]
[INFO] Apache Parquet Benchmarks .......................... SUCCESS [ 16.404 s]
[INFO] Apache Parquet Command-line ........................ SUCCESS [  9.864 s]
[INFO] Apache Parquet Pig ................................. SUCCESS [  7.967 s]
[INFO] Apache Parquet Pig Bundle .......................... SUCCESS [  2.244 s]
[INFO] Apache Parquet Protobuf ............................ SUCCESS [ 10.107 s]
[INFO] Apache Parquet Scala ............................... FAILURE [  0.791 s]
[INFO] Apache Parquet Thrift .............................. SKIPPED
[INFO] Apache Parquet Hadoop Bundle ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:01 min
[INFO] Finished at: 2024-11-17T02:48:05Z
[INFO] ------------------------------------------------------------------------
Error:  Plugin net.alchim31.maven:scala-maven-plugin:4.9.1 or one of its dependencies could not be resolved:
Error:  	1 problem was encountered while building the effective model
Error:  [FATAL] Non-parseable POM /home/runner/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom: attribute value must start with quotation or apostrophe not M (position: START_DOCUMENT seen <!DOCTYPE html><html lang="en"  data-adblockkey=M... @1:50)  @ line 1, column 50
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
Error: Process completed with exit code 1.

See: https://github.com/apache/parquet-java/actions/runs/11875285553/job/33092288845?pr=3064

Should we move forward with this PR?

@Fokko
Copy link
Contributor Author

Fokko commented Nov 18, 2024

It looks like the Scala version is not available anymore. I would be in favor of removing the module.

I think there are no objections on the dev list either: https://lists.apache.org/thread/scdq9t2gvvs4glhq0qx4qcvfp62j793s

wgtmac
wgtmac approved these changes Nov 18, 2024
View reviewed changes
Copy link
Member

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Perhaps we need to create an issue for this since it is not a minor one?

@Fokko Fokko changed the title Remove parquet-scala GH-3068: Remove parquet-scala Nov 18, 2024
@Fokko
Copy link
Contributor Author

Fokko commented Nov 18, 2024

@wgtmac Good point, I've added an issue 👍

@Fokko Fokko merged commit 1e04ec7 into apache:master Nov 20, 2024
9 checks passed
@Fokko
Copy link
Contributor Author

Fokko commented Nov 20, 2024

Let's move this forward to the CI works again, thanks for the review @pan3793 and @wgtmac 👍

@Fokko Fokko deleted the fd-remove-scala branch November 20, 2024 11:31
ConeyLiu pushed a commit to ConeyLiu/parquet-mr that referenced this pull request Nov 26, 2024
@Fokko @ConeyLiu
apacheGH-3068: Remove parquet-scala (apache#3063)
87aaac8
ConeyLiu pushed a commit to ConeyLiu/parquet-mr that referenced this pull request Nov 26, 2024
@Fokko @ConeyLiu
apacheGH-3068: Remove parquet-scala (apache#3063)
18e3aab
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pan3793 pan3793 pan3793 approved these changes

@wgtmac wgtmac wgtmac approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Remove parquet-scala from the project
3 participants
@Fokko @pan3793 @wgtmac