Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Maven-project] Incorporate maven-proto-plugin into build of Maven projects #535

Merged
merged 7 commits into from
Jul 10, 2024

Conversation

Thespica
Copy link
Contributor

@Thespica Thespica commented Jul 7, 2024

Reason for this PR

See #522

What changes are included in this PR?

Add maven-proto-plugin into pom.xml of java-info.

When the maven project is built, this plugin will compile protobuf into java source code into java-info/src/main/java/org/apache/graphar/proto.

Potential problem:

  1. Are generated-code located in java-info/src/main/java/org/apache/graphar/proto suitable?
  2. Once the code was generated, developer should remove the code manually to avoid compile error.

Are these changes tested?

No

Are there any user-facing changes?

@SemyonSinchenko
Copy link
Member

If protobuf is in maven build already let's remove it from buf.gen.yaml: https://github.com/apache/incubator-graphar/blob/main/buf.gen.yaml#L33

@@ -42,6 +42,9 @@
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<os.plugin.version>1.6.2</os.plugin.version>
<protobuf.plugin.version>0.6.1</protobuf.plugin.version>
<protobuf.version>3.21.1</protobuf.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest to pin the version not only for Maven builds but globally for the whole GraphAr. Otherwise we may face problem one day if we will try to serialize from java and, for example, read from cpp

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we can follow our doc and pin it at 1.32.0

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.32.0 is the version of buf, not protoc.

Th actual version of protobuf that we are using is 3.27.1
image

(explanation: we are on proto3 syntax and the version of protobuf plugin for a minor version of protoc itself)

I mean, I have a strong feeling, that we need to pin it globally. Otherwise it may become hard to maintain it in two different places (Maven and buf.gen.yaml)

Copy link
Contributor

@acezen acezen Jul 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.32.0 is the version of buf, not protoc.

Th actual version of protobuf that we are using is 3.27.1 image

(explanation: we are on proto3 syntax and the version of protobuf plugin for a minor version of protoc itself)

I mean, I have a strong feeling, that we need to pin it globally. Otherwise it may become hard to maintain it in two different places (Maven and buf.gen.yaml)

agree, we need to unify the protobuf version in project.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we need to decide which version of buf and protoc shoud we pin.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current protoc version is 3.27.1; at least we may pin it in Maven properties

If we are not going to use buf for generatign Java code, we do not need to touch buf version from this PR

</extensions>
<plugins>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this xolostice plugin is more popular at the moment, I like this one more: https://github.com/ascopes/protobuf-maven-plugin

I found it more human-friendly and easier to work with.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this plugin we do not need to have os-plugin

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plugin requires maven 3.8.2, which means we have to upgrade the requirement of maven version if we accept the newer plugin.

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would update the Maven. I see no reason to keep older version without a reason

@acezen
Copy link
Contributor

acezen commented Jul 8, 2024

Maybe we can put the location of generated code to maven-projects/src/main/java/org/apache/graphar/proto/? since we want the generated code can be a independent package like arrow java's flatbuf

@acezen
Copy link
Contributor

acezen commented Jul 8, 2024

Why it would raise compile error if not delete the generated code?

@Thespica
Copy link
Contributor Author

Thespica commented Jul 8, 2024

maven-projects/src/main/java/org/apache/graphar/proto/

With modifications, the code well generated at maven-projects/proto/src/main/java/org/apache/graphar/proto, and will be clean automatically when regenerating.

@SemyonSinchenko
Copy link
Member

Guys, why do we need both, buf-java plugin and maven-protobuf plugin? For me, if we do not want to store the code in the repository, let's remove java-generation from buf and leave only the maven plugin. Or did I miss something?

@acezen
Copy link
Contributor

acezen commented Jul 8, 2024

Guys, why do we need both, buf-java plugin and maven-protobuf plugin? For me, if we do not want to store the code in the repository, let's remove java-generation from buf and leave only the maven plugin. Or did I miss something?

hi, Sem, you are not miss anything. We do need leave only the maven plugin. @Thespica, could you delete the java-generation from buf if we use maven plugin?

@acezen
Copy link
Contributor

acezen commented Jul 9, 2024

maven-projects/src/main/java/org/apache/graphar/proto/

With modifications, the code well generated at maven-projects/proto/src/main/java/org/apache/graphar/proto, and will be clean automatically when regenerating.

Sorry for spell the path wrong, I suggest we can put the generated code to maven-projects/format/src/main/java/org/apache/graphar/proto, since it's the generated code for format.

@Thespica
Copy link
Contributor Author

Thespica commented Jul 9, 2024

maven-projects/src/main/java/org/apache/graphar/proto/

With modifications, the code well generated at maven-projects/proto/src/main/java/org/apache/graphar/proto, and will be clean automatically when regenerating.

Sorry for spell the path wrong, I suggest we can put the generated code to maven-projects/format/src/main/java/org/apache/graphar/proto, since it's the generated code for format.

I think it's okay to naming module as proto, since the package name is proto.

Copy link
Contributor

@acezen acezen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have checked the compilation. LGTM, can you add compile process to format CI?

@Thespica
Copy link
Contributor Author

Thespica commented Jul 9, 2024

Hi @SemyonSinchenko

I have transfered to the newer plugin, could you review again?

@SemyonSinchenko
Copy link
Member

It looks cool. Thank you!

I still have some concerns. It seems to me that we should use a maven-shade plugin to shade protobuf and rename generated classes... Otherwise, users may face dependency hell. But it is out of scoop for the current PR. Let's go step by step.

run: |
pushd maven-projects/proto
mvn clean compile
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Minor] why compile, not generate-sources?

@Thespica Thespica merged commit 98f3850 into apache:main Jul 10, 2024
2 checks passed
Elssky pushed a commit to Elssky/incubator-graphar that referenced this pull request Oct 8, 2024
…ojects (apache#535)

* Add proto-maven-plugin for java-info

* upgrade proto version adn patched optinal

* split proto into a single package

* modify proto java package name

* use ascopes maven plugin and delete java code gen in buf.gen.yaml

* Add ci for Java code-gen

* modify code-gen command
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants