Skip to content

A tiny project to create shaded Protobuf Java classes suitable for Spark's Protobuf connector

License

Notifications You must be signed in to change notification settings

rangadi/shaded-protobuf-classes

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

shaded-protobuf-classes

This project creates shaded Java classes for Protobuf files placed in src/main/protobuf directory. These classes are suitable for use with Protobuf functions in Spark.

Java class support in Spark Protobuf connector requires the classes to be shaded. Specifically references to com.google.protobuf.* classes should be rewritten to org.sparkproject.spark_protobuf.protobuf.*.

This project is useful to generate shaded classes in adhoc manner for a set of Protobuf files. Production use cases utilize build system support for generating Java classes and shading them. Common build tools like mvn, bazel, sbt, and others support shading.

How to create shaded Java classes for your Protobufs

  • Clone or download this repo
  • Add Protobuf files to src/main/protobuf directory
  • Run mvn clean package
    • Once the above is successful, target/shaded-protobuf-classes-1.0.jar contains shaded Java classes.

How to verify proper shading

The jar file can be inspected with a Java decompiler like JD-GUI. The generated classes should have import statements that look similar to

   import org.sparkproject.spark_protobuf.protobuf.AbstractMessage;

as shown in screenshot for AppEvent below:

java decompiler screenshot for AppEvent

About

A tiny project to create shaded Protobuf Java classes suitable for Spark's Protobuf connector

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published