Building Apache Ozone on Mac M3

[siddhantsangwan]

Using Rosetta (non native build)

Check if Rosetta is already installed:

/usr/bin/pgrep -q oahd && echo Yes || echo No

Install Rosetta:

softwareupdate --install-rosetta

Next, follow https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217392150 and then build Ozone with the usual mvn clean package -DskipTests -DskipShade.

Native Build

I found a way to make this work through a lot of debugging, googling, and experimentation. Note that I had already installed Rosetta at this point, so I'm not even 100% sure that this is actually a native build.

settings.xml (skip this if you hadn't used Rosetta earlier)

If you had edited your ~/.m2/settings.xml earlier for Rosetta, use a different .xml file for the native build. You can use the -s option of the mvn command to use a different .xml file while building.

For example, I have two .xml files in ~/.m2. The first is called settings.xml and I use this for the native build (the build uses this by default):

<repositories>
    <repository>
        <id>local</id>
        <url>file://${user.home}/.m2/repository</url>
    </repository>
</repositories>

The second is called s.xml and I use this with Rosetta:

<settings>
    <activeProfiles>
        <activeProfile>apple-silicon</activeProfile>
    </activeProfiles>
   
    <profiles>
        <profile>
            <id>apple-silicon</id>
            <properties>
                <os.detected.classifier>osx-x86_64</os.detected.classifier>
            </properties>
        </profile>
    </profiles>
</settings>

My build command for Rosetta is mvn clean package -DskipTests -DskipShade -s s.xml.

Change pom.xml files

Follow the commands in the 'Protobuf' section here https://gist.github.com/smengcl/6a269f44d909ef9245a9ad8c7c6d41e0/#protobuf. Next, apply the following diff. It changes the versions of some dependencies in various pom.xml files. Replace ~/.m2 with the path to your .m2 directory.

diff --git a/hadoop-hdds/interface-admin/pom.xml b/hadoop-hdds/interface-admin/pom.xml
index 8f38fb28db..27b0035954 100644
--- a/hadoop-hdds/interface-admin/pom.xml
+++ b/hadoop-hdds/interface-admin/pom.xml
@@ -63,9 +63,9 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
               <goal>test-compile</goal>
             </goals>
             <configuration>
-              <protocArtifact>
-                com.google.protobuf:protoc:${proto2.hadooprpc.protobuf.version}:exe:${os.detected.classifier}
-              </protocArtifact>
+              <protocExecutable>
+                ~/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-osx-aarch_64
+              </protocExecutable>
               <protoSourceRoot>${basedir}/src/main/proto/</protoSourceRoot>
               <outputDirectory>target/generated-sources/java</outputDirectory>
               <clearOutputDirectory>false</clearOutputDirectory>
diff --git a/hadoop-hdds/interface-client/pom.xml b/hadoop-hdds/interface-client/pom.xml
index 2160f7c5ed..5e89b203c9 100644
--- a/hadoop-hdds/interface-client/pom.xml
+++ b/hadoop-hdds/interface-client/pom.xml
@@ -96,9 +96,9 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
               <goal>test-compile</goal>
             </goals>
             <configuration>
-              <protocArtifact>
-                com.google.protobuf:protoc:${proto2.hadooprpc.protobuf.version}:exe:${os.detected.classifier}
-              </protocArtifact>
+              <protocExecutable>
+                ~/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-osx-aarch_64
+              </protocExecutable>
               <protoSourceRoot>${basedir}/src/main/proto/</protoSourceRoot>
               <includes>
                 <include>hdds.proto</include>
@@ -115,9 +115,9 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
               <goal>test-compile</goal>
             </goals>
             <configuration>
-              <protocArtifact>
-                com.google.protobuf:protoc:${proto3.hadooprpc.protobuf.version}:exe:${os.detected.classifier}
-              </protocArtifact>
+              <protocExecutable>
+                ~/.m2/repository/com/google/protobuf/protoc/3.7.1/protoc-3.7.1-osx-aarch_64
+              </protocExecutable>
               <protoSourceRoot>${basedir}/src/main/proto/</protoSourceRoot>
               <includes>
                 <include>hdds.proto</include>
diff --git a/hadoop-hdds/interface-server/pom.xml b/hadoop-hdds/interface-server/pom.xml
index b0e8d152b1..a97b8b24ce 100644
--- a/hadoop-hdds/interface-server/pom.xml
+++ b/hadoop-hdds/interface-server/pom.xml
@@ -89,9 +89,9 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
               <goal>test-compile</goal>
             </goals>
             <configuration>
-              <protocArtifact>
-                com.google.protobuf:protoc:${proto2.hadooprpc.protobuf.version}:exe:${os.detected.classifier}
-              </protocArtifact>
+              <protocExecutable>
+                ~/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-osx-aarch_64
+              </protocExecutable>
               <protoSourceRoot>${basedir}/src/main/proto/</protoSourceRoot>
               <excludes>
                 <exclude>InterSCMProtocol.proto</exclude>
diff --git a/hadoop-ozone/dist/pom.xml b/hadoop-ozone/dist/pom.xml
index 64fc2cfa37..3ec4b83b03 100644
--- a/hadoop-ozone/dist/pom.xml
+++ b/hadoop-ozone/dist/pom.xml
@@ -28,7 +28,7 @@
   <properties>
     <file.encoding>UTF-8</file.encoding>
     <downloadSources>true</downloadSources>
-    <docker.ozone-runner.version>20240316-jdk17-1</docker.ozone-runner.version>
+    <docker.ozone-runner.version>20230615-1</docker.ozone-runner.version>
     <docker.ozone-testkr5b.image>apache/ozone-testkrb5:20230318-1</docker.ozone-testkr5b.image>
     <maven.test.skip>true</maven.test.skip> <!-- no tests in this module so far -->
   </properties>
diff --git a/hadoop-ozone/interface-client/pom.xml b/hadoop-ozone/interface-client/pom.xml
index b92de2f5bc..44fbd129a0 100644
--- a/hadoop-ozone/interface-client/pom.xml
+++ b/hadoop-ozone/interface-client/pom.xml
@@ -85,9 +85,9 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
               <goal>test-compile-custom</goal>
             </goals>
             <configuration>
-              <protocArtifact>
-                com.google.protobuf:protoc:${proto2.hadooprpc.protobuf.version}:exe:${os.detected.classifier}
-              </protocArtifact>
+              <protocExecutable>
+                ~/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-osx-aarch_64
+              </protocExecutable>
               <protoSourceRoot>${basedir}/src/main/proto/</protoSourceRoot>
               <outputDirectory>target/generated-sources/protobuf/java</outputDirectory>
               <clearOutputDirectory>false</clearOutputDirectory>
@@ -105,9 +105,9 @@ https://maven.apache.org/xsd/maven-4.0.0.xsd">
             </goals>
             <configuration>
               <protoSourceRoot>${basedir}/src/main/proto/</protoSourceRoot>
-              <protocArtifact>
-                com.google.protobuf:protoc:${proto3.hadooprpc.protobuf.version}:exe:${os.detected.classifier}
-              </protocArtifact>
+              <protocExecutable>
+                ~/.m2/repository/com/google/protobuf/protoc/3.7.1/protoc-3.7.1-osx-aarch_64
+              </protocExecutable>
               <outputDirectory>target/generated-sources/protobuf/java/proto3</outputDirectory>
               <clearOutputDirectory>false</clearOutputDirectory>
             </configuration>
diff --git a/hadoop-ozone/interface-storage/pom.xml b/hadoop-ozone/interface-storage/pom.xml
index 8ec4e0d994..350b63ebc7 100644
--- a/hadoop-ozone/interface-storage/pom.xml
+++ b/hadoop-ozone/interface-storage/pom.xml
@@ -91,9 +91,9 @@
             </goals>
             <configuration>
               <protoSourceRoot>${basedir}/src/main/proto/</protoSourceRoot>
-              <protocArtifact>
-                com.google.protobuf:protoc:${proto2.hadooprpc.protobuf.version}:exe:${os.detected.classifier}
-              </protocArtifact>
+              <protocExecutable>
+                ~/.m2/repository/com/google/protobuf/protoc/2.5.0/protoc-2.5.0-osx-aarch_64
+              </protocExecutable>
             </configuration>
           </execution>
         </executions>
diff --git a/pom.xml b/pom.xml
index f1a910985f..f666aeccc6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -217,13 +217,13 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xs
 
     <!-- Maven protoc compiler -->
     <protobuf-maven-plugin.version>0.6.1</protobuf-maven-plugin.version>
-    <grpc.protobuf-compile.version>3.19.6</grpc.protobuf-compile.version>
+    <grpc.protobuf-compile.version>3.20.1</grpc.protobuf-compile.version>
     <os-maven-plugin.version>1.7.1</os-maven-plugin.version>
 
     <!-- Use netty version known to work with grpc-netty. See table: -->
     <!-- https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty -->
     <netty.version>4.1.109.Final</netty.version>
-    <io.grpc.version>1.58.0</io.grpc.version>
+    <io.grpc.version>1.63.0</io.grpc.version>
 
     <rocksdb.version>7.7.3</rocksdb.version>
     <sqlite.version>3.44.1.0</sqlite.version>