Provides built-in GraalVM Reachability Metadata for core features of …

…ShardingSphere JDBC and adds nativeTest-related unit test subsets
apache · Nov 14, 2023 · ecebd31 · ecebd31
1 parent d4379eb
commit ecebd31
Show file tree

Hide file tree

Showing 44 changed files with 4,274 additions and 104 deletions.
diff --git a/.github/workflows/nightly-build.yml b/.github/workflows/nightly-build.yml
@@ -136,7 +136,6 @@ jobs:
           password: ${{ secrets.GITHUB_TOKEN }}
       - name: Push Docker Image
         run: |
-          ./mvnw -PgenerateStandardMetadata -DskipNativeTests -B -T1C clean test
           ./mvnw -am -pl distribution/proxy-native -Prelease.native,docker.buildx.push.native -B -T1C -DskipTests -Dproxy.image.repository=${{ env.PROXY_NATIVE }} -Dproxy.image.tag=${{ github.sha }} clean package
 
   build-cache:

diff --git a/distribution/proxy-native/access-filter.json b/distribution/proxy-native/access-filter.json
diff --git a/...ent/content/user-manual/shardingsphere-proxy/startup/graalvm-native-image.cn.md b/...ent/content/user-manual/shardingsphere-proxy/startup/graalvm-native-image.cn.md
@@ -25,10 +25,6 @@ services:
       - "3307:3307"
 ```
 
-- 若你发现构建过程存在缺失的 GraalVM Reachability Metadata,
-  应当在 https://github.com/oracle/graalvm-reachability-metadata 打开新的 issue ，
-  并提交包含 ShardingSphere 自身或依赖的第三方库缺失的 GraalVM Reachability Metadata 的 PR。
-
 - ShardingSphere 的 master 分支尚未准备好处理 Native Image 中的单元测试 ， 你总是需要在构建 GraalVM Native Image 的过程中，
   加上特定于 `GraalVM Native Build Tools` 的 `-DskipNativeTests` 或 `-DskipTests` 参数跳过 Native Image 中的单元测试。
 
@@ -54,19 +50,13 @@ services:
    或 `GraalVM Community Edition` 的下游发行版。若使用 `SDKMAN!`，
 
 ```shell
-sdk install java 17.0.8-graalce
+sdk install java 17.0.9-graalce
 ```
 
 2. 根据 https://www.graalvm.org/jdk17/reference-manual/native-image/#prerequisites 的要求安装本地工具链。
 
 3. 如果需要构建 Docker Image， 确保 `docker-ce` 已安装。
 
-4. 首先需要在项目的根目录下，执行如下命令以为所有子模块采集 Standard 形态的 GraalVM 可达性元数据。
-
-```shell
-./mvnw -PgenerateStandardMetadata -DskipNativeTests -B -T1C clean test
-```
-
 ## 操作步骤
 
 1. 获取 Apache ShardingSphere Git Source
@@ -148,7 +138,7 @@ services:
   另请注意，某些第三方依赖将需要在 `Dockerfile` 安装更多系统库，例如 `libdl`。
   因此请确保根据你的使用情况调整 `distribution/proxy-native` 下的 `pom.xml` 和 `Dockerfile` 的内容。
 
-# 可观察性
+## 可观察性
 
 - 针对 GraalVM Native Image 形态的 ShardingSphere Proxy，其提供的可观察性的能力与
   https://shardingsphere.apache.org/document/current/cn/user-manual/shardingsphere-proxy/observability/ 并不一致。
@@ -160,7 +150,7 @@ services:
   https://github.com/oracle/graal/issues/5648 。
 
 - 对于使用 `ShardingSphere Agent` 等 APM Java Agent 的情形， GraalVM 的 `native-image` 组件尚未完全支持在构建 Native
-  Image 时使用 javaagent，你需要关注尚未关闭的 https://github.com/oracle/graal/issues/1065。
+  Image 时使用 javaagent，你需要关注尚未关闭的 https://github.com/oracle/graal/issues/1065 。
 
 - 以下部分采用 `Apache SkyWalking Java Agent` 作为示例，可用于跟踪 GraalVM 社区的对应 issue。
 
@@ -184,3 +174,58 @@ services:
 ```bash
 ./mvnw -am -pl distribution/proxy-native -B -T1C -Prelease.native -DskipTests clean package
 ```
+
+## 贡献 GraalVM Reachability Metadata
+
+ShardingSphere 对在 GraalVM Native Image 下的可用性的验证，是通过 GraalVM Native Build Tools 的 Maven Plugin 子项目来完成的。
+通过在 JVM 下运行单元测试，为单元测试打上 `junit-platform-unique-ids*` 标签，此后构建为 GraalVM Native Image 进行 nativeTest 来测试
+在 GraalVM Native Image 下的单元测试覆盖率。请不要使用 `io.kotest:kotest-runner-junit5-jvm:5.5.4` 等在 `test listener` mode 下
+failed to discover tests 的测试库。
+
+ShardingSphere 定义了 `shardingsphere-infra-nativetest` 的 Maven Module 用于为 native Test 提供小型的单元测试子集，
+此单元测试子集避免了使用 Mockito 等 native Test 下无法使用的第三方库。
+
+ShardingSphere 定义了 `nativeTestInShardingSphere` 的 Maven Profile 用于为 `shardingsphere-infra-nativetest` 模块执行 nativeTest 。
+
+假设贡献者处于新的 Ubuntu 22.04.3 LTS 实例下，其可通过如下 bash 命令通过 SDKMAN! 管理 JDK 和工具链，
+并为 `shardingsphere-infra-nativetest` 子模块执行 nativeTest。
+
+```bash
+sudo apt install unzip zip curl sed -y
+curl -s "https://get.sdkman.io" | bash
+source "$HOME/.sdkman/bin/sdkman-init.sh"
+sdk install java 17.0.9-graalce
+sdk use java 17.0.9-graalce
+sudo apt-get install build-essential libz-dev zlib1g-dev -y
+
+git clone [email protected]:apache/shardingsphere.git
+cd ./shardingsphere/
+./mvnw -PnativeTestInShardingSphere -T1C -e clean test
+```
+
+当贡献者发现缺少与 ShardingSphere 无关的第三方库的 GraalVM Reachability Metadata 时，应当在
+https://github.com/oracle/graalvm-reachability-metadata 打开新的 issue， 并提交包含依赖的第三方库缺失的 GraalVM Reachability
+Metadata 的 PR。ShardingSphere 在 `shardingsphere-infra-reachability-metadata` 子模块主动托管了部分第三方库的 GraalVM Reachability Metadata。
+
+如果 nativeTest 执行失败， 应为单元测试生成初步的 GraalVM Reachability Metadata，并手动调整以修复 nativeTest。
+如有需要，请使用 `org.junit.jupiter.api.condition.DisabledInNativeImage` 注解或 `org.graalvm.nativeimage.imagecode` 的
+System Property 屏蔽部分单元测试在 GraalVM Native Image 下运行。
+
+ShardingSphere 定义了 `generateMetadata` 的 Maven Profile 用于在普通 JVM 下携带 GraalVM Tracing Agent 执行单元测试，并在特定目录下生成或合并
+已有的 GraalVM Reachability Metadata 文件。可通过如下 bash 命令简单处理此流程。贡献者仍可能需要手动调整具体的 JSON 条目，并在适当的时候
+调整 Maven Profile 和 GraalVM Tracing Agent 的 Filter 链。
+
+以下命令仅为 `shardingsphere-infra-nativetest` 生成 Conditional 形态的 GraalVM Reachability Metadata 的一个举例。生成的 GraalVM 
+Reachability Metadata 位于 `shardingsphere-infra-reachability-metadata` 子模块下。
+
+对于测试类和测试文件独立使用的 GraalVM Reachability Metadata，贡献者应该放置到
+`${user.dir}/infra/nativetest/src/test/resources/META-INF/native-image/shardingsphere-infra-nativetest-test-metadata/` 
+文件夹下。`${}` 内为相关子模块对应的 POM 4.0 的常规系统变量，自行替换。
+
+```bash
+git clone [email protected]:apache/shardingsphere.git
+cd ./shardingsphere/
+./mvnw -PgenerateMetadata -DskipNativeTests -e -T1C clean test native:metadata-copy
+```
+
+请手动删除无任何具体条目的 JSON 文件。
diff --git a/...ent/content/user-manual/shardingsphere-proxy/startup/graalvm-native-image.en.md b/...ent/content/user-manual/shardingsphere-proxy/startup/graalvm-native-image.en.md
@@ -27,10 +27,6 @@ services:
       - "3307:3307"
 ````
 
-- If you find that the build process has missing GraalVM Reachability Metadata, a new issue should be opened
-  at https://github.com/oracle/graalvm-reachability-metadata, and submit a PR containing GraalVM Reachability Metadata
-  missing from ShardingSphere itself or dependent third-party libraries.
-
 - The master branch of ShardingSphere is not yet ready to handle unit tests in Native Image,
   you always need to build GraalVM Native Image in the process,
   Plus `-DskipNativeTests` or `-DskipTests` parameter specific to `GraalVM Native Build Tools` to skip unit tests in
@@ -62,20 +58,13 @@ services:
 JDK 17 according to https://www.graalvm.org/downloads/. If `SDKMAN!` is used,
 
 ```shell
-sdk install java 17.0.8-graalce
+sdk install java 17.0.9-graalce
 ```
 
 2. Install the local toolchain as required by https://www.graalvm.org/jdk17/reference-manual/native-image/#prerequisites.
 
 3. If you need to build a Docker Image, make sure `docker-ce` is installed.
 
-4. First, you need to execute the following command in the root directory of the project to collect the GraalVM
-   Reachability Metadata of the Standard form for all submodules.
-
-```shell
-./mvnw -PgenerateStandardMetadata -DskipNativeTests -B -T1C clean test
-```
-
 ## Steps
 
 1. Get Apache ShardingSphere Git Source
@@ -163,7 +152,7 @@ services:
   the `Dockerfile`. So make sure to tune `distribution/proxy-native` according to your usage `pom.xml` and `Dockerfile`
   below.
 
-# Observability
+## Observability
 
 - ShardingSphere for GraalVM Native Image form Proxy, which provides observability capabilities
   with https://shardingsphere.apache.org/document/current/cn/user-manual/shardingsphere-proxy/observability/

diff --git a/infra/nativetest/native-image-filter/extra-filter.json b/infra/nativetest/native-image-filter/extra-filter.json
@@ -0,0 +1,31 @@
+{
+  "rules": [
+    {"includeClasses": "**"},
+
+    {"excludeClasses": "com.sun.crypto.provider.**"},
+    {"excludeClasses": "com.sun.org.apache.xerces.internal.jaxp.**"},
+    {"excludeClasses": "java.**"},
+    {"includeClasses": "java.lang.Boolean"},
+    {"includeClasses": "java.lang.Thread"},
+    {"includeClasses": "java.util.Properties"},
+    {"excludeClasses": "javax.security.auth.x500.**"},
+    {"excludeClasses": "sun.misc.**"},
+    {"excludeClasses": "sun.security.**"},
+
+    {"excludeClasses": "com.github.benmanes.caffeine.cache.**"},
+    {"excludeClasses": "com.google.common.util.concurrent.**"},
+    {"excludeClasses": "com.sun.xml.bind.v2.**"},
+    {"excludeClasses": "com.zaxxer.hikari.**"},
+    {"excludeClasses": "javax.xml.bind.annotation.**"},
+    {"excludeClasses": "groovy.**"},
+    {"excludeClasses": "org.apache.calcite.**"},
+    {"excludeClasses": "org.codehaus.groovy.**"},
+    {"excludeClasses": "org.h2.**"},
+    {"excludeClasses": "org.locationtech.jts.geom.**"},
+    {"excludeClasses": "org.slf4j.event.**"},
+
+    {"excludeClasses": "org.apache.shardingsphere.infra.nativetest.**"}
+  ],
+  "regexRules": [
+  ]
+}
diff --git a/distribution/proxy-native/caller-filter.json → ...native-image-filter/user-code-filter.json b/distribution/proxy-native/caller-filter.json → ...native-image-filter/user-code-filter.json
diff --git a/infra/nativetest/pom.xml b/infra/nativetest/pom.xml
@@ -0,0 +1,67 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>org.apache.shardingsphere</groupId>
+        <artifactId>shardingsphere-infra</artifactId>
+        <version>5.4.2-SNAPSHOT</version>
+    </parent>
+    <artifactId>shardingsphere-infra-nativetest</artifactId>
+    <name>${project.artifactId}</name>
+
+    <properties>
+        <maven.deploy.skip>true</maven.deploy.skip>
+    </properties>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.shardingsphere</groupId>
+            <artifactId>shardingsphere-jdbc-core</artifactId>
+            <version>${project.version}</version>
+            <scope>test</scope>
+        </dependency>
+
+        <dependency>
+            <groupId>org.hamcrest</groupId>
+            <artifactId>hamcrest</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.zaxxer</groupId>
+            <artifactId>HikariCP</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.h2database</groupId>
+            <artifactId>h2</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.graalvm.buildtools</groupId>
+                <artifactId>native-maven-plugin</artifactId>
+                <version>${native-maven-plugin.version}</version>
+            </plugin>
+        </plugins>
+    </build>
+</project>
diff --git a/infra/nativetest/src/test/java/org/apache/shardingsphere/infra/nativetest/FileTestUtils.java b/infra/nativetest/src/test/java/org/apache/shardingsphere/infra/nativetest/FileTestUtils.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.shardingsphere.infra.nativetest;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.logging.Level;
+import java.util.logging.Logger;
+
+/**
+ * The background for this class comes from <a href="https://github.com/oracle/graal/issues/7682">oracle/graal#7682</a>
+ * and <a href="https://github.com/oracle/graal/blob/vm-ce-23.0.2/docs/reference-manual/native-image/Resources.md">Accessing Resources in Native Image</a>.
+ * GraalVM Native Image has special features in its handling of file systems.
+ * This means we are better off reading the file via `java.io.InputStream` instead of `java.net.URL` to avoid extra code
+ * processing.
+ *
+ * @see java.net.URL
+ * @see InputStream
+ */
+public class FileTestUtils {
+
+    /**
+     * read file From file URL string.
+     * @param fileUrl fileUrl
+     * @return byte array
+     */
+    public static byte[] readFromFileURLString(final String fileUrl) {
+        return readInputStream(ClassLoader.getSystemResourceAsStream(fileUrl)).getBytes(StandardCharsets.UTF_8);
+    }
+
+    private static String readInputStream(final InputStream is) {
+        StringBuilder out = new StringBuilder();
+        try (
+                InputStreamReader streamReader = new InputStreamReader(is, StandardCharsets.UTF_8);
+                BufferedReader reader = new BufferedReader(streamReader)) {
+            String line;
+            while ((line = reader.readLine()) != null) {
+                out.append(line);
+                // ShardingSphere does not actively handle line separators when parsing YAML and needs to be actively added.
+                out.append(System.lineSeparator());
+            }
+        } catch (IOException e) {
+            Logger.getLogger(FileTestUtils.class.getName()).log(Level.SEVERE, null, e);
+        }
+        return out.toString();
+    }
+}