-
Notifications
You must be signed in to change notification settings - Fork 6.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provides built-in GraalVM Reachability Metadata for core features of ShardingSphere JDBC and adds nativeTest-related unit test subsets #29000
Conversation
1fe94c5
to
c0d6d1d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-
Update in 2023.11.12: maybe fixes in Open the
resource:/
related FileSystem for JDBCRepositorySQLLoader in the GraalVM Native Image environment #29009 . -
This PR is still pending. Some classes of ShardingSphere need to be reconstructed for file system-related operations, which I hope to handle in another PR. For example
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader
. -
The current execution of nativeTest using
./mvnw -PnativeTestInShardingSphere -T1C -e clean test
will cause the following Error. This is caused by GraalVM's inherent design of the file system, which involves the background of Cannot open native image resource using Path, NativeImageResourceFileSystemProvider not initialised oracle/graal#7682 .
Error Log
[2/8] Performing analysis... [********] (148.8s @ 2.93GB)
29,136 (81.15%) of 35,902 types reachable
49,445 (57.37%) of 86,186 fields reachable
149,418 (58.31%) of 256,235 methods reachable
7,787 types, 979 fields, and 6,877 methods registered for reflection
68 types, 82 fields, and 57 methods registered for JNI access
4 native libraries: dl, pthread, rt, z
[3/8] Building universe... (16.4s @ 3.28GB)
[4/8] Parsing methods... [***] (9.3s @ 2.32GB)
[5/8] Inlining methods... [****] (4.0s @ 2.39GB)
[6/8] Compiling methods... [********] (67.8s @ 2.69GB)
[7/8] Layouting methods... [***] (10.5s @ 2.88GB)
[8/8] Creating image... [****] (15.9s @ 3.58GB)
73.72MB (54.04%) for code area: 94,894 compilation units
57.31MB (42.01%) for image heap: 576,269 objects and 277 resources
5.38MB ( 3.95%) for other data
136.41MB in total
------------------------------------------------------------------------------------------------------------------------
Top 10 origins of code area: Top 10 object types in image heap:
13.33MB calcite-core-1.35.0.jar 16.03MB byte[] for code metadata
13.17MB java.base 7.42MB byte[] for java.lang.String
5.79MB h2-2.2.224.jar 7.18MB java.lang.Class
5.66MB java.xml 5.80MB java.lang.String
5.29MB groovy-4.0.10.jar 4.52MB byte[] for general heap data
3.89MB org.apache.shardingsphere.sql.parser.autogen 2.45MB com.oracle.svm.core.hub.DynamicHubCompanion
2.96MB java.desktop 2.12MB byte[] for embedded resources
1.91MB jackson-databind-2.14.0.jar 1.74MB byte[] for reflection metadata
1.54MB avatica-core-1.23.0.jar 1.22MB java.lang.String[]
1.43MB janino-3.1.9.jar 1.05MB c.o.svm.core.hub.DynamicHub$ReflectionMetadata
18.06MB for 727 more packages 7.56MB for 5238 more object types
------------------------------------------------------------------------------------------------------------------------
Recommendations:
AWT: Use the tracing agent to collect metadata for AWT.
HEAP: Set max heap for improved and more predictable memory usage.
CPU: Enable more CPU features with '-march=native' for improved performance.
------------------------------------------------------------------------------------------------------------------------
36.5s (12.0% of total time) in 181 GCs | Peak RSS: 5.27GB | CPU load: 4.20
------------------------------------------------------------------------------------------------------------------------
Produced artifacts:
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libawt.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libawt_headless.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libawt_xawt.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libfontmanager.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libjava.so (jdk_library_shim)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libjavajpeg.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/libjvm.so (jdk_library_shim)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/liblcms.so (jdk_library)
/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/native-tests (executable)
========================================================================================================================
Finished generating 'native-tests' in 5m 1s.
[INFO] Executing: /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/native-tests --xml-output-dir /home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/native-test-reports -Djunit.platform.listeners.uid.tracking.output.dir=/home/linghengqian/TwinklingLiftWorks/git/public/shardingsphere/infra/nativetest/target/test-ids
JUnit Platform on Native Image - report
----------------------------------------
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - This is logback-classic version 1.3.11
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - No custom configurators were discovered as a service.
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Trying to configure with ch.qos.logback.classic.joran.SerializedModelConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Constructed configurator of type class ch.qos.logback.classic.joran.SerializedModelConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Could NOT find resource [logback-test.scmo]
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Could NOT find resource [logback.scmo]
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - ch.qos.logback.classic.joran.SerializedModelConfigurator.configure() call lasted 0 milliseconds. ExecutionStatus=INVOKE_NEXT_IF_ANY
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Trying to configure with ch.qos.logback.classic.util.DefaultJoranConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - Constructed configurator of type class ch.qos.logback.classic.util.DefaultJoranConfigurator
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Could NOT find resource [logback-test.xml]
18:08:36,098 |-INFO in ch.qos.logback.classic.LoggerContext[shardingsphere-infra-nativetest] - Found resource [logback.xml] at [resource:/logback.xml]
18:08:36,098 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@6b3ecc43 - Resource [logback.xml] occurs multiple times on the classpath.
18:08:36,098 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@6b3ecc43 - Resource [logback.xml] occurs at [resource:/logback.xml#1]
18:08:36,098 |-WARN in ch.qos.logback.classic.util.DefaultJoranConfigurator@6b3ecc43 - Resource [logback.xml] occurs at [resource:/logback.xml]
18:08:36,098 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@3f7d7df1 - URL [resource:/logback.xml] is not of type file
18:08:36,099 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - <level> element is deprecated. Near [level] on line 31
18:08:36,099 |-WARN in ch.qos.logback.classic.joran.action.LevelAction - Please use "level" attribute within <logger> or <root> elements instead.
18:08:36,099 |-INFO in ch.qos.logback.classic.model.processor.ContextNameModelHandler - Setting logger context name as [shardingsphere-infra-nativetest]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
18:08:36,099 |-INFO in ch.qos.logback.classic.model.processor.LevelModelHandler - ROOT level set to INFO
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
18:08:36,099 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@12459199 - End of configuration.
18:08:36,099 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@85c50cc - Registering current configuration as safe fallback point
18:08:36,099 |-INFO in ch.qos.logback.classic.util.ContextInitializer@2f4b8fab - ch.qos.logback.classic.util.DefaultJoranConfigurator.configure() call lasted 1 milliseconds. ExecutionStatus=DO_NOT_INVOKE_NEXT_IF_ANY
org.apache.shardingsphere.infra.nativetest.jdbc.features.ShardingTest > testShardingInLocalTransactions() FAILED
org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest > testReadWriteSplittingInLocalTransactions() FAILED
org.apache.shardingsphere.infra.nativetest.jdbc.features.EncryptTest > testEncryptInLocalTransactions() FAILED
org.apache.shardingsphere.infra.nativetest.jdbc.features.MaskTest > testMaskInLocalTransactions() FAILED
org.apache.shardingsphere.infra.nativetest.jdbc.features.ShadowTest > testShadowInLocalTransactions() FAILED
Failures (5):
JUnit Jupiter:ShardingTest:testShardingInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.ShardingTest', methodName = 'testShardingInLocalTransactions', methodParameterTypes = '']
=> java.nio.file.FileSystemNotFoundException
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
[email protected]/java.nio.file.Path.of(Path.java:208)
[email protected]/java.nio.file.Paths.get(Paths.java:98)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
[...]
JUnit Jupiter:ReadWriteSplittingTest:testReadWriteSplittingInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest', methodName = 'testReadWriteSplittingInLocalTransactions', methodParameterTypes = '']
=> java.lang.NullPointerException
[email protected]/java.io.Reader.<init>(Reader.java:168)
[email protected]/java.io.InputStreamReader.<init>(InputStreamReader.java:112)
org.apache.shardingsphere.infra.nativetest.FileTestUtils.readInputStream(FileTestUtils.java:52)
org.apache.shardingsphere.infra.nativetest.FileTestUtils.readFromFileURLString(FileTestUtils.java:46)
org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest.testReadWriteSplittingInLocalTransactions(ReadWriteSplittingTest.java:53)
[email protected]/java.lang.reflect.Method.invoke(Method.java:568)
[email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
[email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
JUnit Jupiter:EncryptTest:testEncryptInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.EncryptTest', methodName = 'testEncryptInLocalTransactions', methodParameterTypes = '']
=> java.nio.file.FileSystemNotFoundException
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
[email protected]/java.nio.file.Path.of(Path.java:208)
[email protected]/java.nio.file.Paths.get(Paths.java:98)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
[...]
JUnit Jupiter:MaskTest:testMaskInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.MaskTest', methodName = 'testMaskInLocalTransactions', methodParameterTypes = '']
=> java.nio.file.FileSystemNotFoundException
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
[email protected]/java.nio.file.Path.of(Path.java:208)
[email protected]/java.nio.file.Paths.get(Paths.java:98)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
[...]
JUnit Jupiter:ShadowTest:testShadowInLocalTransactions()
MethodSource [className = 'org.apache.shardingsphere.infra.nativetest.jdbc.features.ShadowTest', methodName = 'testShadowInLocalTransactions', methodParameterTypes = '']
=> java.nio.file.FileSystemNotFoundException
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getFileSystem(NativeImageResourceFileSystemProvider.java:144)
org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk.resources.NativeImageResourceFileSystemProvider.getPath(NativeImageResourceFileSystemProvider.java:154)
[email protected]/java.nio.file.Path.of(Path.java:208)
[email protected]/java.nio.file.Paths.get(Paths.java:98)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.loadFromDirectory(JDBCRepositorySQLLoader.java:82)
org.apache.shardingsphere.mode.repository.standalone.jdbc.sql.JDBCRepositorySQLLoader.load(JDBCRepositorySQLLoader.java:72)
org.apache.shardingsphere.mode.repository.standalone.jdbc.JDBCRepository.init(JDBCRepository.java:59)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.findService(TypedSPILoader.java:61)
org.apache.shardingsphere.infra.spi.type.typed.TypedSPILoader.getService(TypedSPILoader.java:110)
org.apache.shardingsphere.mode.manager.standalone.NewStandaloneContextManagerBuilder.build(NewStandaloneContextManagerBuilder.java:48)
[...]
Test run finished after 98 ms
[ 6 containers found ]
[ 0 containers skipped ]
[ 6 containers started ]
[ 0 containers aborted ]
[ 6 containers successful ]
[ 0 containers failed ]
[ 5 tests found ]
[ 0 tests skipped ]
[ 5 tests started ]
[ 0 tests aborted ]
[ 0 tests successful ]
[ 5 tests failed ]
c9aac11
to
09dcdc8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
09dcdc8
to
c8e5545
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
c8e5545
to
7427de4
Compare
ecebd31
to
65e3ae0
Compare
…ShardingSphere JDBC and adds nativeTest-related unit test subsets
65e3ae0
to
138fb68
Compare
...a/org/apache/shardingsphere/infra/nativetest/jdbc/features/repository/AddressRepository.java
Show resolved
Hide resolved
...test/src/test/java/org/apache/shardingsphere/infra/nativetest/jdbc/features/EncryptTest.java
Outdated
Show resolved
Hide resolved
...test/src/test/java/org/apache/shardingsphere/infra/nativetest/jdbc/features/EncryptTest.java
Outdated
Show resolved
Hide resolved
...ivetest/src/test/java/org/apache/shardingsphere/infra/nativetest/jdbc/features/MaskTest.java
Outdated
Show resolved
Hide resolved
...st/java/org/apache/shardingsphere/infra/nativetest/jdbc/features/ReadWriteSplittingTest.java
Outdated
Show resolved
Hide resolved
...st/java/org/apache/shardingsphere/infra/nativetest/jdbc/features/ReadWriteSplittingTest.java
Outdated
Show resolved
Hide resolved
...st/java/org/apache/shardingsphere/infra/nativetest/jdbc/features/ReadWriteSplittingTest.java
Outdated
Show resolved
Hide resolved
...etest/src/test/java/org/apache/shardingsphere/infra/nativetest/jdbc/features/ShadowTest.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for your perfect pr. @linghengqian Merged.
Fixes #21347.
Changes proposed in this pull request:
org.apache.shardingsphere:shardingsphere-infra-reachability-metadata
for hosting the GraalVM Reachability Metadata of ShardingSphere itself and some third-party libraries. This involves me personally not having much time in the central repository of GraalVM Reachability Metadata to handle PRs requested by the GraalVM Committer.org.apache.shardingsphere:shardingsphere-infra-reachability-metadata
will become an internal dependency oforg.apache.shardingsphere:shardingsphere-jdbc-core
to be passed out.org.apache.shardingsphere:shardingsphere-infra-nativetest
.org.apache.shardingsphere:shardingsphere-infra-nativetest
is adapted from the ShardingSphere Example. Use YAML instead of Java API to define ShardingSphere's DataSource because this is closest to the actual application scenario - this is the actual meaning of native Test. We only focus on core features in these unit test.org.apache.shardingsphere.infra.nativetest.jdbc.features.ReadWriteSplittingTest
seems to have hit a bug in ReadWrite Splitting. I will discuss the phenomenon I saw in a separate issue later.generateMetadata
andnativeTestInShardingSphere
. They both only managepluginManagement
. Submodules that need to use the relevant Profile will automatically introduce the relevantplugin
. This involves best practices defined by GraalVM Native Build Tools. Refer to Maven Native plugin fails on POM packaging projects graalvm/native-build-tools#490 .Before committing this PR, I'm sure that I have checked the following options:
./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e
.