java.lang.OutOfMemoryError: Required array length 2147483638 + 22 is too large #4809

spiritedRunning · 2025-07-08T07:50:23Z

spiritedRunning
Jul 8, 2025

Describe the bug

15:32:14 WARNING: File '/android-15.0.0_r1/external/expat/testdata/largefiles/aaaaaa_comment.xml' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:32:14 WARNING: No retry: '/home/zach/opengrok/src/android-15.0.0_r1/external/expat/testdata/largefiles/aaaaaa_comment.xml'
15:32:14 WARNING: File '/android-15.0.0_r1/out/build-aosp_cf_x86_64_auto.ninja' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:32:19 WARNING: File '/android-15.0.0_r1/hardware/qcom/sm8150/display/config/qdcm_calib_data_default.xml' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:32:24 WARNING: File '/android-15.0.0_r1/hardware/qcom/sm7250/display/config/qdcm_calib_data_sw43404_amoled_cmd_mode_dsi_boe_panel_with_DSC.xml' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:32:24 WARNING: No retry: '/home/zach/opengrok/src/android-15.0.0_r1/hardware/qcom/sm7250/display/config/qdcm_calib_data_sw43404_amoled_cmd_mode_dsi_boe_panel_with_DSC.xml'
15:32:24 WARNING: Error from ctags: ctags: Warning: Unknown language "Yaml:+.yaml.Yaml.YAML" in "langmap" option
15:32:24 WARNING: Error from ctags: ctags: Warning: Unknown language "Yaml:+.yml.Yml.YML" in "langmap" option
15:32:32 WARNING: File '/android-15.0.0_r1/out/soong/late-aosp_cf_x86_64_tv.mk' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:32:43 WARNING: File '/android-15.0.0_r1/hardware/qcom/sm8150/display/config/qdcm_calib_data_sw43404_amoled_cmd_mode_dsi_boe_panel_with_DSC.xml' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:32:43 WARNING: No retry: '/home/zach/opengrok/src/android-15.0.0_r1/hardware/qcom/sm8150/display/config/qdcm_calib_data_sw43404_amoled_cmd_mode_dsi_boe_panel_with_DSC.xml'
15:32:44 WARNING: File '/android-15.0.0_r1/out/build-aosp_cf_x86_64_auto.ninja' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:32:44 WARNING: No retry: '/home/zach/opengrok/src/android-15.0.0_r1/out/build-aosp_cf_x86_64_auto.ninja'
15:32:49 WARNING: File '/android-15.0.0_r1/hardware/qcom/sm8150/display/config/qdcm_calib_data_default.xml' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:32:49 WARNING: No retry: '/home/zach/opengrok/src/android-15.0.0_r1/hardware/qcom/sm8150/display/config/qdcm_calib_data_default.xml'
15:32:51 WARNING: File '/android-15.0.0_r1/out/build-aosp_x86_64-cts.ninja' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:32:51 WARNING: Error from ctags: ctags: Warning: Unknown language "Yaml:+.yaml.Yaml.YAML" in "langmap" option
15:32:51 WARNING: Error from ctags: ctags: Warning: Unknown language "Yaml:+.yml.Yml.YML" in "langmap" option
15:32:54 WARNING: File '/android-15.0.0_r1/out/soong/build.aosp_cf_x86_64_tv.ninja' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:33:04 WARNING: File '/android-15.0.0_r1/out/soong/late-aosp_cf_x86_64_tv.mk' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:33:04 WARNING: No retry: '/home/zach/opengrok/src/android-15.0.0_r1/out/soong/late-aosp_cf_x86_64_tv.mk'
15:33:21 WARNING: File '/android-15.0.0_r1/out/build-aosp_x86_64-cts.ninja' interrupted--failed to generate xref :java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException
15:33:21 WARNING: No retry: '/home/zach/opengrok/src/android-15.0.0_r1/out/build-aosp_x86_64-cts.ninja'
15:33:44 SEVERE: 811 successes (94.9%) after aborting parallel-indexing
15:33:45 SEVERE: Problem updating index database in directory '/home/zach/aosp/opengrok/data/index/android-15.0.0_r1': 
org.opengrok.indexer.index.IndexerException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Required array length 2147483638 + 22 is too large
	at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1908)
	at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:666)
	at org.opengrok.indexer.index.IndexDatabase.lambda$updateAll$5(IndexDatabase.java:346)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Required array length 2147483638 + 22 is too large
	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1900)
	... 7 more
Caused by: java.lang.OutOfMemoryError: Required array length 2147483638 + 22 is too large
	at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
	at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
	at java.base/java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:257)
	at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:229)
	at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582)
	at java.base/java.lang.StringBuffer.append(StringBuffer.java:313)
	at java.base/java.io.StringWriter.write(StringWriter.java:106)
	at org.opengrok.indexer.analysis.executables.JavaClassAnalyzer.getContent(JavaClassAnalyzer.java:312)
	at org.opengrok.indexer.analysis.executables.JavaClassAnalyzer.analyze(JavaClassAnalyzer.java:139)
	at org.opengrok.indexer.analysis.executables.JarAnalyzer.analyze(JarAnalyzer.java:101)
	at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:632)
	at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:1190)
	at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$8(IndexDatabase.java:1863)
	... 4 more

15:33:45 SEVERE: Indexer failed with IndexerException

To Reproduce
java -Djava.util.logging.config.file=opengrok/etc/logging.properties -jar opengrok/dist/lib/opengrok.jar -c /usr/local/bin/ctags -s opengrok/src/ -d aosp/opengrok/data/ -P -S -m 256 --connectTimeout 3 -W etc/configuration.xml -U http://localhost:8080/source

ctags
Universal Ctags 6.2.0(p6.2.20250615.0), Copyright (C) 2015-2025 Universal Ctags Team
Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
Compiled: Jul 6 2025, 14:50:11
URL: https://ctags.io/
Output version: 1.1
Optional compiled features: +wildcards, +regex, +gnulib_regex, +iconv, +option-directory, +xpath, +packcc, +optscript, +pcre2

Host Info
Linux ubuntu-20 5.15.0-139-generic #149~20.04.1-Ubuntu SMP Wed Apr 16 08:29:56 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
VERSION="20.04.6 LTS (Focal Fossa)"

Answered by vladak

Jul 8, 2025

The indexer seems to be running with the default Java heap size. The size is listed in the indexer logs near the beginning, unfortunately the logs are trimmed.

AOSP is certainly considered a large code base that would require tuning, see https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases and note similar tuning ought to be done for the webapp as well.

Also, in addition to the tuning consider ignoring potentially huge files with the -i option.

View full answer

vladak · 2025-07-08T08:52:10Z

vladak
Jul 8, 2025
Maintainer

The indexer seems to be running with the default Java heap size. The size is listed in the indexer logs near the beginning, unfortunately the logs are trimmed.

AOSP is certainly considered a large code base that would require tuning, see https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases and note similar tuning ought to be done for the webapp as well.

Also, in addition to the tuning consider ignoring potentially huge files with the -i option.

0 replies

spiritedRunning · 2025-07-08T09:28:09Z

spiritedRunning
Jul 8, 2025
Author

I scroll to the very beginning, seems no java heap size info printed.

zach@ubuntu-20:~$ java -Djava.util.logging.config.file=opengrok/etc/logging.properties -jar opengrok/dist/lib/opengrok.jar -c /usr/local/bin/ctags -s opengrok/src/ -d aosp/opengrok/data/ -P -S -m 256 --connectTimeout 3 -W etc/configuration.xml -U http://localhost:8080/source
15:15:03 WARNING: Error from ctags: ctags: Warning: Unknown language "Yaml:+.yaml.Yaml.YAML" in "langmap" option
15:15:03 WARNING: Error from ctags: ctags: Warning: Unknown language "Yaml:+.yml.Yml.YML" in "langmap" option
SLF4J(W): No SLF4J providers were found.
SLF4J(W): Defaulting to no-operation (NOP) logger implementation
SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
15:15:10 WARNING: Index in '/home/zach/aosp/opengrok/data/index/android-15.0.0_r1' is dirty, the last indexing was likely interrupted. It might be worthwhile to reindex from scratch.
15:15:22 WARNING: cannot get cached file for file '/kernel-android14-6.1/build/kernel/build-tools/path/linux-x86/avbtool' - the cache entry will not be cleared
org.opengrok.indexer.history.CacheException: Failed to get path relative to source root for '/home/zach/opengrok/src/kernel-android14-6.1/build/kernel/build-tools/path/linux-x86/avbtool'
	at org.opengrok.indexer.history.AbstractCache.getCachedFile(AbstractCache.java:75)
	at org.opengrok.indexer.history.AbstractCache.clearFile(AbstractCache.java:124)
	at org.opengrok.indexer.history.HistoryGuru.clearAnnotationCacheFile(HistoryGuru.java:1255)
	at org.opengrok.indexer.index.IndexDatabase.removeAnnotationFile(IndexDatabase.java:1081)
	at org.opengrok.indexer.index.IndexDatabase.removeFile(IndexDatabase.java:1109)
	at org.opengrok.indexer.index.IndexDatabase.processFile(IndexDatabase.java:1814)
	at org.opengrok.indexer.index.IndexDatabase.indexDown(IndexDatabase.java:1638)
	at org.opengrok.indexer.index.IndexDatabase.indexDown(IndexDatabase.java:1636)
	at org.opengrok.indexer.index.IndexDatabase.indexDown(IndexDatabase.java:1636)
	at org.opengrok.indexer.index.IndexDatabase.indexDown(IndexDatabase.java:1636)
	at org.opengrok.indexer.index.IndexDatabase.indexDown(IndexDatabase.java:1636)
	at org.opengrok.indexer.index.IndexDatabase.indexDown(IndexDatabase.java:1636)
	at org.opengrok.indexer.index.IndexDatabase.getIndexDownArgs(IndexDatabase.java:849)
	at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:657)
	at org.opengrok.indexer.index.IndexDatabase.lambda$updateAll$5(IndexDatabase.java:346)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.opengrok.indexer.util.ForbiddenSymlinkException: '/home/zach/opengrok/src/kernel-android14-6.1/build/kernel/build-tools/path/linux-x86/avbtool' is prohibited symlink
	at org.opengrok.indexer.util.PathUtils.getRelativeToCanonical(PathUtils.java:145)
	at org.opengrok.indexer.configuration.RuntimeEnvironment.getPathRelativeToSourceRoot(RuntimeEnvironment.java:491)
	at org.opengrok.indexer.history.AbstractCache.getCachedFile(AbstractCache.java:69)
	... 19 more

Thanks for the tuning guide, I'll look through it.

1 reply

vladak Jul 8, 2025
Maintainer

Depends on the logging setup done in the logging.properties file. It might be that only some messages get printed to the console and the rest of the messages go somewhere else. Without seeing the contents of the file I cannot tell.

Also, the Java information was added to the recent versions of OpenGrok. You did not say which OpenGrok version you are running.

spiritedRunning · 2025-07-08T10:18:43Z

spiritedRunning
Jul 8, 2025
Author

My logging file as following, do I need tune something to print logs on console?

# default file output is in user's home directory.
#java.util.logging.FileHandler.pattern = %hjava%u.log
java.util.logging.FileHandler.pattern = opengrok%g.%u.log
java.util.logging.FileHandler.append = false
java.util.logging.FileHandler.limit = 0 
java.util.logging.FileHandler.count = 30
java.util.logging.FileHandler.level = ALL 
#java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter
java.util.logging.FileHandler.formatter = org.opengrok.indexer.logger.formatter.SimpleFileLogFormatter

# Limit the message that are printed on the console to WARNING and above (default "quiet" mode).
# if verbose is set, then log INFO and above
java.util.logging.ConsoleHandler.level = WARNING
java.util.logging.ConsoleHandler.formatter = org.opengrok.indexer.logger.formatter.SimpleConsoleFormatter

# For example, set the com.xyz.foo logger to only log SEVERE
# messages:
#com.xyz.foo.level = SEVERE
org.opengrok.level = ALL

BTW, I use the latest opengrok 1.14.1

1 reply

vladak Jul 8, 2025
Maintainer

The java.util.logging.ConsoleHandler.level = WARNING line will limit the level to WARNING and higher, the Java log messages are logged with INFO, I believe, which is below WARNING, hence these do not appear.

spiritedRunning · 2025-07-08T13:23:54Z

spiritedRunning
Jul 8, 2025
Author

After change the level of log, does it means the java heap size is 19G?

21:21:50 INFO: Indexer options: [-m, 256, -c, /usr/local/bin/ctags, -s, opengrok/src, -i, d:prebuilt, -d, aosp/opengrok/data, -P, -S, -W, opengrok/etc/configuration.xml, -U, http://localhost:8080/source]
21:21:50 INFO: History based reindex is on, however history is off. History has to be enabled for history based reindex.
21:21:51 INFO: Indexer version 1.14.1 (0010bd7fd0bbdce260f4e517d9817fdcb232c9a4) running on Java version: 17.0.12+8-LTS-286, name: Java HotSpot(TM) 64-Bit Server VM, vendor: Oracle Corporation, arch: amd64 with properties: ncpu: 6, maxMemory: 19.6 GiB
21:21:51 WARNING: Error from ctags: ctags: Warning: Unknown language "Yaml:+.yaml.Yaml.YAML" in "langmap" option
21:21:51 WARNING: Error from ctags: ctags: Warning: Unknown language "Yaml:+.yml.Yml.YML" in "langmap" option
21:21:51 INFO: Using ctags: Universal Ctags 6.2.0(p6.2.20250615.0), Copyright (C) 2015-2025 Universal Ctags Team

Universal Ctags is derived from Exuberant Ctags.
Exuberant Ctags 5.8, Copyright (C) 1996-2009 Darren Hiebert
  Compiled: Jul  6 2025, 14:50:11
  URL: https://ctags.io/
  Output version: 1.1
  Optional compiled features: +wildcards, +regex, +gnulib_regex, +iconv, +option-directory, +xpath, +packcc, +optscript, +pcre2

21:21:51 INFO: Scanning for repositories in [/home/zach/opengrok/src/kernel-android14-5.15, /home/zach/opengrok/src/kernel-android13-5.15, /home/zach/opengrok/src/android-15.0.0_r1, /home/zach/opengrok/src/kernel-android15-6.6, /home/zach/opengrok/src/kernel-android14-6.1] (down to 3 levels below source root)

After filter some folders, and adjust java heap size to 32G, it works fine. @vladak Thanks for your guide!

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

java.lang.OutOfMemoryError: Required array length 2147483638 + 22 is too large #4809

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

java.lang.OutOfMemoryError: Required array length 2147483638 + 22 is too large #4809

Uh oh!

spiritedRunning Jul 8, 2025

Replies: 4 comments · 2 replies

Uh oh!

vladak Jul 8, 2025 Maintainer

Uh oh!

spiritedRunning Jul 8, 2025 Author

Uh oh!

vladak Jul 8, 2025 Maintainer

Uh oh!

spiritedRunning Jul 8, 2025 Author

Uh oh!

Uh oh!

vladak Jul 8, 2025 Maintainer

Uh oh!

Uh oh!

spiritedRunning Jul 8, 2025 Author

spiritedRunning
Jul 8, 2025

Replies: 4 comments 2 replies

vladak
Jul 8, 2025
Maintainer

spiritedRunning
Jul 8, 2025
Author

vladak Jul 8, 2025
Maintainer

spiritedRunning
Jul 8, 2025
Author

vladak Jul 8, 2025
Maintainer

spiritedRunning
Jul 8, 2025
Author