AOSP Indexing take long time

[JustSong]

Version:

• Tomcat: apache-tomcat-8.5.37.tar.gz
• Opengrok: opengrok-1.3.10.tar.gz
• Ctags: Exuberant Ctags 5.8

Index command:

java -Djava.util.logging.config.file=$OPENGROK_LOG_FILE_PATH \
     -jar $OPENGROK_PROJECT_PATH/$OPENGROK_VERSION/lib/opengrok.jar \
     -s ${OPENGROK_SRC_ROOT} \
     -d ${OPENGROK_DATA_ROOT} \
     -c ${OPENGROK_CTAGS} \
     -W ${OPENGROK_XML_CONFIGURATION} \
     -U ${OPENGROK_WEBAPP_CFGADDR} \
     --depth 100 \
     -S -H -P -G --threads 8

When I index a large size source code (AOSP-Android10), it takes almost 24 hours to complete.
I have checked the log and find it will take so long time after every "Done optimizing index for project".

Example Log:

2022-09-12 17:16:53.439+0800 FINE t15307945 PendingFileCompleter.complete: renamed 42 file(s) (took 0 seconds)
2022-09-12 17:16:53.439+0800 FINE t15307945 PendingFileCompleter.complete: affirmed links for 0 path(s) (took 0 seconds)
2022-09-12 17:16:53.439+0800 FINE t15307945 IndexDatabase.finishWriting: completed 43 object(s)
2022-09-12 17:16:53.464+0800 INFO t15307945 IndexDatabase.optimize: Optimizing the index for project device
2022-09-12 17:16:57.568+0800 INFO t15307945 Statistics.report: Done optimizing index for project device (took 4.104 seconds)
2022-09-12 18:46:21.227+0800 FINE t15307944 HttpURLConnection.getInputStream0: sun.net.www.MessageHeader@748051322 pairs: {null: HTTP/1.1 204}{Date: Mon, 12 Sep 2022 10:46:21 GMT}

Question 1: The index time: 24 hours, is this normal index time?

Question 2: Is there any solution to speed up the AOSP indexing time?
EX: Do we need to set a special setting when we index AOSP source?

[vladak]

Thanks to the indexer being heavily parallelized, the indexing time depends very much on the underlying hardware and the resources assigned to the indexer's JVM.

Some pieces of advice:

• use the latest OpenGrok version
• increase thread count (assuming the machine has ample CPU power)
• increase Java heap (assuming enough real RAM)

Also check out https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases

[vladak]

As for the 1.5 hr delay between IndexDatabase.optimize() (that has been removed in the latest OpenGrok release) and the configuration being sent to the webapp, this is really strange. Could you share the complete end of the log ?

[JustSong]

Thanks a lot for your advice, I will try to use a better performance PC to index. If I have any improve I will that you know.

[JustSong]

• JDK: Java11.0.16.1
• Web server: Tomcat10.0.23
• Ctags: Universal Ctags 5.9.0(aaf5fa4f)
• Opengrok: 1.7.35

After I updated the opengrok version, we got a huge improve for indexing time.
The first time for indexing only takes 3hrs. Reindex with no-changes source only take 20-40min.

Thanks again for @vladak advices

[chenchuanliang]

I have the same problem. We need to index more than 60 items.From android 7 to android 12.It still takes a long time without new projects

[vladak]

After adding xref timeout to XMLAnalyzer (PR #4328), the initial AOSP indexing time using the above setup got reduced to 35 minutes.

The fix (sic!) will be part of 1.12.4.

[vladak]

Building on top of the XMLAnalyzer changes, I tried to add --threads 256 --historyThreads 256 to the indexer arguments. This had marginal effect on the indexing time.

[vladak]

Tried with --threads 512 --historyThreads 512, no change either. Also used this with bigger amount of heap (-Xmx484G), no improvement.

Looking at the CPU utilization near the end of indexing, even after the XMLAnalyzer changes, the CPU utilization is still low there, so perhaps there is more room for improvement.

[vladak]

Re-running the original indexer invocation with just released 1.12.4 and captures whole indexer run:

There is one repository with larger history that leads to the long tail in the first phase of indexing (which took ~20 minutes, the end visible as the blue triangle annotation in the CPU pane) and there is also some trailing processing in the 2nd phase of the indexing, which might be worth further investigation.

[vladak]

I tried setting historyChunkCount tunable to 50000000 (the default for Git is to process 65536 changesets and repeat until the end of history for given repo) to process all commits from the trailing repository in one go, did not have much effect w.r.t. overall indexing time (0:32:47), maybe shaving couple of minutes however I'd have to re-run the indexer with such settings multiple times in order to be sure.

Here's the whole indexer run graph for comparison:

[vladak]

Another improvement for AOSP indexing is being done in PR #4552. This is for a specific set of settings (renamed file handling on, annotation cache on) however might speed up general AOSP reindex time.