Disable re-index for large projects

[bspdev]

Hello. First of all thanks for your work on this tool. This is not a bug report, is more a a desperate call for help.

I am trying to set up a local server with large storage repositories using Opengrok in a docker container. I have everything up and running, but I am facing a problem with the automatic reindexing of the projects.

I would like to completely disable the reindexing since the repositories are large and complex to index. It spends hours to process the index data and I find that if the container is stopped and started again, the work is kind of discarded and do_sync() starts re-indexing again.

I tried all sorts of options without success and I can't find a suitable solution for this problem in the documentation.
I will be very thankful if someone can give me some clue on how to solve this problem.

[vladak]

Someone asked recently (#3874) what is the point where using Dockerized install of OpenGrok might not cut it. This might be such case.

Anyhow, the Docker README has a section with environment variables which says that setting SYNC_PERIOD_MINUTES to 0 will disable syncing (and thus indexing).

[bspdev]

Thanks for your quick response, @vladak

I saw the SYNC_PERIOD_MINUTES option in the docs and it's set to 0 already. It prevents periodic syncing but it stills re-index at the very beginning of the container startup.

I also tried more dirty approaches to try to disable this initial syncing. For example, patching /usr/local/bin/opengrok-mirror and /usr/local/bin/opengrok-reindex-project to just sys.exit(2). I also tried several options with the sync.yml and mirror.yml without success. Checking the logs, I can see that indexing is NOT done, however the webview just shows nothing at all, as if the index data has been discarded for some reason.

Do you think is there any way I can achieve my purpose? I don't mind if I have to patch some files.
Thank you again.

[vladak]

The web app needs a valid configuration for the index page to display list of projects. This configuration is created using the -W indexer option. In the Docker image the configuration is located in /opengrok/etc/configuration.xml.

I am hesitant to add an option to the Docker image that would disable the indexing altogether as that would sort of defeat its purpose. If you need to run the indexer outside of the Docker container, that very much hints at the environment not being suitable for the job. Also, in such case the container will provide just a wrapper for Tomcat and with large projects it is probably desirable to tune the Tomcat as well.

[bspdev]

Do you think I will be able to patch opengrok-reindex-project and opengrok-mirror to just execute sys.exit(2) (therefore disabling the indexing) and somehow notify the webapp to use the existing contents in /opengrok/data/ ?
I already tried that as well without success, using read-only configuration and executing the following command:

curl -d @/opengrok/etc/configuration.xml -H Content-Type: application/xml -X PUT /tmp/proper-configuration-with-projects.xml

I know that it seems it breaks the original purpose of Opengrok, but anyway it is an approach that would allow one-time indexing of the data and reuse of the logic implemented in Opengrok's Tomcat to display the information with its xrefs.

[vladak]

Instead of patching the programs in the container, you can supply opengrok-mirror configuration in /opengrok/etc/mirror.yml which would disable mirroring for all projects (see https://github.com/oracle/opengrok/wiki/Repository-synchronization for the syntax - it allows wildcards in project names). That should skip the indexing.

As for the read-only configuration, keep in mind that the paths embedded therein need to match the paths used inside the container. Again, this is a clear hint that the Docker OpenGrok environment is bent beyond its intended purpose.

If you already took the trouble of running the indexer separately, running the Tomcat and deploying the web app is just minor step I'd say. Also, keep in mind that the indexer OpenGrok version needs to be generally the same as the web app version running in the container.

[bspdev]

Thanks for your responses @vladak
I will try to achieve it using your hints. Cheers

[vladak]

Yet another idea - if you insist on running Tomcat from Docker container, it would make sense to grab the stock Docker Tomcat image , deploy the OpenGrok web app there and make it mount the OpenGrok data/configuration/source volumes. As this is just a Tomcat-in-a-box, it would not run the indexer. Also, this would remove the need to make sure the container runs the same OpenGrok version as the indexer. Lastly, creating your own Docker image based on the pre-existing images could make sense as well.

[bspdev]

Sorry for my ignorance, but I still don't understand why it is easier to start another different container with a Tomcat deploying the OpenGrok web app rather than use the existing OpenGrok container and reuse the logic for the Tomcat itself + add/delete projects

[vladak]

Because the OpenGrok container has logic that you have to change in some way as opposed to no logic in the stock Tomcat container. As for project addition/deletion, that could bring another set of problems due to the split brain.

[bspdev]

Alright @vladak. I naively thought it was somehow possible to just disable calls to opengrok-reindex-project without breaking the Tomcat's webapp. I will try your suggestions.

Thank you very much!

[vladak]

That's certainly possible (using the mirror config as suggested above) however there are other hurdles that need to be overcome.

[Sironas]

Instead of patching the programs in the container, you can supply opengrok-mirror configuration in /opengrok/etc/mirror.yml which would disable mirroring for all projects (see https://github.com/oracle/opengrok/wiki/Repository-synchronization for the syntax - it allows wildcards in project names). That should skip the indexing.

As for the read-only configuration, keep in mind that the paths embedded therein need to match the paths used inside the container. Again, this is a clear hint that the Docker OpenGrok environment is bent beyond its intended purpose.

If you already took the trouble of running the indexer separately, running the Tomcat and deploying the web app is just minor step I'd say. Also, keep in mind that the indexer OpenGrok version needs to be generally the same as the web app version running in the container.

Hello, I try to not reindexing some of my project from the mirror.yml during automatic synchronization

From the documentation on repository synchronization it says that disabling project will not run the reindex but it continues to do so

Opengrok_Prod | Output from 'opengrok-mirror -c /opengrok/etc/mirror.yml -I -U http://localhost:8080/ urbawind':
Opengrok_Prod | DEBUG:filelock:Attempting to acquire lock 139929086339840 on /tmp/urbawind-mirror.lock
Opengrok_Prod | DEBUG:filelock:Lock 139929086339840 acquired on /tmp/urbawind-mirror.lock
Opengrok_Prod | Project 'urbawind' disabled, exiting
Opengrok_Prod | DEBUG:filelock:Attempting to release lock 139929086339840 on /tmp/urbawind-mirror.lock
Opengrok_Prod | DEBUG:filelock:Lock 139929086339840 released on /tmp/urbawind-mirror.lock
Opengrok_Prod |
Opengrok_Prod | Output from 'opengrok-reindex-project --printoutput --jar /opengrok/lib/opengrok.jar -U http://localhost:8080/ -P urbawind -- -r dirbased -G -m 256 --leadingWildCards on -c /usr/local/bin/ctags -U http://localhost:8080/ -H urbawind':
Opengrok_Prod | Dec 14, 2023 9:33:05 AM org.opengrok.indexer.configuration.Configuration read
Opengrok_Prod | INFO: Reading configuration from /tmp/tmpxwgoiqq0
Opengrok_Prod | Dec 14, 2023 9:33:05 AM org.opengrok.indexer.index.Indexer parseOptions
Opengrok_Prod | INFO: Indexer options: [-R, /tmp/tmpxwgoiqq0, -r, dirbased, -G, -m, 256, --leadingWildCards, on, -c, /usr/local/bin/ctags, -U, http://localhost:8080/, -H, urbawind]
...

At least the mirroring of the project is not done, but I will have a lot of big project that I don't need to reindex

Is there a way to do that please ?