migration strategy from old versions

[jkjha]

Describe the bug
After a simple restart/ minor system upgrade, the History and Annotate features stopped working

If you provide the exact version of OpenGrok, JDK used, OS(and its version) used and Tomcat(or your webapp server) used it can help figuring out an environment issue.
For performance problem OS and JDK tunables might be needed. For SCM problems also version of SCM is helpfull (e.g. some mercurial versions have issues, some not).

Opengrok version: 0.10 (I know its very old, but we can't upgrade at the moment since it has >500GB of indexed data)
Java: 1.8
Tomcat: 6.0

To Reproduce
Everything else works except History and Annotate feature which shows disabled.

I see below errors in my logs after this issue:

Error 1:

16-Mar-2022 07:30:38.668 WARNING org.opensolaris.opengrok.history.RepositoryFactory.getRepository {0} not working (missing binaries?): {1}
16-Mar-2022 07:30:38.678 WARNING org.opensolaris.opengrok.history.HistoryGuru.invalidateRepositories Failed to instanciate internal repository data for {0} in {1} 

Error 2:

1. 31-Mar-2022 00:44:42.560 SEVERE org.apache.jasper.compiler.JDTCompiler$1.findType Compilation error
 org.eclipse.jdt.internal.compiler.classfmt.ClassFormatException
        at org.eclipse.jdt.internal.compiler.classfmt.ClassFileReader.<init>(ClassFileReader.java:342)

Expected behavior
History and Annotate feature should work.

Can anyone please help to debug this more and fix.

[vladak]

Um, 0.10 is more than 10 years old (released in 2011, actually before move to Github), so I'd say there is infinitesimal chance of getting the help.

[jkjha]

@vladak , yes I understand...due to huge index size we are afraid of upgrading (Got ownership of this recently).
Any idea how complex it will be to upgrade opengrok from here to something latest? (Complexity level on the scale of 10).
This will help us to think about upgrading if above question is unanswered.

[vladak]

This is interesting question. It depends on many things. Without knowing more about your deployment I can give you only generic hints/topics. For the following I will be assuming that your deployment has the projects enabled.

Not sure if the 500GB is source root or index size, however there are certainly much bigger deployments out there. I maintain an instance for an organization in Oracle that has source root size close to 1TB with hundreds of projects with a mixture of Source Code Management types, data formats etc. and do not consider it a big deployment. That said, at this size these are definitely not small deployments. It also depends on the number of projects/repositories. There are deployments out there that have tens of thousands of repositories and that brings its own set of challenges. I'd say almost every non trivial deployment has something special to it.

Here are some things for consideration:

• identify what is special to your deployment. E.g. do you have not very commonly used Source Code Management System ? Or perhaps you have certain requirements for how frequently the indexer should run. Maybe there is a unique traffic pattern of the users. Possibly some of the repositories have very rich history, etc.
• see what is available in the current version: https://github.com/oracle/opengrok/wiki
• HW scaling: it might be that the newer version actually needs more CPU and memory. The indexer in current version is heavily parallelized so one should take this into account. The parallelism can be tuned, see https://github.com/oracle/opengrok/wiki/Indexer-configuration , also the Suggester increased the resource requirements.
• choice of indexing style: per project or the whole source root at once. Per project indexing allows for better flexibility/efficiency at the cost of complexity. It requires different workflow, see https://github.com/oracle/opengrok/wiki/Per-project-management-and-workflow
• tooling: the current version ships with Python tools that arguably make some actions easier
• at this scale you probably want to avoid using the OpenGrok Docker image

My suggestion would be to create a staging deployment with current version and index smaller part of the production source root there, experiment with it, see how it behaves, what features are available, let it run for a while to identify problems, play with the tools, add more projects. Maybe copy the whole source root from the production instance and run the indexer on it, see how it behaves (there are tools for that - see https://github.com/oracle/opengrok/wiki/Monitoring). Then decide the migration strategy. It could be gradually moving the projects over or copying the bulk of data from the old instance, indexing it (which may take a bunch of days), synchronize the projects/repositories once again, reindex and switch.

Also, create a plan how to stay on the current versions. There will be certainly version bumps in the future that will necessitate reindex from scratch (e.g. major Lucene upgrades). For the deployment I maintain I have a reindex-from-scratch and minor-upgrade scenarios that I make sure are exercised often. It's similar to backup/restore strategies.

There are probably more topics, hopefully someone else will chime in with his experiences.

[jkjha]

Thanks @vladak for your detailed reply. This will definetely help in making decision and our planning.

To provide some info about our setup:

index size: 540GB:
Total data size of root: > 2TB (Including history cache etc)

what is special to your deployment: Mostly common. But there are some really huge projects. It uses mostly Perforce and Gitlab.
choice of indexing style: per project

[vladak]

Time to close this one. Feel free to continue in https://github.com/oracle/opengrok/discussions

[vladak]

migrated to discussion.