Repo dies and DB files get corrupted when switching from 5.3.0 non-multi to multiple-versions-debian

[andywolk]

The problem

Users start complaining so you go check that and see that your repo is half-dead, dist files became really small, for example: dists/buster/main/binary-amd64/Packages may become just a few kilobytes of size when it has to be much bigger in big repos. People stop seeing packages in the repo's "index", but deb files are there still.

reprepro check returns:

packages.db/stretch|main|armhf: BDB0088 DB_SECONDARY_BAD: Secondary index inconsistent with primary
Internal error of the underlying BerkeleyDB database:
Within packages.db subtable stretch|main|armhf at c_get(DB_NEXT): BDB0088 DB_SECONDARY_BAD: Secondary index inconsistent with primary
There have been errors!

The cause

When you update reprepro binary to a newer version and then execute includedeb after that - dist files start containing only what you have just added. After that you may run reprepro check and start seeing the error message above.
The thing is that originally there is no db/packagenames.db file and it is created on first run, but it's somehow already broken.

To avoid that issue

Hold your breath, don't add anything YET ... Once reprepro binary is updated simply do reprepro check. After that do whatever you want.

If already broken!

Don't worry, it seems that you don't have to have a backup this time!
To FIX everything, simply:

• remove broken db/packagenames.db
• run reprepro check to populate db/packagenames.db in a right way
• run reprepro export to re-generate those almost-empty dist files so they are good and healthy and heavy.
• next time it's not gonna break, keep using reprepro.

[bdrung]

Thanks for the report! It is well written. I will look into fixing it when I have time (currently busy with too many different stuff). Test case (see tests/basic.sh and tests/multiversion.sh) and/or patches are always welcome.

[paulharris]

I got hit by this problem too.

I reset the db/ dir to what it was before, but I left the conf/distributions file with my new changes still there,
ie I had added a bionic-archive distribution.

Then, rerepro check would fail with:

# reprepro check
Checking bionic-archive...
Checking bionic...
BDB3028 packages.db: unable to flush: No such file or directory
Error: DB_ENV->close: No such file or directory

I had to remove all traces of bionic-archive from the distributions file, reset the db directory (again),
and only then would reprepro-check work correctly.
After that, I could add the bionic-archive distrib and run the check again, this time successfully.

[Zeno-sole]

BDB0689 checksums.db page 1286 is on free list with type 5 BDB0061 PANIC: Invalid argument Internal error of the underlying BerkeleyDB database: Within checksums.db subtable pool at put(uniq): BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery BDB0060 PANIC: fatal region error detected; run recovery Internal error of the underlying BerkeleyDB database: Within references.db subtable references at get: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery BDB0060 PANIC: fatal region error detected; run recovery Internal error of the underlying BerkeleyDB database: Within references.db subtable references at get: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery BDB0060 PANIC: fatal region error detected; run recovery db_close(references.db, references): BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery BDB0060 PANIC: fatal region error detected; run recovery db_close(checksums.db, pool): BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery BDB0060 PANIC: fatal region error detected; run recovery db_close(contents.cache.db, compressedfilelists): BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery BDB1581 File handles still open at environment close BDB1582 Open file handle: ./db/references.db BDB1582 Open file handle: ./db/checksums.db BDB1582 Open file handle: ./db/contents.cache.db BDB0060 PANIC: fatal region error detected; run recovery Error: DB_ENV->close: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery There have been errors!
reprepro 5.4.2 Have the same problem