Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistency between REST API description and CLI description of the same mirror, db files are missing. #1337

Open
cheeeee opened this issue Sep 11, 2024 · 5 comments
Open

Inconsistency between REST API description and CLI description of the same mirror, db files are missing. #1337

cheeeee opened this issue Sep 11, 2024 · 5 comments
Labels
bug please confirm resolved We believe the issue is resolved ! if so, please close the issue, thanks ;-)

Comments

@cheeeee
Copy link

cheeeee commented Sep 11, 2024

Detailed Description

Found inconsistency between cli output and REST API output:

aptly mirror show ubuntu-jammy-updates-full
Name: ubuntu-jammy-updates-full
Archive Root URL: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/
Distribution: jammy-updates
Components: main, restricted, universe, multiverse
Architectures: i386, amd64
Download Sources: no
Download .udebs: no
Last update: 2024-09-10 23:42:22 UTC
Number of packages: 31724

Information from release file:
Acquire-By-Hash: yes
Architectures: amd64 arm64 armhf i386 ppc64el riscv64 s390x
Codename: jammy
Components: main restricted universe multiverse
Date: Tue, 10 Sep 2024 22:20:16 UTC
Description:  Ubuntu Jammy Updates

Label: Ubuntu
Origin: Ubuntu
Suite: jammy-updates
Version: 22.04

curl http://localhost:8080/api/mirrors/ubuntu-jammy-updates-full | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   832  100   832    0     0  99059      0 --:--:-- --:--:-- --:--:--  101k
{
  "UUID": "e267beaa-f65e-4c16-89d0-5413d368e560",
  "Name": "ubuntu-jammy-updates-full",
  "ArchiveRoot": "http://us-east-1.ec2.archive.ubuntu.com/ubuntu/",
  "Distribution": "jammy-updates",
  "Components": [
    "main",
    "restricted",
    "universe",
    "multiverse"
  ],
  "Architectures": [
    "i386",
    "amd64"
  ],
  "Meta": {
    "Acquire-By-Hash": "yes",
    "Architectures": "amd64 arm64 armhf i386 ppc64el riscv64 s390x",
    "Codename": "jammy",
    "Components": "main restricted universe multiverse",
    "Date": "Thu, 08 Aug 2024 14:56:37 UTC",
    "Description": " Ubuntu Jammy Updates\n",
    "Label": "Ubuntu",
    "Origin": "Ubuntu",
    "Suite": "jammy-updates",
    "Version": "22.04"
  },
  "LastDownloadDate": "2024-08-09T02:42:42.564974851Z",
  "Filter": "",
  "Status": 0,
  "WorkerPID": 0,
  "FilterWithDeps": false,
  "SkipComponentCheck": false,
  "SkipArchitectureCheck": false,
  "DownloadSources": false,
  "DownloadUdebs": false,
  "DownloadInstaller": false
}

Also, the mirror update does not work for this mirror:

│ [GIN] 2024/09/11 - 00:07:42 | 200 |    4.479296ms |       127.0.0.1 | GET      "/api/mirrors/ubuntu-jammy-updates-full"                                                                 │
│ 2024/09/11 00:07:43 ubuntu-jammy-updates-full: Starting mirror update                                                                                                                   │
│ 2024/09/11 00:07:43 Executing task synchronously                                                                                                                                        │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease...                                                                                             │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease                                                                                        │
│ gpgv: can't allocate lock for 'key.gpg'                                                                                                                       │
│ gpgv: Signature made Tue Sep 10 23:23:19 2024 UTC                                                                                                                                       │
│ gpgv:                using RSA key F6ECB3762474EDA9D21B7022871920D1991BC93C                                                                                                             │
│ gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2018) <[email protected]>"                                                                                          │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-i386/Packages.gz...                                                                          │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-i386/Packages.gz                                                                     │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-amd64/Packages.gz...                                                                         │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/main/binary-amd64/Packages.gz                                                                    │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/restricted/binary-i386/Packages.gz...                                                                    │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/restricted/binary-i386/Packages.gz                                                               │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/restricted/binary-amd64/Packages.gz...                                                                   │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/restricted/binary-amd64/Packages.gz                                                              │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-i386/Packages.gz...                                                                      │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-i386/Packages.gz                                                                 │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-amd64/Packages.gz...                                                                     │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/universe/binary-amd64/Packages.gz                                                                │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/multiverse/binary-i386/Packages.gz...                                                                    │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/multiverse/binary-i386/Packages.gz                                                               │
│ Downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/multiverse/binary-amd64/Packages.gz...                                                                   │
│ Success downloading http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/jammy-updates/multiverse/binary-amd64/Packages.gz                                                              │
│ [GIN] 2024/09/11 - 00:07:44 | 500 |   979.47889ms |       127.0.0.1 | PUT      "/api/mirrors/ubuntu-jammy-updates-full"                                                                 │
│ Error #01: unable to update: open /root/aptly/storage/db/060118.sst: no such file or directory

Note the last line: error locating part of the DB.
I tried to cleanup and recover, but the error is the same, part of DB is missing.

Aptly running in the docker container, most operations are done using cli, and some operations are done with REST API. I also have db backups if this will help.

I do not have particular steps to reproduce, since I do not know exactly when this inconsistency started.
Please let me know how to approach this problem. Mirror re-creation is an option, but I would like to avoid it if possible.

To the best of my understanding, the issue is caused by opening db during the mirror update on this line.

Context

I want consistent data about the same object returned via REST API and CLI.

Possible Implementation

Your Environment

Aptly 1.5.0
Docker container with mounted volume for data.
@iofq
Copy link
Contributor

iofq commented Sep 11, 2024

@cheeeee If you would be willing to attach your database backups here while you have them, they could potentially be of use in troubleshooting. Though without a log of commands run against the aptly it would be difficult to pinpoint exactly what caused the corruption, if anything.

You mentioned this is running in a container, was the container potentially killed by a docker rm -f or a pod rollover? I could see SIGKILL potentially causing some db corruption if something wasn't written to disk.

Tbh we are just running high-level goleveldb API commands here and aren't well equipped to troubleshoot the database corruption. Maybe upstream leveldb repos could offer more support?

@neolynx
Copy link
Member

neolynx commented Sep 13, 2024

how did you mount the data volume ? what is the filesystem ?

@neolynx
Copy link
Member

neolynx commented Nov 3, 2024

if this is an option, you could try the CI builds. A lot of race conditions and database issues were fixed...

does the inconsistency persist over aptly restarts ?

@neolynx
Copy link
Member

neolynx commented Jan 11, 2025

aptly 1.6.0 has been released #1414 which includes many bug fixes compared to 1.5.0.

Could you give it a try ?

@neolynx neolynx added please confirm resolved We believe the issue is resolved ! if so, please close the issue, thanks ;-) and removed bug api cleanup labels Jan 11, 2025
@neolynx
Copy link
Member

neolynx commented Jan 11, 2025

I will look into this db transaction though...

@neolynx neolynx added the bug label Jan 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug please confirm resolved We believe the issue is resolved ! if so, please close the issue, thanks ;-)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@neolynx @cheeeee @iofq