'NoneType' object has no attribute 'artifact_set_sha256' when syncing a recently updated deb remote

[paralllax]

Version

Installed and setup via ansible (pulp_installer)

    {
      "component": "core",
      "version": "3.21.3",
      "package": "pulpcore"
    },
    {
      "component": "deb",
      "version": "2.20.0",
      "package": "pulp_deb"
    }

Describe the bug

It looks like there is an issue comparing the indices for a remote when syncing a repository after the remote has updated. When running a sync job, it returns with this error:

.Error: Task /pulp/api/v3/tasks/28fcc35c-15cf-4e00-be84-98041c6dd7f2/ failed: ''NoneType' object has no attribute 'artifact_set_sha256''

The stack trace for the error is:

File "/pulp/pulpenv/lib64/python3.9/site-packages/pulpcore/tasking/pulpcore_worker.py", line 452, in _perform_task
  result = func(*args, **kwargs)
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 170, in synchronize
  DebDeclarativeVersion(first_stage, repository, mirror=mirror).create()
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
  loop.run_until_complete(pipeline)
File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
  return future.result()
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
  await asyncio.gather(*futures)
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
  await self.run()
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 544, in run
  await asyncio.gather(
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 679, in _handle_distribution
  await asyncio.gather(*sub_tasks)
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 763, in _handle_component
  await asyncio.gather(*pending_tasks)
File "/pulp/pulpenv/lib64/python3.9/site-packages/pulp_deb/app/tasks/synchronizing.py", line 861, in _handle_package_index
  if previous_package_index.artifact_set_sha256 == package_index.artifact_set_sha256:

To Reproduce

It may be difficult to reproduce. This is only happening on the microsoft repositories. We are syncing arch amd64 for jammy, focal, and bionic. In order to reproduce it you need to have a previous version of the repository/release. If you setup a brand new remote and repository, it doesn't encounter this issue when syncing, since it has nothing to compare from.

These are the basic settings

pulp deb remote create --name "microsoft_deb_focal" --architecture amd64 --distribution focal --url "https://packages.microsoft.com/ubuntu/20.04/prod"
pulp deb repository create --description "Microsoft focal repo" --remote microsoft_deb_focal --name repo_microsoft_deb_focal
pulp deb repository sync --name repo_microsoft_deb_focal

Expected behavior

The repository should sync and pull in the new updates without any issues.

Additional context

The remote microsoft repos we're mirror'ing were updated several days ago. The day they were updated is when our sync's started to fail. It's only for those repos. I am uncertain if they configured something incorrectly in their release, or it's simply a bug with the plugin. The timing is odd nonetheless. These are the URLs:

• https://packages.microsoft.com/ubuntu/18.04/prod
• https://packages.microsoft.com/ubuntu/20.04/prod
• https://packages.microsoft.com/ubuntu/22.04/prod

if it's of any relevance, here is the remote config for one of the above. They're identical aside from name and URL

{
  "pulp_href": "/pulp/api/v3/remotes/deb/apt/819cf3bf-4559-4027-8ed6-fa2c27eb462b/",
  "pulp_created": "2022-12-09T22:11:46.085397Z",
  "name": "deb_remote_microsoft_focal",
  "url": "https://packages.microsoft.com/ubuntu/20.04/prod",
  "ca_cert": null,
  "client_cert": null,
  "tls_validation": true,
  "proxy_url": null,
  "pulp_labels": {},
  "pulp_last_updated": "2023-02-06T14:53:20.757223Z",
  "download_concurrency": 2,
  "max_retries": 10,
  "policy": "on_demand",
  "total_timeout": 1200.0,
  "connect_timeout": null,
  "sock_connect_timeout": null,
  "sock_read_timeout": 60.0,
  "headers": null,
  "rate_limit": null,
  "distributions": "focal",
  "components": null,
  "architectures": "amd64",
  "sync_sources": false,
  "sync_udebs": false,
  "sync_installer": false,
  "gpgkey": null,
  "ignore_missing_package_indices": false
}

[quba42]

@paralllax
Try running your sync command using the --no-optimize flag as a workaround:

pulp deb repository sync --no-optimize --name repo_microsoft_deb_focal

If that fixes the issue, please try syncing again once more using optimize mode:

pulp deb repository sync --optimize --name repo_microsoft_deb_focal

If this second sync now works (immediately after the other sync, it should be very quick), then this was likely a one time effect due to a previous sync that was corrected by the next successful sync. If you are consistently getting the error when using --optimize, but not when you are using --no-optimize, then it would be good if you could give us the output of the following command for further analysis:

pulp deb repository version show --repository repo_microsoft_deb_focal

[paralllax]

Hi @quba42,

My apologies, I somehow missed the notification for your update! After about a week or so, the issue went away, I assume because something changed on the endpoint. However it came back a couple days ago once more. It looks like that directive may not be available in our version:

Error: No such option: --no-optimize

I tried with the following:

 http -a <user>:<pass> POST http://localhost:24817$(pulp deb repository show --name repo_microsoft_deb_focal | jq -r '.pulp_href')sync/ optimize=false

I did a normal sync after which worked. Thank you for sharing the work around!

Since it's come up a couple times for the same set of 3 repos (microsoft repos for bionic, focal, & jammy), in case it's still relevant, here is the version output for bionic before I used your work around.

{
  "pulp_href": "/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/",
  "pulp_created": "2023-03-04T06:22:58.752350Z",
  "number": 33,
  "repository": "/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/",
  "base_version": null,
  "content_summary": {
    "added": {
      "deb.release_file": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/release_files/?repository_version_added=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.package_index": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/package_indices/?repository_version_added=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.package": {
        "count": 8,
        "href": "/pulp/api/v3/content/deb/packages/?repository_version_added=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.package_release_component": {
        "count": 8,
        "href": "/pulp/api/v3/content/deb/package_release_components/?repository_version_added=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      }
    },
    "removed": {
      "deb.package_index": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/package_indices/?repository_version_removed=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.release_file": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/release_files/?repository_version_removed=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      }
    },
    "present": {
      "deb.package": {
        "count": 1947,
        "href": "/pulp/api/v3/content/deb/packages/?repository_version=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.package_index": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/package_indices/?repository_version=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.package_release_component": {
        "count": 1947,
        "href": "/pulp/api/v3/content/deb/package_release_components/?repository_version=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.release": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/releases/?repository_version=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.release_architecture": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/release_architectures/?repository_version=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.release_component": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/release_components/?repository_version=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      },
      "deb.release_file": {
        "count": 1,
        "href": "/pulp/api/v3/content/deb/release_files/?repository_version=/pulp/api/v3/repositories/deb/apt/8c79d58d-3367-4706-9568-ca57810237f4/versions/33/"
      }
    }
  }
}

[quba42]

--optimize/--no-optimize was added in version 0.0.3 of pulp-cli-deb: https://github.com/pulp/pulp-cli-deb/blob/main/CHANGES.md#003-2023-01-03

Apart from that, you have given us just about all the information you can for us to keep digging for the root cause.