Allow source files to publish at by-hash paths

[acheng-01]

Refactored the previous AptByHash code to extend publishing at by-hash paths for source files.

Closes #1059

[acheng-01]

I've confirmed that source files have their metadata published to /source/by-hash.
In my testing I have also confirmed that the by-hash file is used upon downloading the package with apt-get.

[hstct]

Verified that this works. I'm inclined to merge this. In the end though it would be great to have tests for apt-by-hash in general not necessary source but all packages. But I can create an extra issue for this for now.

[quba42]

@acheng-01 I am going to second what @hstct has said.

Since I do not naturally use/manually test the APT by hash story, I will not notice if anything breaks unless it has test coverage, so it is really in the interest of those using the story to add that test coverage.

That being said, I am willing to merge this as is (especially since we did not have the common curtsy to give an initial review for this in a timely manner).

@acheng-01, @daviddavis (if you are involved with this): Please let us know how you want to proceed.

[daviddavis]

Yea, I thought we had tests for apt-by-hash but it looks like we just added one for caching content to be served for apt-by-hash.

I think adding a test for apt-by-hash makes sense. I don't have a preference if we add it in this PR or a followup PR. Also, I might suggest we add a general apt-by-hash test and not one for the source files since source files are a pain to test with.

Also, how do we enable APT_BY_HASH in the test environment. Just set it here?

[hstct]

Also, how do we enable APT_BY_HASH in the test environment. Just set it here?

@daviddavis Yea I think that should do it.

It might be sufficient to add a test here and see if we can just download the correct files from the by-hash folders? That way it should be pretty simple to test both standard and source packages. The default repository in the test fixtures provides both types of packages.

[acheng-01]

Thanks for chiming in everyone. I'll look into adding tests in within the next week or so as part of this PR!

[acheng-01]

Apologies that getting this test in has taken so long, but it's finally here! Please let me know if there is anything I should change with this test - otherwise it is passing on my end. Thanks everyone for your patience :)

[daviddavis]

@quba42 @hstct The new test here is passing locally for me and @acheng-01 but it seems to fail in the CI. Any ideas?

[daviddavis]

My guess is that the CI scripts need to be regenerated given the change to template_config.yml but I have no idea how to do that anymore.

[quba42]

My guess is that the CI scripts need to be regenerated given the change to template_config.yml but I have no idea how to do that anymore.

What you need to do is have both pulp_deb and plugin template cloned locally. Make sure you check out this PR branch in pulp_deb. In the pugin_template repo make sure you check out the commit referenced here (e.g.: git checkout 2021.08.26-322-g8700c1c). Then in the plugin_template folder run: ./plugin-template --github pulp_deb. Go back to the pulp_deb repo and check what the plugin template has done. Add it to the PR and hopefully everything will work.

[acheng-01]

@quba42 I did the following:

1. rebased my PR branch against main
2. ran the plugin-template script as you had instructed
3. pushed whatever changed into this PR

There are a lot of files changed after those steps and I'm not sure if they're correct. For some reason the gitref says unknown now. Is that supposed to happen?

[acheng-01]

Never mind, @daviddavis helped me identify what I did wrong in the process. I did the template thing again and things look right now. Could use a second set of eyes to verify this though.

[acheng-01]

I'm not entirely sure why this test is not working in the CI pipeline. It appears that the publication isn't created successfully somehow? The gen_object_with_cleanup fixture uses the monitor_task fixture that seems to be triggering the PulpTaskError for the task not being in a completed state. However, I can't tell if this means the task is skipped, canceled, or failed...

I haven't been able to repro this failure locally so it's been difficult trying to understand why this is happening. The way that this test initiates the repo, publication, and distribution should be relatively similar to the other two content tests, with the only difference being that I'm passing in "sync_sources": True for remote args. Maybe that's what's causing this test to behave differently?

Would greatly appreciate if anyone might know why this is happening :)

[acheng-01]

One more find in the logs:

pulp [None]: aiohttp.access:INFO: 127.0.0.1 [11/Jul/2024:21:22:37 +0000] "GET /debian/dists/ragnarok/jotunheimr/source/Sources HTTP/1.1" 404 172 "-" "pulpcore/3.55.1 (cpython 3.9.19-final0, Linux x86_64) (aiohttp 3.9.5)"
pulp [None]: aiohttp.access:INFO: 127.0.0.1 [11/Jul/2024:21:22:37 +0000] "GET /debian/dists/ragnarok/jotunheimr/source/Release HTTP/1.1" 200 247 "-" "pulpcore/3.55.1 (cpython 3.9.19-final0, Linux x86_64) (aiohttp 3.9.5)"

I was expecting source/Sources to be available instead of source/Release. For some reason some step is pulling for both. Is there a verification somewhere that checks that all these metadata files exist? It's not immediately obvious to me how these endpoints are being checked.

[daviddavis]

@acheng-01 I was able to reproduce the error locally. If you rebase your branch against main, your test fails. I believe something about this change that was merged yesterday is causing your change to fail.

[acheng-01]

Bug fix for awareness. I implemented @daviddavis's change so that the ci tests are passing for this PR, but let me know if I should revert it.

Again, huge thanks for the assist!

[hstct]

Ah yea, sorry I missed that when adding the change last week. I merged @daviddavis's PR.

Thanks for implementing the test. Reviewed your changes again, they are looking good! You can revert that patch from yesterday and everything should be good to go!

[acheng-01]

@hstct I did another rebase on main and it looks like all the tests are passing. Ready for merge!

[hstct]

Awesome! Thanks for this contribution!