Authorization error

[carlsoane]

Describe the bug
The connaisseur MutatingWebhook is throwing a 401 error when trying to verify a signed image during pod deployment. See "Optional: To reproduce" for more information.

Expected behavior
The image would be verified and would deploy.

Optional: To reproduce
I get this error message from one of the connaisseur pods when I try to apply a pod containing the image mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test:0.0.2:

[2022-10-28 21:04:45,415] ERROR: Traceback (most recent call last):
  File "/app/connaisseur/flask_application.py", line 71, in mutate
    response = asyncio.run(__admit(admission_request))
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/app/connaisseur/flask_application.py", line 139, in __admit
    await patches
  File "/app/connaisseur/flask_application.py", line 186, in __validate_image
    trusted_digest = await validator.validate(image, **policy_rule.arguments)
  File "/app/connaisseur/validators/notaryv1/notaryv1_validator.py", line 46, in validate
    signed_image_targets = await self.__process_chain_of_trust(
  File "/app/connaisseur/validators/notaryv1/notaryv1_validator.py", line 116, in __process_chain_of_trust
    trust_data_list = await asyncio.gather(
  File "/app/connaisseur/validators/notaryv1/notary.py", line 125, in get_trust_data
    return await self.get_trust_data(image, role, token)
  File "/app/connaisseur/validators/notaryv1/notary.py", line 133, in get_trust_data
    response.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://notary-test.repositories.cloud.mycompany/v2/mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test/_trust/tuf/targets.json')

I can see that the JWT is successfully retrieved from https://auth-test.repositories.cloud.mycompany/auth?service=Notary&scope=repository:mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test:pull.
I've also verified that a token is returned by executing curl -v -u MYUSER:MYPASSWORD "https://auth-test.repositories.cloud.mycompany/auth?service=Notary&scope=repository:mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test:pull".

I can successfully pull the docker image from the docker registry if I set the notary url to "https://notary-test.repositories.cloud.mycompany"

export DOCKER_CONTENT_TRUST=1
export DOCKER_CONTENT_TRUST_SERVER=https://notary-test.repositories.cloud.mycompany
docker login mycomponent-dss-poc.common.repositories.cloud.mycompany -u MYUSER -p MYPASSWORD
docker pull mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test:0.0.2

I've also verified that I am able to lookup the GUN successfully using my notary client:

notary lookup -s https://notary-test.repositories.cloud.mycompany -d ~/.docker/trust  mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test 0.0.2
0.0.2 sha256:103e35bf7d8af9ea10a4662f6562f7e4ed6eceaab07e6d3275a319907afcdf64 526

Here is an excerpt from helm/values.yaml:

validators:
  - name: mycomponent-dss-poc
    type: notaryv1
    host: notary-test.repositories.cloud.mycompany
    auth:
      username: MYUSER
      password: MYPASSWORD
    trust_roots:
       - name: root
         key: |
           -----BEGIN PUBLIC KEY-----
           MyPublicKey...
           -----END PUBLIC KEY-----
policy:
  - pattern: "mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test:*"
    validator: mycomponent-dss-poc
    with:
      trust_root: root

Optional: Versions (please complete the following information as relevant):

• OS: gardenlinux | Version: 576.12.0
• Kubernetes Cluster: Gardener. Kubernetes v. 1.21.14
• Notary Server: Not sure
• Container registry: Docker 20.10.11
• Connaisseur: securesystemsengineering/connaisseur:v2.6.4
• Other:

Thanks

[phbelitz]

Hey @carlsoane !

Thank you for the issue! If you are able to retrieve the JWT manually, did you also try to access https://notary-test.repositories.cloud.mycompany/v2/mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test/_trust/tuf/targets.json using the token? Does that also result in a 401 unauthorized or can you access the trust data?

[carlsoane]

Hi @phbelitz:
Thanks for getting back to me. I get back 401 unauthorized. Here's a little script I'm running to test.

import json
import requests
import aiohttp
import os

myUser = os.getenv("REG_USER")
myPass = os.getenv("REG_PASS")
notaryHost = os.getenv("NOTARY_HOST")
myGUN = os.getenv("GUN")
authURL = f"https://auth-{notaryHost}/auth?service=Notary&scope=repository:{myGUN}:pull"
trustURL = f"https://notary-{notaryHost}/v2/{myGUN}/_trust/tuf/root.json"
authcreds = (myUser, myPass)

try:
    response = requests.get(authURL, auth=authcreds)
    content = json.loads(response.content.decode("utf-8"))
    print(f"Auth response: {response.status_code}; Content: {content}")
    access_token = content['access_token']
    headers = {"Authorization": f"Bearer {access_token}" }
    response = requests.get(trustURL, headers=headers)
    print(f"Trust response code: {response.status_code}; Content: {response.content.decode('utf-8')}")
except Exception as e:
    print(f"Failed: {e}")

The response I get back is:
Trust response code: 401; Content: {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test","Action":"pull"}]}]}
Note: I also got back the token in the auth response code that include the access_token and token keys but didn't think those were necessary for troubleshooting. Please let me know if they are.

My environment file is:

export REG_USER="MYUSER"
export REG_PASS="MYPASSWORD"
export NOTARY_HOST="test.repositories.cloud.mycompany"
export GUN="mycomponent-dss-poc.common.repositories.cloud.mycompany/busybox-test"

Much appreciated,
Carl

[phbelitz]

Hmm 🤔 then I guess this isn't a problem with Connaisseur, but your Notary instance. What kind of setup do you have? Is this a Harbor registry, Azure Container Registry or something custom made?

[carlsoane]

Hi @phbelitz:
I may be misunderstanding the question, but the registry is a JFrog Artifactory.

[phbelitz]

SORRY. It's been awhile ... from what I see in your script, you did pretty much everything right. Only thing that is curious, is that in the response for the token you got, the token itself is in the access_token field, not the token field. I thought this would only be the case when using ACR as your registry+notary. But apparently you are using JFrog Artifactory + a standalone Notary server.

The only thing I can imagine here, is that you have different credentials for your registry and your notary, since the script get unauthorized, but the notary-cli works fine ... maybe you have a ~/.notary/config.json file with different credentials?