Merge pull request #78 from Colin-b/bugfix/remaining_aws_issues

Fix remaining discrepancies with AWS documentation

CHANGELOG.md

@@ -5,20 +5,21 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+
+## [0.20.0] - 2024-02-12
 ### Fixed
 - Remove deprecation warnings due to usage of `utcnow` and `utcfromtimestamp`. Thanks to [`Raphael Krupinski`](https://github.com/rafalkrupinski).
 - `httpx_auth.AWS4Auth.default_include_headers` value kept growing in size every time a new `httpx_auth.AWS4Auth` instance was created with `security_token` parameter provided. Thanks to [`Miikka Koskinen`](https://github.com/miikka).
-- `httpx_auth.AWS4Auth` is now processing included headers without spaces in value faster.
-- `httpx_auth.AWS4Auth` is now based almost entirely on AWS documentation, solving bugs in the original implementation from `requests-aws4auth`.
+- `httpx_auth.AWS4Auth` is now based almost entirely on AWS documentation, diverging from the original implementation based on `requests-aws4auth` and solving implementation issues in the process.
   - As the AWS documentation might be wrong or not exhaustive enough, feel free to open issues, should you encounter edge cases.
 
 ### Changed
 - `httpx_auth.AWS4Auth.default_include_headers` is not available anymore, use `httpx_auth.AWS4Auth` `include_headers` parameter instead to include additional headers if the default does not fit your need (refer to documentation for an exhaustive list).
 - `httpx_auth.AWS4Auth` `include_headers` values will not be stripped anymore, meaning that you can now include headers prefixed and/or suffixed with blank spaces.
-- `httpx_auth.AWS4Auth` query fragment (`#` and everything following) is not considered as part of the canonical query string anymore. Feel free to open an issue if this is one.
 - `httpx_auth.AWS4Auth` does not includes `date` header by default anymore. You will have to provide it via `include_headers` yourself if you need to.
   - Note that it should not be required as `httpx_auth.AWS4Auth` is sending `x-amz-date` by default and AWS documentation states that the request date can be specified by using either the HTTP `Date` or the `x-amz-date` header. If both headers are present, `x-amz-date` takes precedence.
 - `httpx_auth.AWS4Auth` `include_headers` does not needs to include `host`, `content-type` or `x-amz-*` anymore as those headers will always be included. It is now expected to be provided as a list of additional headers.
+- `httpx_auth.AWS4Auth` will not modify the headers values spaces when computing the canonical headers, only trim leading and trailing whitespaces as per AWS documentation.
 
 ## [0.19.0] - 2024-01-09
 ### Added
@@ -207,7 +208,8 @@ Note that a few changes were made:
 ### Added
 - Placeholder for port of requests_auth to httpx
 
-[Unreleased]: https://github.com/Colin-b/httpx_auth/compare/v0.19.0...HEAD
+[Unreleased]: https://github.com/Colin-b/httpx_auth/compare/v0.20.0...HEAD
+[0.20.0]: https://github.com/Colin-b/httpx_auth/compare/v0.19.0...v0.20.0
 [0.19.0]: https://github.com/Colin-b/httpx_auth/compare/v0.18.0...v0.19.0
 [0.18.0]: https://github.com/Colin-b/httpx_auth/compare/v0.17.0...v0.18.0
 [0.17.0]: https://github.com/Colin-b/httpx_auth/compare/v0.16.0...v0.17.0

README.md

@@ -5,7 +5,7 @@
 <a href="https://github.com/Colin-b/httpx_auth/actions"><img alt="Build status" src="https://github.com/Colin-b/httpx_auth/workflows/Release/badge.svg"></a>
 <a href="https://github.com/Colin-b/httpx_auth/actions"><img alt="Coverage" src="https://img.shields.io/badge/coverage-100%25-brightgreen"></a>
 <a href="https://github.com/psf/black"><img alt="Code style: black" src="https://img.shields.io/badge/code%20style-black-000000.svg"></a>
-<a href="https://github.com/Colin-b/httpx_auth/actions"><img alt="Number of tests" src="https://img.shields.io/badge/tests-675 passed-blue"></a>
+<a href="https://github.com/Colin-b/httpx_auth/actions"><img alt="Number of tests" src="https://img.shields.io/badge/tests-681 passed-blue"></a>
 <a href="https://pypi.org/project/httpx-auth/"><img alt="Number of downloads" src="https://img.shields.io/pypi/dm/httpx_auth"></a>
 </p>
 
@@ -692,9 +692,9 @@ Note that the following changes were made compared to `requests-aws4auth`:
   - It is not possible to provide an `AWSSigningKey` instance, use explicit parameters instead.
   - It is not possible to provide `raise_invalid_date` parameter anymore as the date will always be valid.
   - `host` is not considered as a specific Amazon service anymore (no test specific code).
-  - Canonical query string computation is entirely based on AWS documentation (and consider undocumented fragment (`#` and following characters) as not part of the query string).
+  - Canonical query string computation is entirely based on AWS documentation (and consider undocumented fragment (`#` and following characters) as part of the query string).
   - Canonical uri computation is entirely based on AWS documentation.
-  - Canonical headers computation is almost entirely based on AWS documentation.
+  - Canonical headers computation is entirely based on AWS documentation.
 
 ### Parameters
 

httpx_auth/aws.py

@@ -1,16 +1,14 @@
 """
-Provides code for AWSAuth ported to httpx from Sam Washington's requests-aws4auth
+Provides code for AWSAuth initially ported to httpx from Sam Washington's requests-aws4auth
 https://github.com/sam-washington/requests-aws4auth
 """
 
 import datetime
 import hashlib
 import hmac
-import re
-import shlex
 from collections import defaultdict
 from posixpath import normpath
-from typing import Generator, Tuple
+from typing import Generator
 from urllib.parse import quote
 
 import httpx
@@ -135,6 +133,9 @@ def canonical_and_signed_headers(
     ...
     Lowercase(<HeaderNameN>)+":"+Trim(<value>)+"\n"
 
+    >>> canonical_and_signed_headers(httpx.Headers({"X-AMZ-Whatever": "  value with  spaces  "}), include_headers=set())
+    ('x-amz-whatever:value with  spaces\n', 'x-amz-whatever')
+
     The Lowercase() and Trim() functions used in this example are described in the preceding section.
 
     The CanonicalHeaders list must include the following:
@@ -178,17 +179,15 @@ def canonical_and_signed_headers(
             # x-amz-client-context break mobile analytics auth if included
             and not header == "x-amz-client-context"
         ):
-            included_headers[header] = trim(header_value)
+            included_headers[header] = header_value.strip()
 
     canonical_headers = ""
     signed_headers = []
     for header in sorted(included_headers):
         signed_headers.append(header)
         canonical_headers += f"{header}:{included_headers[header]}\n"
 
-    signed_headers = ";".join(signed_headers)
-
-    return canonical_headers, signed_headers
+    return canonical_headers, ";".join(signed_headers)
 
 
 def _string_to_sign(request: httpx.Request, canonical_request: str, scope: str) -> str:
@@ -281,7 +280,23 @@ def canonical_query_string(url: httpx.URL) -> str:
     ''
 
     You will still need to include the "\n".
+
+    Undocumented:
+
+    As URL fragment are not mentionned in AWS documentation, it is assumed they don't treat it as what it is and part of the query string instead
+    >>> canonical_query_string(httpx.URL("http://s3.amazonaws.com/examplebucket?#this_will_be_a_parameter=and_its_value"))
+    '%23this_will_be_a_parameter=and_its_value'
+
+    >>> canonical_query_string(httpx.URL("http://s3.amazonaws.com/examplebucket?#first=1#invalue"))
+    '%23first=1%23invalue'
+
+    >>> canonical_query_string(httpx.URL("http://s3.amazonaws.com/examplebucket?first#=1&#second=invalue&#"))
+    '%23second=invalue&first%23=1'
     """
+    if fragment := url.fragment:
+        url_without_fragment = url.copy_with(fragment=None)
+        return canonical_query_string(httpx.URL(f"{url_without_fragment}%23{fragment}"))
+
     encoded_params = defaultdict(list)
     for name, value in url.params.multi_items():
         encoded_params[uri_encode(name, is_key=True)].append(uri_encode(value))
@@ -306,17 +321,6 @@ def sign_sha256(signing_key: bytes, message: str) -> bytes:
     return hmac.new(signing_key, message.encode("utf-8"), hashlib.sha256).digest()
 
 
-def trim(value: str) -> str:
-    """
-    >>> trim(" this  is  the  value ")
-    'this is the value'
-    """
-    # TODO AWS documentation expects only leading or trailing whitespace to be removed.
-    if re.search(r"\s", value):
-        return " ".join(shlex.split(value, posix=False)).strip()
-    return value
-
-
 def uri_encode(value: str, is_key: bool = False) -> str:
     """
     See https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html for more details.

httpx_auth/version.py

@@ -3,4 +3,4 @@
 # Major should be incremented in case there is a breaking change. (eg: 2.5.8 -> 3.0.0)
 # Minor should be incremented in case there is an enhancement. (eg: 2.5.8 -> 2.6.0)
 # Patch should be incremented in case there is a bug fix. (eg: 2.5.8 -> 2.5.9)
-__version__ = "0.19.0"
+__version__ = "0.20.0"

tests/aws_signature_v4/test_aws4auth_async.py

@@ -312,7 +312,7 @@ async def test_aws_auth_header_performances_with_spaces_in_value(
         method="GET",
         match_headers={
             "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
-            "Authorization": "AWS4-HMAC-SHA256 Credential=access_id/20181011/us-east-1/iam/aws4_request, SignedHeaders=custom_with_spaces;host;x-amz-content-sha256;x-amz-date, Signature=77d54dbb83fdcd5d7086c47c67e489ba4c66f69a1493d215ca417cdec71c5a95",
+            "Authorization": "AWS4-HMAC-SHA256 Credential=access_id/20181011/us-east-1/iam/aws4_request, SignedHeaders=custom_with_spaces;host;x-amz-content-sha256;x-amz-date, Signature=ea0663a29c9f4a5225d9e882121e5c3744321c12b07ce5e6d4e7081b2e26ad8b",
             "x-amz-date": "20181011T150505Z",
             "custom_with_spaces": header_value,
         },
@@ -384,7 +384,7 @@ async def test_aws_auth_header_performances_without_spaces_in_value(
         ],
         [
             "a   b   c",
-            "7b6aea4a2378417c631c5621ddc99a94591022c775cfbb9dbf5c360492e238ef",
+            "226515100ad91c335cd215dd918807637b6f24c6ce83679f988ad953e2b80010",
         ],
         ["\nab", "3072938eb28cff19726cc2a27d5e570f916887a639b26475b390dd0edacf6496"],
     ],
@@ -686,7 +686,7 @@ async def test_aws_auth_query_reserved_with_fragment(httpx_mock: HTTPXMock):
         method="POST",
         match_headers={
             "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
-            "Authorization": "AWS4-HMAC-SHA256 Credential=access_id/20181011/us-east-1/iam/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=40f1e969709d1e89729fd9883dd2caca0ed2a8e9ec6f5fe320b5ee5629291116",
+            "Authorization": "AWS4-HMAC-SHA256 Credential=access_id/20181011/us-east-1/iam/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=73a30ab39b554b5d6b2d0e6b575b4d108794334a532068a2e388027e7914288f",
             "x-amz-date": "20181011T150505Z",
         },
     )

tests/aws_signature_v4/test_aws4auth_sync.py

@@ -302,7 +302,7 @@ def test_aws_auth_header_performances_with_spaces_in_value(
         method="GET",
         match_headers={
             "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
-            "Authorization": "AWS4-HMAC-SHA256 Credential=access_id/20181011/us-east-1/iam/aws4_request, SignedHeaders=custom_with_spaces;host;x-amz-content-sha256;x-amz-date, Signature=77d54dbb83fdcd5d7086c47c67e489ba4c66f69a1493d215ca417cdec71c5a95",
+            "Authorization": "AWS4-HMAC-SHA256 Credential=access_id/20181011/us-east-1/iam/aws4_request, SignedHeaders=custom_with_spaces;host;x-amz-content-sha256;x-amz-date, Signature=ea0663a29c9f4a5225d9e882121e5c3744321c12b07ce5e6d4e7081b2e26ad8b",
             "x-amz-date": "20181011T150505Z",
             "custom_with_spaces": header_value,
         },
@@ -373,7 +373,7 @@ def test_aws_auth_header_performances_without_spaces_in_value(
         ],
         [
             "a   b   c",
-            "7b6aea4a2378417c631c5621ddc99a94591022c775cfbb9dbf5c360492e238ef",
+            "226515100ad91c335cd215dd918807637b6f24c6ce83679f988ad953e2b80010",
         ],
         ["\nab", "3072938eb28cff19726cc2a27d5e570f916887a639b26475b390dd0edacf6496"],
     ],
@@ -663,7 +663,7 @@ def test_aws_auth_query_reserved_with_fragment(httpx_mock: HTTPXMock):
         method="POST",
         match_headers={
             "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
-            "Authorization": "AWS4-HMAC-SHA256 Credential=access_id/20181011/us-east-1/iam/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=40f1e969709d1e89729fd9883dd2caca0ed2a8e9ec6f5fe320b5ee5629291116",
+            "Authorization": "AWS4-HMAC-SHA256 Credential=access_id/20181011/us-east-1/iam/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=73a30ab39b554b5d6b2d0e6b575b4d108794334a532068a2e388027e7914288f",
             "x-amz-date": "20181011T150505Z",
         },
     )