Split up smithy-python

.pre-commit-config.yaml

@@ -4,7 +4,7 @@ repos:
     hooks:
       - id: check-yaml
       - id: end-of-file-fixer
-        exclude: python-packages/aws-smithy-python/tests/unit/auth/aws4_testsuite/
+        exclude: python-packages/smithy-aws-core/tests/unit/auth/aws4_testsuite/
       - id: trailing-whitespace
   - repo: local
     hooks:

BUILD

@@ -0,0 +1,2 @@
+# This is needed for mypy to be able to typecheck the tests properly
+python_requirements(name="test-reqs", source="requirements-dev.txt")

Makefile

@@ -50,20 +50,23 @@ test-protocols: install-python-components generate-protocol-tests run-protocol-t
 
 ## Runs formatters/fixers/linters for the python packages.
 lint-py: pants
-	./pants fix lint python-packages/smithy-python::
-	./pants fix lint python-packages/aws-smithy-python::
+	./pants fix lint python-packages/smithy-core::
+	./pants fix lint python-packages/smithy-http::
+	./pants fix lint python-packages/smithy-aws-core::
 
 
 ## Runs checkers for the python packages.
 check-py: pants
-	./pants check python-packages/smithy-python::
-	./pants check python-packages/aws-smithy-python::
+	./pants check python-packages/smithy-core::
+	./pants check python-packages/smithy-http::
+	./pants check python-packages/smithy-aws-core::
 
 
 ## Runs tests for the python packages.
 test-py: pants
-	./pants test python-packages/smithy-python::
-	./pants test python-packages/aws-smithy-python::
+	./pants test python-packages/smithy-core::
+	./pants test python-packages/smithy-http::
+	./pants test python-packages/smithy-aws-core::
 
 
 ## Runs formatters/fixers/linters/checkers/tests for the python packages.

README.md

@@ -147,12 +147,15 @@ helper libraries have been published you won't need to install them manually.
 
 ### Core Modules and Interfaces
 
-The `smithy-python` package provides the core modules and interfaces required
-to build a service client. These basic modules include things like:
-an HTTP/1.1 and HTTP/2 client implementation, retry strategies, etc.
-
-The `aws-smithy-python` package provides implementations of those interfaces
-for AWS, such as SigV4 signers.
+* `smithy-core` provides transport-agnostic core modules and interfaces
+  required to build a service client. This includes things like retry
+  strategies, URI interfaces, shared types, etc.
+* `smithy-http` provides HTTP core modules and interfaces required to build
+  HTTP service clients, including optional HTTP client implementations.
+  Currently it provides two async HTTP clients that are useable with the
+  `aiohttp` or `awscrt` optional dependency sets respectively.
+* `smithy-aws-core` provides implementations of those interfaces for AWS, such
+  as SigV4 signers.
 
 ### What are the design goals of this project?
 
@@ -256,10 +259,10 @@ Important to note is those pairs of colons. These are pants
 [targets](https://www.pantsbuild.org/docs/targets#target-addresses). The double
 colon is a special target that means "everything". So running exactly what's
 listed above will run those goals on every python file or other relevant file.
-You can also target just `smithy_python`, for example, with
-`./pants check python-packages/smithy-python/smithy_python:source`, or even
+You can also target just `smithy_core`, for example, with
+`./pants check python-packages/smithy-python/smithy_core:source`, or even
 individual files with something like
-`./pants check python-packages/smithy-python/smithy_python/interfaces/http.py:../source`.
+`./pants check python-packages/smithy-python/smithy_core/interfaces/identity.py:../source`.
 To list what targets are available in a directory, run
 `./pants list path/to/dir:`. For more detailed information, see the
 [docs](https://www.pantsbuild.org/docs/targets#target-addresses).

codegen/README.md

@@ -14,14 +14,16 @@ clients.
 * Python 3.11 is required to run the generated code, but not run the generator.
   If it is present on the path, the generator will use it for linting and
   formatting.
-* Use of a [Python virtual environment](https://docs.python.org/3/library/venv.html)
-  is highly recommended.
-* If `black` is installed in the version of python found on the path, it will
-  be used to format the generated code.
-* If `mypy` is installed in the version of python found on the path, it will
-  be used to check the generated code. For mypy to pass, the `smithy_python`
-  package will need to be installed. To install those into your active environment,
-  run `make install-python-components` from the repository root.
+* Use of a
+  [Python virtual environment](https://docs.python.org/3/library/venv.html) is
+  highly recommended.
+* If `black` is installed in the version of python found on the path, it will be
+  used to format the generated code.
+* If `mypy` is installed in the version of python found on the path, it will be
+  used to check the generated code. For mypy to pass, the `smithy_core` and
+  `smithy_http` packages will need to be installed. To install those into your
+  active environment, run `make install-python-components` from the repository
+  root.
 
 ### Building the generator
 
@@ -34,52 +36,52 @@ root. To run the protocol tests, run `make test-protocols`.
 
 ### Where should I get started?
 
-The best place to start is the [Smithy documentation](https://smithy.io/) to understand
-what Smithy is and how this project relates to it. In particular, the [Creating a
-Smithy Code Generator](https://smithy.io/2.0/guides/building-codegen/index.html) guide
+The best place to start is the [Smithy documentation](https://smithy.io/) to
+understand what Smithy is and how this project relates to it. In particular, the
+[Creating a Smithy Code
+Generator](https://smithy.io/2.0/guides/building-codegen/index.html) guide
 covers the overall design of Smithy generators.
 
 
-[`PythonClientCodegenPlugin`
-](https://github.com/awslabs/smithy-python/blob/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/PythonClientCodegenPlugin.java),
-a [Smithy build plugin
-](https://smithy.io/2.0/guides/building-codegen/creating-codegen-repo.html#creating-a-smithy-build-plugin),
+[`PythonClientCodegenPlugin`](https://github.com/awslabs/smithy-python/blob/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/PythonClientCodegenPlugin.java),
+a
+[Smithy build plugin](https://smithy.io/2.0/guides/building-codegen/creating-codegen-repo.html#creating-a-smithy-build-plugin),
 is the entry point where this code generator links to the Smithy build process
-(see also: [SmithyBuildPlugin javadoc
-](https://smithy.io/javadoc/1.26.1/software/amazon/smithy/build/SmithyBuildPlugin.html)).
+(see also:
+[SmithyBuildPlugin javadoc](https://smithy.io/javadoc/1.26.1/software/amazon/smithy/build/SmithyBuildPlugin.html)).
 This class doesn't do much by itself, but everything flows from here.
 
-Another good place to start is [`DirectedPythonCodegen`
-](https://github.com/awslabs/smithy-python/blob/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/DirectedPythonCodegen.java).
-This is an implementation of Smithy's [directed codegen interface
-](https://smithy.io/javadoc/1.26.1/software/amazon/smithy/codegen/core/directed/DirectedCodegen.html)
-, which enables us to make use of shared orchestration code and provides a more guided
-path to generating a client. This class is the heart of the generator, everything else
-follows from here. For example, you could look here to find out how Smithy structures
-are generated into Python objects. Most of the code that does that is somewhere else,
-but it's called directly from here. This class is constructed by `PythonCodegenPlugin`
-and handed off to a [`CodegenDirector`
-](https://smithy.io/javadoc/1.26.1/software/amazon/smithy/codegen/core/directed/CodegenDirector.html)
+Another good place to start is
+[`DirectedPythonCodegen`](https://github.com/awslabs/smithy-python/blob/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/DirectedPythonCodegen.java).
+This is an implementation of Smithy's
+[directed codegen interface](https://smithy.io/javadoc/1.26.1/software/amazon/smithy/codegen/core/directed/DirectedCodegen.html)
+, which enables us to make use of shared orchestration code and provides a more
+guided path to generating a client. This class is the heart of the generator,
+everything else follows from here. For example, you could look here to find out
+how Smithy structures are generated into Python objects. Most of the code that
+does that is somewhere else, but it's called directly from here. This class is
+constructed by `PythonCodegenPlugin` and handed off to a
+[`CodegenDirector`](https://smithy.io/javadoc/1.26.1/software/amazon/smithy/codegen/core/directed/CodegenDirector.html)
 which calls its public methods.
 
-One more possible starting point is [`SymbolVisitor`
-](https://github.com/awslabs/smithy-python/blob/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/SymbolVisitor.java).
+One more possible starting point is
+[`SymbolVisitor`](https://github.com/awslabs/smithy-python/blob/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/SymbolVisitor.java).
 This class is responsible for taking a Smithy shape and determining what its
-name in Python should be, where it should be defined in Python, what dependencies
-it has, and attaching any other important properties that are needed for generating
-the python types. See the [smithy docs
+name in Python should be, where it should be defined in Python, what
+dependencies it has, and attaching any other important properties that are
+needed for generating the python types. See the [smithy docs
 ](https://smithy.io/2.0/guides/building-codegen/decoupling-codegen-with-symbols.html)
 for more details on the symbol provider concept.
 
-Finally, you might look at the [`integration` package
-](https://github.com/awslabs/smithy-python/tree/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/integration).
-This package is what provides [plugins
-](https://smithy.io/2.0/guides/building-codegen/making-codegen-pluggable.html)
-to the python generator. Anything that implements [`PythonIntegration`
-](https://github.com/awslabs/smithy-python/blob/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/integration/PythonIntegration.java)
-which is present on the Java classpath can hook into the code generation process.
-Crucially, this is how protocol implementations are implemented and how default
-runtime customizations are added. See
+Finally, you might look at the
+[`integration` package](https://github.com/awslabs/smithy-python/tree/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/integration).
+This package is what provides
+[plugins](https://smithy.io/2.0/guides/building-codegen/making-codegen-pluggable.html)
+to the python generator. Anything that implements
+[`PythonIntegration`](https://github.com/awslabs/smithy-python/blob/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/integration/PythonIntegration.java)
+which is present on the Java classpath can hook into the code generation
+process. Crucially, this is how protocol implementations are implemented and how
+default runtime customizations are added. See
 [`RestJsonIntegration`](https://github.com/awslabs/smithy-python/blob/develop/codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/integration/RestJsonIntegration.java)
 for an example of a protocol implementation.
 

codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/ApplicationProtocol.java

@@ -53,9 +53,9 @@ public static ApplicationProtocol createDefaultHttpApplicationProtocol() {
     }
 
     private static Symbol createHttpSymbol(String symbolName) {
-        PythonDependency dependency = SmithyPythonDependency.SMITHY_PYTHON;
+        PythonDependency dependency = SmithyPythonDependency.SMITHY_HTTP;
         return Symbol.builder()
-                .namespace(dependency.packageName() + ".interfaces.http", ".")
+                .namespace(dependency.packageName() + ".aio.interfaces", ".")
                 .name(symbolName)
                 .addDependency(dependency)
                 .build();

codegen/smithy-python-codegen/src/main/java/software/amazon/smithy/python/codegen/ClientGenerator.java

@@ -125,12 +125,12 @@ private void generateOperationExecutor(PythonWriter writer) {
         writer.addStdlibImport("copy", "deepcopy");
         writer.addStdlibImport("asyncio", "sleep");
 
-        writer.addDependency(SmithyPythonDependency.SMITHY_PYTHON);
-        writer.addImport("smithy_python.exceptions", "SmithyRetryException");
-        writer.addImport("smithy_python.interfaces.interceptor", "Interceptor");
-        writer.addImport("smithy_python.interfaces.interceptor", "InterceptorContext");
-        writer.addImport("smithy_python.interfaces.retries", "RetryErrorInfo");
-        writer.addImport("smithy_python.interfaces.retries", "RetryErrorType");
+        writer.addDependency(SmithyPythonDependency.SMITHY_CORE);
+        writer.addImport("smithy_core.exceptions", "SmithyRetryException");
+        writer.addImport("smithy_core.interceptors", "Interceptor");
+        writer.addImport("smithy_core.interceptors", "InterceptorContext");
+        writer.addImport("smithy_core.interfaces.retries", "RetryErrorInfo");
+        writer.addImport("smithy_core.interfaces.retries", "RetryErrorType");
 
         writer.indent();
         writer.write("""
@@ -309,9 +309,10 @@ async def _handle_attempt(
                 writer.consumer(this::initializeHttpAuthParameters));
             writer.popState();
 
-            writer.addDependency(SmithyPythonDependency.SMITHY_PYTHON);
-            writer.addImport("smithy_python.interfaces.identity", "Identity");
-            writer.addImports("smithy_python.interfaces.auth", Set.of("HTTPSigner", "HTTPAuthOption"));
+            writer.addDependency(SmithyPythonDependency.SMITHY_CORE);
+            writer.addDependency(SmithyPythonDependency.SMITHY_HTTP);
+            writer.addImport("smithy_core.interfaces.identity", "Identity");
+            writer.addImports("smithy_http.aio.interfaces.auth", Set.of("HTTPSigner", "HTTPAuthOption"));
             writer.addStdlibImport("typing", "Any");
             writer.write("""
                         auth_options = config.http_auth_scheme_resolver.resolve_auth_scheme(
@@ -345,10 +346,11 @@ async def _handle_attempt(
 
         writer.pushState(new ResolveEndpointSection());
         if (context.applicationProtocol().isHttpProtocol()) {
-            writer.addDependency(SmithyPythonDependency.SMITHY_PYTHON);
+            writer.addDependency(SmithyPythonDependency.SMITHY_CORE);
+            writer.addDependency(SmithyPythonDependency.SMITHY_HTTP);
             // TODO: implement the endpoints 2.0 spec and remove the hard-coded handling of static params.
-            writer.addImport("smithy_python._private.http", "StaticEndpointParams");
-            writer.addImport("smithy_python._private", "URI");
+            writer.addImport("smithy_http.endpoints", "StaticEndpointParams");
+            writer.addImport("smithy_core", "URI");
             writer.write("""
                         # Step 7f: Invoke endpoint_resolver.resolve_endpoint
                         if config.endpoint_resolver is None:
@@ -425,8 +427,8 @@ async def _handle_attempt(
 
         writer.pushState(new SendRequestSection());
         if (context.applicationProtocol().isHttpProtocol()) {
-            writer.addDependency(SmithyPythonDependency.SMITHY_PYTHON);
-            writer.addImport("smithy_python.interfaces.http", "HTTPRequestConfiguration");
+            writer.addDependency(SmithyPythonDependency.SMITHY_HTTP);
+            writer.addImport("smithy_http.interfaces", "HTTPRequestConfiguration");
             writer.write("""
                         # Step 7m: Invoke http_client.send
                         request_config = config.http_request_config or HTTPRequestConfiguration()