Improve codegen docs

codegen/README.md

@@ -1,131 +1,153 @@
-# WGPU-py codegen
+# wgpu-py codegen
+
+
 
 ## Introduction
 
-The purpose of this helper package is to:
+### How wgpu-py is maintained
+
+The wgpu-py library provides a Pythonic interpretation of the [WebGPU API](https://www.w3.org/TR/webgpu/). It closely follows the official spec (in the form of an [IDL file](https://gpuweb.github.io/gpuweb/webgpu.idl)). Further below is a section on how we deviate from the spec.
+
+The actual implementation is implemented in backends. At the moment there is only one backend, based on [wgpu-native](https://github.com/gfx-rs/wgpu-native). We make API calls into this dynamic library, as specified by [two header files](https://github.com/gfx-rs/wgpu-native/tree/trunk/ffi).
+
+The API (based on the IDL) and the backend (based on the header files) can be updated independently. In both cases, however, we are dealing with a relatively large API, which is (currently) changing quite a bit, and we need the implementation to be precise. Therefore, doing the maintenance completely by hand would be a big burden and prone to errors.
+
+On the other hand, applying fully automated code generation is also not feasible, because of the many edge-cases that have to be taken into account. Plus the code-generation code must also be maintained.
+
+Therefore we aim for a hybrid approach in which the aforementioned specs are used to *check* the implementations and introduce code and comments to make updates easier.
+
+### The purpose of `codegen`
 
 * Make maintaining wgpu-py as easy as possible;
 * In particular the process of updating to new versions of WebGPU and wgpu-native;
 * To validate that our API matches the WebGPU spec, and know where it differs.
 * To validate that our calls into wgpu-native are correct.
 
-We try to hit a balance between automatic code generation and providing
-hints to help with manual updating. It should *not* be necessarry to
-check the diffs of `webgpu.idl` or `wgpu.h`. Instead, by running the
-codegen, any relevant differences (in webgpu.idl and wgpu.h) should
-result in changes (of code or annotations) in the respective `.py`
-files. That said, during development it can be helpful to use the WebGPU
-spec and the header file as a reference.
+During an update, it should *not* be necessary to check the diffs of `webgpu.idl` or `webgpu.h`. Instead, by running the
+codegen, any relevant differences in these specs should result in changes (of code or annotations) in the respective `.py`files. That said, during development it can be helpful to use the WebGPU spec and the header files as a reference.
+
+This package is *not* part of the wgpu library - it is a tool to help maintain it. It has its own tests, which try to cover the utils well,
+but the parsers and generators are less important to fully cover by tests, because we are the only users. If it breaks, we fix it.
+
+### General tips
+
+* It's probably easier to update relatively often, so that each increment is small.
+* Sometimes certain features or changes are present in WebGPU, but not in wgpu-native. This may result in some manual mappings etc. which make the code less elegant. These hacks are generally temporary though.
+* It's generally recommended to update `webgpu.idl` and `webgpu.h` separately. Though it could also be advantageous to combine them, to avoid the hacky stuff mentioned in the previous point.
 
-This package is *not* part of the wgpu-lib - it is a tool to help
-maintain it. It has its own tests, which try to cover the utils well,
-but the parsers and generators are less important to fully cover by
-tests, because we are the only users. If it breaks, we fix it.
 
 
-## Links
+## What the codegen does in general
 
-* WebGPU Spec: https://gpuweb.github.io/gpuweb/
-* IDL: https://gpuweb.github.io/gpuweb/webgpu.idl
-* C header: https://github.com/gfx-rs/wgpu/blob/master/ffi/wgpu.h
+* Help update the front API.
+  * Make changes to `base.py`.
+  * Generate `flags.py`, `enums.py`, and `structs.py`.
+*  Help update the wgpu-native backend:
+  * Make changes to `backends/rs.py`.
+  * Generate `backends/rs_mappings.py`.
+* Write `resources/codegen_report.md`  providing a summary of the codegen process.
 
 
-## Types of work that the codegen does
 
-* Fully automatically generate modules (e.g. for constants and mappings).
-* Patch manually written code:
-    * Insert annotation-comments with IDL or C definitions.
-    * Insert FIXME comments for code that is new or wrong.
-* Generate a summarizing report. This report also contains information about
-  flag/enum mismatches between IDL and wgpu.h.
+## Updating the front API
+
+### Introduction
+
+The WebGPU API is specified by `webgpu.idl` (in the resources directory). We parse this file with a custom parser (`idlparser.py`) to obtain a description of the interfaces, enums, and flags.
+
+Note that while `base.py` defines the API (and corresponding docstrings), the implementation of the majority of methods occurs in the backends, so most methods simply `raise NotimplementedError()`.
+
+### Changes with respect to JS
+
+In some cases we may want to deviate from the WebGPU API, because well ... Python is not JavaScript. There is a simple system in place to mark any such differences, that also makes sure that these changes are listed in the docs. To mark how the py API deviates from the WebGPU spec:
+
+* Decorate a method with `@apidiff.hide` to mark it as not supported by our API.
+* Decorate a method with `@apidiff.add` to mark it as intended even though it does not
+  match the WebGPU spec.
+* Decorate a method with `@apidiff.change` to mark that our method has a different signature.
 
+Other changes include:
 
-## Updating the base API (`base.py`)
+* Where in JS the input args are provided via a dict, we use kwargs directly. Nevertheless, some input args have subdicts (and sub-sub-dicts)
+*  For methods that are async in IDL, we also provide sync methods. The Async method names have an "_async" suffix. 
 
-The WebGPU API is specified by `webgpu.idl` (in the resources directory).
-We parse this file with a custom parser (`idlparser.py`) to obtain a description
-of the interfaces, enums, and flags.
+### Codegen summary
 
-The Python implementation of the flags, enums and structs is automatically generated.
-Next, the Python base API (`base.py`) is updated:
+* Generate `flags.py`, `enums.py`, and `structs.py`.
 
-* Add missing classes, methods and properties, along with a FIXME comment.
-* Modify changed signatures, along with a FIXME comment.
-* Put a comment that contains the corresponding IDL-line for each method and attribute.
-* Mark unknown classes, methods and properties with a FIXME comment.
+* Make changes to `base.py`.
 
-The update process to follow:
+  * Add missing classes, methods and properties, along with a FIXME comment..
 
-* Download the latest `webgpu.idl` (see link above) and place in the resources folder.
+  * Modify changed signatures, along with a FIXME comment.
+  * Mark unknown classes, methods and properties with a FIXME comment.
+
+  * Put a comment that contains the corresponding IDL-line for each method and attribute.
+
+
+### The update process
+
+* Download the latest [webgpu.idl](https://gpuweb.github.io/gpuweb/webgpu.idl) and place in the resources folder.
 * Run `python codegen` to apply the automatic patches to the code.
-* It may be necessary to tweak the `idlparser.py` to adjust it to new formatting.
+* It may be necessary to tweak the `idlparser.py` to adjust to new formatting.
 * Check the diff of `flags.py`, `enums.py`, `structs.py` for any changes that might need manual work.
 * Go through all FIXME comments that were added in `base.py`:
     * Apply any necessary changes.
     * Remove the FIXME comment if no further action is needed, or turn into a TODO for later.
     * Note that all new classes/methods/properties (instead those marked as hidden) need a docstring.
 * Run `python codegen` again to validate that all is well. Repeat the step above if necessary.
-* Make a summary of the API changes to put in the release notes.
+* Make sure that the tests run and provide full coverage.
+* Make sure that the examples all work.
 * Update downstream code, like our own tests and examples, but also e.g. pygfx.
+* Make a summary of the API changes to put in the release notes.
 
-In some cases we may want to deviate from the WebGPU API, because well ...
-Python is not JavaScript. There is a simple system in place to mark any
-such differences, that also makes sure that these changes are listed
-in the docs. To mark how the py API deviates from the WebGPU spec:
 
-* Decorate a method with `@apidiff.hide` to mark it as not supported by our API.
-* Decorate a method with `@apidiff.add` to mark it as intended even though it does not
-  match the WebGPU spec.
-* Decorate a method with `@apidiff.change` to mark that our method has a different signature.
 
-While `base.py` defines the API (and corresponding docstrings), the implementation
-of the majority of methods occurs in the backends.
+## Updating the Rust backend (`rs.py`)
 
+### Introduction
 
-## Updating the API of the backend implementations
+The backends are almost a copy of `base.py`: all methods in `base.py` that `raise NotImplementedError()` must be implemented.
 
-The backend implementations of the API (e.g. `rs.py`) are also patched.
-The backends are almost a copy of `base.py`: all methods in `base.py`
-that `raise NotImplementedError()` must be implemented.
- The API of the backends should not
-deviate from the base API - only additions are allowed (and should be
-used sparingly).
+The `rs.py` backend calls into a dynamic library, which interface is specified by  `webgpu.h` and `wgpu.h` (in the resources directory). We parse these files with a custom parser (`hparser.py`) to obtain a description of the interfaces, enums, flags, and structs.
 
-You'll typically need to update the backends while you're updating `base.py`.
+The majority of work in `rs.py` is the conversion of  Python dicts to C structs, and then using them to call into the dynamic library. The codegen helps by validating the structs and API calls.
 
+### Tips
 
-## Updating the Rust backend (`rs.py`)
+* In the code, use `new_struct()` and `new_struct_p()` to create a C structure with minimal boilerplate. It also converts string enum values to their corresponding integers.
 
-The `rs.py` backend calls into a C library (wgpu-native). The codegen
-helps here, by parsing the corresponding `wgpu.h` and:
+* Since the codegen adds comments for missing fields, you can instantiate a struct without any fields, then run the codegen to fill it in, and then further implement the logic.
+* The API of the backends should not deviate from the base API - only`@apidiff.add` is allowed (and should be used sparingly).
+* Use `webgpu.py` and  `wgpu.h` as a reference to check available functions and structs.
+* No docstrings needed in this module.
+* This process typically does not introduce changes to the API, but wgpu may now be more strict on specific usage or require changes to the shaders.
 
-* Detect and report missing flags and enum fields.
-* Generate mappings for enum field names to ints.
-* Validate and annotate struct creations (missing struct fields are filled in).
-* Validate and annotate function calls into the lib.
+### Codegen summary
 
-The update process to follow:
+* Generate `backends/rs_mappings.py`.
+  * Generate mappings for enum field names to ints.
+  * Detect and report missing flags and enum fields.
 
-* Download the latest `wgpu.h` using `python download-wgpu-native.py --version xx`
-* Run `python codegen` to generate code, apply patches, and produce a report.
-* Diff the report for new differences to take into account.
-* Diff `rs.py` to see what structs and functions have changed. Lines
-  marked with a FIXME comment should be fixed. Others may or may not.
-  Use `wgpu.h` as a reference to check available functions and structs.
-* Run `python codegen` again to validate that all is well.
-* Make sure that the tests run and provide full coverage.
-* This process typically does not introduce changes to the API, but certain
-  features that were previously not supported could now be withing reach.
+* Make changes to `backends/rs.py`.
+  * Validate and annotate function calls into the lib.
+  * Validate and annotate struct creations (missing struct fields are filled in).
+  * Ensure that each incoming struct is checked to catch invalid input.
 
+### The update process
 
-## Further tips
+* Download the latest `webgpu.h` and DLL using `python download-wgpu-native.py --version xx`
+* Run `python codegen` to apply the automatic patches to the code.
+* It may be necessary to tweak the `hparser.py` to adjust to new formatting.
+* Diff the report for new differences to take into account.
+* Diff `rs.py` to get an idea of what structs and functions have changed.
+* Go through all FIXME comments that were added in `rs.py`:
+  *  Apply any necessary changes.
+  * Remove the FIXME comment if no further action is needed, or turn into a TODO for later.
 
-* It's probably easier to update relatively often, so that each
-  increment is small.
-* Sometimes certain features or changes are present in WebGPU, but not
-  in wgpu-native. This may result in some manual mappings etc. which
-  make the code less elegant. These hacks are generally temporary
-  though.
-* It's generally recommended to update `webgpu.idl` and `wgpu.h`
-  separately. Though it could also be advantageous to combine them, to
-  avoid the hacky stuff mentioned in the previous point.
+* Run `python codegen` again to validate that all is well. Repeat the steps above if necessary.
+* Make sure that the tests run and provide full coverage.
+* Make sure that the examples all work.
+* Update downstream code, like our own tests and examples, but also e.g. pygfx. 
+
+* Make a summary of the API changes to put in the release notes.

wgpu/backends/rs.py

@@ -1,31 +1,17 @@
 """
-WGPU backend implementation based on wgpu-native
+WGPU backend implementation based on wgpu-native.
 
 The wgpu-native project (https://github.com/gfx-rs/wgpu-native) is a Rust
-library based on wgou-core, which wraps Metal, Vulkan, DX12, and more.
+library based on wgpu-core, which wraps Metal, Vulkan, DX12, and more.
 It compiles to a dynamic library exposing a C-API, accompanied by a C
 header file. We wrap this using cffi, which uses the header file to do
 most type conversions for us.
 
-Developer notes and tips:
-
-* The purpose of this module is to tie our Pythonic API, which closely
-  resembles the WebGPU spec, to the C API of wgpu-native.
-* Most of it is converting dicts to ffi structs. You may think that
-  this can be automated, and this would indeed be possible for 80-90%
-  of the methods. However, the API's do not always line up, and there's
-  async stuff to take into account too. Therefore we do it manually.
-  In the end, I think that this makes the code easier to maintain.
-* Use new_struct() and new_struct_p() to create a C structure with
-  minimal boilerplate. It also converts string enum values to their
-  corresponding integers.
-* The codegen will validate and annotate all struct creations and
-  function calls. So you could instantiate it without any fields and
-  then run the codegen to fill it in.
-* You may also need wgpu.h as a reference.
-* To update to the latest wgpu.h, run codegen to validate all structs
-  and function calls. Then check the diffs where changes are needed.
-  See the codegen's readme for details.
+This module is maintained using a combination of manual code and
+automatically inserted code. In short, the codegen utility inserts
+new methods and checks plus annotates all structs and C api calls.
+
+Read the codegen/readme.md for more information.
 """
 
 

wgpu/backends/rs_ffi.py

@@ -1,3 +1,6 @@
+"""Loading the header, the lib, and setting up its logging.
+"""
+
 import os
 import sys
 import logging

wgpu/backends/rs_helpers.py

@@ -1,3 +1,6 @@
+"""Utilities used in rs.py.
+"""
+
 import os
 import re
 import sys

wgpu/base.py

@@ -3,15 +3,9 @@
 properties, methods and documentation. The majority of methods are
 implemented in backend modules.
 
-Developer notes and tips:
-
-* We follow the IDL spec, with the exception that where in JS the input args
-  are provided via a dict, we use kwargs directly.
-* Nevertheless, some input args have subdicts (and sub-sub-dicts).
-* For methods that are async in IDL, we also provide sync methods.
-* The Async method names have an "_async" suffix.
-* We will try hard not to depend on asyncio specifically.
-
+This module is maintained using a combination of manual code and
+automatically inserted code. Read the codegen/readme.md for more
+information.
 """
 
 import weakref