Generate W&B Python SDK with lazydocs

Scripts that generated markdown docs for Workspaces API and broader W&B Python API documentation.

Setup

1. Navigate to the root directory that contains your local clone of the wandb repository. This is the directory that contains the wandb package.

   cd path/to/your/root/directory/with/wandb/package

2. Clone generate-wandb-python-reference repository:

   git clone https://github.com/user-attachments/generate-wandb-python-reference.git

   Your local directory structure should look like this:

   awesome-directory/
   ├── generate-wandb-python-reference/
   │   ├── create_wandb_sdk_docs.sh
   │   ├── generate_sdk_docs.py
   │   ├── process_markdown.py
   │   ├── sort_markdown_files.py
   │   ├── create_landing_pages.py
   │   ├── requirements.txt
   │   └── configuration.py
   └── wandb/
       ├── wandb/
       │   ├── __init__.py
       │   ├── __init__.template.pyi
       │   └── ... # other files
       └──
   

3. Install the required dependencies:

   cd generate-wandb-python-reference
   pip install -r requirements.txt

Create W&B Python SDK Docs

These scripts use the local cloned version of wandb package to generate the markdown files. (This is why you need to clone the generate-wandb-python-reference repository into the same directory as your local wandb package.)

Check out the branch or commit that you want to generate the docs for.

git checkout <branch-or-commit>

The entrypoint for generating the W&B Python SDK docs is the generate-wandb-python-reference/create_wandb_sdk_docs.sh script.

bash create_wandb_sdk_docs.sh

The output will be generated in the wandb/wandb/docs/python directory. The generated markdown files will be organized into subdirectories based on the object_type specified in the front matter of each markdown file.

Add W&B (Models) APIs to the reference docs

Start off by asking yourself: Is the API you defined already in an existing namespace? E.g. wandb.sdk, wandb.apis.public, or wandb.sdk.launch. For a full list, see the "module" keys specifed in configuration.py.

If yes, then:

1. Add your new APIs to the __all__ contsant within the appropriate __init__.py or __init__.template.pyi file. See wandb/wandb/__init__.template.pyi for an example.

2. Add an ignore marker to the docstring of the class, method, or function that you want to exclude from the documentation. For example:

   class MyClass:
       """This is an awesome Python Class.
   
       <!-- lazydoc-ignore-class: internal -->
       """
       def my_method(self):
           """This is an awesome method."""
           pass

   See the ignore markers section for more details on how to use ignore markers.

If no, then:

1. Open configuration.py
2. Add a new entry to the SOURCE dictionary. Here is a template for reference:

    "API_NAME": {
        "module": "", # The module that contains your code (e.g. wandb.apis.public)
        "file_path": "", # File path of the local wandb/wandb source files
        "hugo_specs": {
            "title": "", # Title of the folder (What appears in the left navigation)
            "description": "", # Description of the top most _index.md file
            "frontmatter": "object_type: ", # frontmatter, used for sorting
            "folder_name": "", # Desired directory within python E.g. python/launch-library, python/data-type/
        }
    }

For file_path, use the variable BASE_DIR as a prefix. For example, for wandb.apis.public, the file path would look like this:

"file_path": BASE_DIR / "wandb" / "wandb" / "apis" / "public" / "__init__.py"

Ignore markers

To exclude certain classes, methods, or functions from the documentation generated by lazydocs, you can use the following ignore markers in your code:

1. <!-- lazydoc-ignore: internal --> - Removes internal class method definitions
2. <!-- lazydoc-ignore-class: internal --> - Removes entire class definitions
3. <!-- lazydoc-ignore-function: internal --> - Removes function definitions
4. <!-- lazydoc-ignore-classmethod: internal --> - Removes @classmethod definitions
5. <!-- lazydoc-ignore-init: internal --> - Removes __init__ method definitions
6. <!-- lazydoc-ignore-class-attributes --> - Removes individual attribute bullet points

Why does this exist?

• There are internal classes, methods, properties, etc. in the Python SDK that are not meant to be publicly exposed and do not use the Python convention of preprending a single underscore (_) to indicate that they are private.
• lazydocs is a generic tool that generates documentation for any Python code, it may include some artifacts that are not relevant to the W&B Python SDK.

Architecture and key components

Documentation pipeline

create_wandb_sdk_docs.sh follows a 4-step process to generate the W&B Python SDK documentation:

1. Call generate_sdk_docs.py script to generate markdown docs using lazydocs. The script uses Classes, functions, and W&B Data Types from */__init__.pyi file.
2. Call process_markdown.py script to process the markdown docs generated by lazydocs. Namely, the processing script removes weird artifacts that lazydocs generates and adds the necessary front matter to the markdown files that Hugo requires.
3. Call sort_markdown_files.py to reorganize the markdown files into the correct directory structure for the W&B Python docs. It reads in the front matter of processed markdown files (processed by process_ask_markdown.py) and, based on the value specified for object_type, move files to "data-types", "launch-library", etc. directories.
4. Call create_landing_pages.py to generate the _index.md files

Directory Structure

• /wandb_sdk_docs/ - Temporary directory for initial lazydocs output
• /python/ - Final organized documentation with subdirectories:
  • /sdk/ - Core SDK functions and classes
  • /public-api/ - Query API documentation
  • /automations/ - Automation features
  • /data-types/ - Data type definitions
  • /custom-charts/ - Chart visualization tools