Skip to content

Commit

Permalink
Add experimental multi.py WRITEME front end
Browse files Browse the repository at this point in the history
  • Loading branch information
DavidSouther committed Sep 11, 2023
1 parent bef5ee6 commit 23e86e7
Show file tree
Hide file tree
Showing 2 changed files with 148 additions and 17 deletions.
79 changes: 62 additions & 17 deletions .doc_gen/readmes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ This is an internal tool intended for use only by the AWS code examples team.

## Prerequisites

You must have a recent version of Python installed to run this tool,
You must have a recent version of Python installed to run this tool,
and a recent version of pip (Python's package manager) to install the
required packages.

### Install packages

We recommend a virtual environment. Create a virtual environment
We recommend a virtual environment. Create a virtual environment
and install packages by running the following commands in the
`.doc_gen/readmes` folder:

Expand Down Expand Up @@ -53,35 +53,35 @@ This creates a README.md file in the `python/example_code/s3` folder.

### Parameters

* `language` must match a top-level language in sdks.yaml.
* `version` must be defined for the language in sdks.yaml.
* `service` must match a top-level service in services.yaml.
* `--safe` (optional) when specified, the existing README.md is renamed to the
`saved_readme` value in config.py (such as README.old.md).
* `--svc_folder` (optional) overrides the output folder for the README.
- `language` must match a top-level language in sdks.yaml.
- `version` must be defined for the language in sdks.yaml.
- `service` must match a top-level service in services.yaml.
- `--safe` (optional) when specified, the existing README.md is renamed to the
`saved_readme` value in config.py (such as README.old.md).
- `--svc_folder` (optional) overrides the output folder for the README.

You can get inline usage info by using the `-h` flag:

```
python .doc_gen/readmes/writeme.py -h
```
```

### Configuration

Additional configuration is kept in `config.py`.

* `entities` is a dictionary of entities that are not otherwise defined in
services.yaml.
* `language` is a dictionary of language and version for each SDK. Fields are:
* `base_folder` the root folder for the SDK version.
* `service_folder` a Jinja template of the service folder for the SDK version.
- `entities` is a dictionary of entities that are not otherwise defined in
services.yaml.
- `language` is a dictionary of language and version for each SDK. Fields are:
- `base_folder` the root folder for the SDK version.
- `service_folder` a Jinja template of the service folder for the SDK version.
This might not work in all cases. If not, use the `--svc_folder` override.
* `sdk_api_ref` a Jinja template of the SDK API Reference topic for the SDK version.
- `sdk_api_ref` a Jinja template of the SDK API Reference topic for the SDK version.
This is used to create the link to the reference page in the Additional Resources
section, such as to the Boto3 S3 reference page for Python. This is a best effort,
and if the generated link is wrong, you can update it manually. On subsequent runs
of WRITEME, the existing link is kept.
of WRITEME, the existing link is kept.

### Custom content

Custom content can be per-SDK or per-README.
Expand All @@ -106,3 +106,48 @@ the following:
Any content you add within these comments is preserved in subsequent generations
of the README. Do not change the names of these comments or remove them. Keep them
empty if you don't need custom content.

## Generate multiple READMEs (EXPERIMENTAL)

[`multi.py`](multi.py) executes the WRITEME logic across a set of languages,
versions, and services.

```
python .doc_gen/readmes/multi.py --language <language1>:<version> <language2>:<version> --service <service1> <service2>
```

For example, to generate S3 and STS READMEs for Python sdk version 3 and Go sdk version 2:

```
python .doc_gen/readmes/writeme.py --languages Python:3 Go:2 --services s3 sts
```

This creates the README.md files in `python/example_code/s3` and other folders folder.

To build all READMEs for Rust:

```
$ python .doc_gen/readmes/writeme.py --languages Rust:1
INFO:root:Dry run, no changes will be made.
DEBUG:root:Rendering Rust:1:acm
DEBUG:root:Rendering Rust:1:api-gateway
DEBUG:root:Rendering Rust:1:apigatewaymanagementapi
# ...
DEBUG:root:Rendering Rust:1:transcribe-medical
DEBUG:root:Rendering Rust:1:translate
```

> Note that at this time, there is no way to specify a `svc_folder` override for the SDKs! This is the main reason this tool is still experimental.
### Parameters

- `--languages` a list of `<language>:<sdk_version>` pairs. Each pair must be valid as defined for the languages in [`sdks.yaml`](../metadata/sdks.yaml).
- `--services` a list where each must match a top-level service in [`services.yaml`](../metadata/services.yaml).
- `--safe` (optional) when specified, the existing README.md is renamed to the `saved_readme` value in config.py (such as README.old.md).
- `--dry_run` (default True) because the tool is experimental, you must explicitly opt in with `--no_dry_run` to execute today. This will change in the future.

You can get inline usage info by using the `-h` flag:

```
python .doc_gen/readmes/multi.py -h
```
86 changes: 86 additions & 0 deletions .doc_gen/readmes/multi.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0

import argparse
import config
import logging
from scanner import Scanner
from render import Renderer


def main():
scanner = Scanner(".doc_gen/metadata")
sdks = scanner.sdks()
lang_vers = []
for sdk in sdks:
for v in sdks[sdk]["sdk"]:
lang_vers.append(f"{sdk}:{v}")

parser = argparse.ArgumentParser()
parser.add_argument(
"--languages",
choices=lang_vers + ["all"],
nargs="+",
help="The languages of the SDK. Choose from: %(choices)s.",
default=["all"],
)
parser.add_argument(
"--services",
choices={**scanner.services(), "all": {}},
nargs="+",
help="The targeted service. Choose from: %(choices)s.",
default=["all"],
)
parser.add_argument(
"--safe",
action="store_true",
help=f"Save a copy of the original README as the 'saved_readme' value specified in config.py ({config.saved_readme}).",
)
parser.add_argument(
"--verbose",
action="store_true",
help="When set, output verbose debugging info.",
)
parser.add_argument(
"--dry_run",
action="store_true",
help="This tool is in development. You must pass --dry_run=false to have it run.",
default=True, # Change this to default false when we're ready to use this generally.
)
parser.add_argument("--no_dry_run", dest="dry_run", action="store_false")
args = parser.parse_args()

if "all" in args.languages:
args.languages = lang_vers

if "all" in args.services:
args.services = [*scanner.services().keys()]

if args.verbose:
logging.basicConfig(level=logging.DEBUG)

logging.debug(f"Args configuration: {args}")

if args.dry_run:
logging.info("Dry run, no changes will be made.")

for language_and_version in args.languages:
(language, version) = language_and_version.split(":")
if int(version) not in sdks[language]["sdk"]:
logging.debug(f"Skipping {language}:{version}")
else:
for service in args.services:
try:
scanner.set_example(language, service)
logging.debug(f"Rendering {language}:{version}:{service}")
if not args.dry_run:
Renderer(scanner, int(version), args.safe).render()
except Exception as err:
logging.error(
f"Exception rendering {language}:{version}:{service} - {err}",
)
raise err


if __name__ == "__main__":
main()

0 comments on commit 23e86e7

Please sign in to comment.