uktrade · michalc · Sep 24, 2023 · Sep 24, 2023
diff --git a/.github/workflows/deploy-docs-to-github-pages.yml b/.github/workflows/deploy-docs-to-github-pages.yml
@@ -6,17 +6,21 @@ on:
       - main
 
 jobs:
+
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v4
-        with:
-          python-version: 3.11
-      - run: pip install -r requirements-docs.txt
-      - run: mkdocs build --site-dir _site
-      - name: Upload Pages artifact
-        uses: actions/upload-pages-artifact@v2
+    - name: Checkout
+      uses: actions/checkout@v3
+    - name: Setup Pages
+      uses: actions/configure-pages@v3
+    - name: Install dependencies
+      run: npm ci
+    - name: Build with Eleventy
+      run: npx eleventy
+    - name: Upload artifact
+      uses: actions/upload-pages-artifact@v2
+
   deploy:
     runs-on: ubuntu-latest
     needs: build

diff --git a/README.md b/README.md
@@ -13,7 +13,7 @@ To unZIP files on the fly try [stream-unzip](https://github.com/uktrade/stream-u
 <!-- --8<-- [start:features] -->
 ## Features
 
-In addition to being memory efficient (with some [limitations](https://stream-zip.docs.trade.gov.uk/getting-started/#limitations)) stream-zip:
+In addition to being memory efficient (with some [limitations](https://stream-zip.docs.trade.gov.uk/get-started/#limitations)) stream-zip:
 
 - Constructs ZIP files that can be stream unzipped, for example by [stream-unzip](https://stream-unzip.docs.trade.gov.uk/)
 

diff --git a/docs/advanced-usage.md b/docs/advanced-usage.md
@@ -1,3 +1,10 @@
+---
+layout: sub-navigation
+order: 3
+title: Advanced usage
+---
+
+
 ## Custom zlib options
 
 For the `ZIP_32` or `ZIP_64` methods, you can customise the compression object by overriding the default `get_compressobj` parameter, which is shown below.
@@ -7,7 +14,7 @@ for zipped_chunk in stream_zip(unzipped_files(), get_compressobj=lambda: zlib.co
     print(zipped_chunk)
 ```
 
-If you wish to disable compression entirely for these methods, you can pass `level=0` in the above. There is no way to customize the zlib object for the `ZIP_AUTO` method, other than passing `level` into it. See [Methods](methods.md) for details and other ways to not compress member files.
+If you wish to disable compression entirely for these methods, you can pass `level=0` in the above. There is no way to customize the zlib object for the `ZIP_AUTO` method, other than passing `level` into it. See [Methods](/methods/) for details and other ways to not compress member files.
 
 
 ## Custom chunk size

diff --git a/docs/assets/dit-logo.png b/docs/assets/dit-logo.png
diff --git a/docs/assets/dit-logo.svg b/docs/assets/dit-logo.svg
diff --git a/docs/async-interface.md b/docs/async-interface.md
@@ -1,4 +1,9 @@
-# Async Interface
+---
+layout: sub-navigation
+order: 7
+title: Async interface
+---
+
 
 stream-zip does not include an async interface. However, it is possible to construct an async function that wraps stream-zip to allow the construction of zip files in a streaming way from async code without blocking the event loop.
 

diff --git a/docs/contributing.md b/docs/contributing.md
@@ -1 +1,142 @@
---8<-- "CONTRIBUTING.md"
+---
+layout: sub-navigation
+order: 10
+title: How to contribute
+---
+
+In most cases to contribute you will need a [GitHub account](https://github.com/join).
+
+
+## Issues
+
+Suspected issues with stream-zip can be submitted at [the stream-unzip Issues page](https://github.com/uktrade/stream-zip/issues).
+
+An issue that contains a [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) stands the best chance of being resolved. However, it is understood that this is not possible in all circumstances.
+
+
+## Feature requests
+
+A feature request can be submitted using the [Ideas category in the stream-zip discussions](https://github.com/uktrade/stream-zip/discussions/categories/ideas).
+
+
+## Getting the source code
+
+To contribute changes to documentation or code, you will need the source of stream-unzip locally. The instructions for this depend on if you are a member of the [uktrade GitHub organisation](https://github.com/uktrade). In both cases, experience of working with source code, working on the command line, and working with git is helpful.
+
+### If a member of uktrade
+
+1. [Setup an SSH key and associate it with your GitHub account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)
+
+2. Clone the repository
+
+    ```bash
+    git clone [email protected]:uktrade/stream-zip.git
+    cd stream-zup
+    ```
+
+You should not fork the repository if you're a member of uktrade.
+
+### If not a member of uktrade
+
+1. [Setup an SSH key and associate it with your GitHub account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account).
+
+
+2. [Fork the repository](https://github.com/uktrade/stream-zip/fork). Make a note of the "Owner" that you fork to. This is usually your username.
+
+    There is more documentation on forking in [GitHub's guide on contributing to projects](https://docs.github.com/en/get-started/quickstart/contributing-to-projects).
+
+3. Clone the forked repository. In the following, replace `my-username` with the owner that you forked to in step 2.
+
+    ```bash
+    git clone [email protected]:my-username/stream-zip.git
+    cd stream-zup
+    ```
+
+## Documentation
+
+The source of the documentation is in the [docs/](https://github.com/uktrade/stream-zip/tree/main/docs) directory of the source code, and is written using [Material for mkdocs](https://squidfunk.github.io/mkdocs-material/).
+
+Changes are then submitted via a Pull Request (PR). To do this:
+
+1. Decide on a short hyphen-separated descriptive name for your change, prefixed with `docs/` for example `docs/add-django-recipe`.
+
+2. Make a branch using this descriptive name.
+
+    ```bash
+    git checkout -b docs/add-django-recipe
+    cd stream-zip
+    ```
+
+3. Make your changes in a text editor.
+
+4. Preview your changes locally.
+
+    ```bash
+    pip install -r requirements-docs.txt  # Only needed once
+    mkdocs serve
+    ```
+
+5. Commit your change and push to your fork. Ideally the commit message will follow the [Conventional Commit specification](https://www.conventionalcommits.org/).
+
+    ```bash
+    git add docs/recipies.md  # Repeat for each file changed
+    git commit -m "docs: added a Django recipe"
+    gir push origin docs/add-django-recipe
+    ```
+
+6. Raise a PR at [https://github.com/uktrade/stream-zip/pulls](https://github.com/uktrade/stream-zip/pulls) against the `main` branch in stream-zip.
+
+7. Wait for the PR to be approved and merged, and respond to any questions or suggested changes.
+
+When the PR is merged, the documentation is deployed automatically to [https://stream-zip.docs.trade.gov.uk/](https://stream-zip.docs.trade.gov.uk/).
+
+
+## Code
+
+To contribute most code changes:
+
+- Knowledge of Python is required. Python iterables, and specifically generators, are used heavily in stream-zip.
+
+- Understanding the low-level properties of the ZIP file format is required. These are covered in detail in the specification of the ZIP file format, known as [APPNOTE](https://support.pkware.com/home/pkzip/developer-tools/appnote).
+
+APPNOTE can be difficult to read, and contains a lot of information that isn't needed for stream-zip. A more concise introduction is in the [Wikipedia page on the ZIP file format](https://en.wikipedia.org/wiki/ZIP_(file_format)). However the Wikipedia page is less authoritative.
+
+In both APPNOTE and the Wikipedia page, the most relevant parts are about the "local file header" and the "data descriptor". These are sections of metadata that go before and after the contents of each file respectively.
+
+---
+
+Changes are then submitted via a Pull Request (PR). To do this:
+
+1. Decide on a short hyphen-separated descriptive name for your change, prefixed with the type of change. For example `fix/the-bug-description`.
+
+2. Make a branch using this descriptive name.
+
+    ```bash
+    git checkout -b fix-a-bug-description
+    ```
+
+3. Make sure you can run existing tests locally
+
+    ```bash
+    ./install-libarachive.sh             # Only needed once
+    pip install -r requirements-dev.txt  # Only needed once
+    pytest
+    ```
+
+4. Make your changes in a text editor. In the cases of changing behaviour, this would usually include changing or adding at least one test in [test_stream_zip.py](https://github.com/uktrade/stream-zip/blob/main/test_stream_zip.py), and running them.
+
+    ```bash
+    pytest
+    ```
+
+5. Commit your changes and push to your fork. Ideally the commit message will follow the [Conventional Commit specification](https://www.conventionalcommits.org/).
+
+    ```bash
+    git add stream_zip.py  # Repeat for each file changed
+    git commit -m "feat: the bug description"
+    gir push origin fix/the-bug-description
+    ```
+
+6. Raise a PR at [https://github.com/uktrade/stream-zip/pulls](https://github.com/uktrade/stream-zip/pulls) against the `main` branch in stream-zip.
+
+7. Wait for the PR to be approved and merged, and respond to any questions or suggested changes.
diff --git a/docs/exceptions.md b/docs/exceptions.md
@@ -1,4 +1,9 @@
-# Exceptions
+---
+layout: sub-navigation
+order: 8
+title: Exceptions
+---
+
 
 Exceptions raised by the source iterables are passed through the `stream_zip` function to client code unchanged. Other exceptions are in the `stream_zip` module and derive from `stream_zip.ZipError`.
 

diff --git a/docs/features.md b/docs/features.md
@@ -0,0 +1,19 @@
+---
+layout: sub-navigation
+order: 2
+title: Features
+---
+
+In addition to being memory efficient (with some [limitations](/get-started/#limitations)) stream-zip:
+
+- Constructs ZIP files that can be stream unzipped, for example by [stream-unzip](https://stream-unzip.docs.trade.gov.uk/)
+
+- Can construct Zip64 ZIP files. Zip64 ZIP files allow sizes far beyond the approximate 4GiB limit of the original ZIP format
+
+- Can construct ZIP files that contain symbolic links
+
+- Can construct ZIP files that contain directories, including empty directories
+
+- Allows the specification of permissions on the member files and directories (although not all clients respect them)
+
+- By default stores modification time as an extended timestamp. An extended timestamp is a more accurate timestamp than the original ZIP format allows
diff --git a/docs/getting-started.md → docs/get-started.md b/docs/getting-started.md → docs/get-started.md
@@ -1,3 +1,10 @@
+---
+layout: sub-navigation
+order: 1
+title: Get started
+---
+
+
 ## Prerequisites
 
 Python 3.6.7+
@@ -7,7 +14,7 @@ Python 3.6.7+
 
 You can install stream-zip from [PyPI](https://pypi.org/project/stream-zip/) using pip.
 
-```bash
+```shell
 pip install stream-zip
 ```
 
@@ -142,7 +149,7 @@ It is not required to have a directory member file in order to have files in tha
 
 ## Methods
 
-Each member file is compressed with a method that must be specified in client code. See [Methods](methods.md) for an explanation of each.
+Each member file is compressed with a method that must be specified in client code. See [Methods](/methods/) for an explanation of each.
 
 
 ## Stream unzipping
@@ -154,6 +161,6 @@ In general, not all valid ZIP files are possible to be stream unzipped. However,
 
 The `NO_COMPRESSION_32` and `NO_COMPRESSION_64` methods do not stream - they buffer the entire binary contents of the file in memory before output. They do this to calculate the length and CRC 32 to output them before the binary contents in the ZIP. This is required in order for ZIP to be stream unzippable.
 
-However, if you are able to calculate the length and CRC 32 ahead of time, you can pass `NO_COMPRESSION_32(uncompressed_size, crc_32)` or`NO_COMPRESSION_64(uncompressed_size, crc_32)` as the method. These methods do not buffer the binary contents in memory, and so are streaming methods. See [Methods](methods.md) for details of all supported methods.
+However, if you are able to calculate the length and CRC 32 ahead of time, you can pass `NO_COMPRESSION_32(uncompressed_size, crc_32)` or`NO_COMPRESSION_64(uncompressed_size, crc_32)` as the method. These methods do not buffer the binary contents in memory, and so are streaming methods. See [Methods](/methods/) for details of all supported methods.
 
 Note that even for the streaming methods, it's not possible to _completely_ stream-write ZIP files. Small bits of metadata for each member file, such as its name, must be placed at the _end_ of the ZIP. In order to do this, stream-zip buffers this metadata in memory until it can be output. This is likely to only to make a meaningful difference to memory usage for extremely high numbers of member files.
diff --git a/docs/index.md b/docs/index.md
@@ -1,9 +1,34 @@
---8<-- "README.md:intro"
+---
+homepage: true
+layout: product
+title: Construct ZIP files on the fly
+description: Use this Python package to output compressed ZIP files while receiving the uncompressed input
+startButton:
+  href: "get-started"
+  text: Get started
+---
 
-To unZIP files on the fly try [stream-unzip](https://stream-unzip.docs.trade.gov.uk/).
 
---8<-- "README.md:features"
+<div class="govuk-grid-row">
+  <section class="govuk-grid-column-one-third-from-desktop govuk-!-margin-bottom-7">
+    <h2 class="govuk-heading-m govuk-!-font-size-27">Memory</h2>
+    <p class="govuk-body">Only a small amount of data is kept in memory at any one time, with some <a href="/get-started/#limitations" class="govuk-link">limitations</a>.</p>
+  </section>
+  <section class="govuk-grid-column-one-third-from-desktop govuk-!-margin-bottom-7">
+    <h2 class="govuk-heading-m govuk-!-font-size-27">On-demand ZIPs</h2>
+    <p class="govuk-body">Generating ZIPs on-demand in a web server is a typical use case for stream-zip.</p>
+  </section>
+  <section class="govuk-grid-column-one-third-from-desktop govuk-!-margin-bottom-7">
+    <h2 class="govuk-heading-m govuk-!-font-size-27">Zip64</h2>
+    <p class="govuk-body">Optionally constructs Zip64 files, allowing sizes beyond the 4GiB limit of the original ZIP format.</p>
+  </section>
+</div>
 
----
+<hr class="govuk-section-break govuk-section-break--visible govuk-section-break--xl govuk-!-margin-top-0">
 
-Visit [Getting started](getting-started.md) to get started.
+<div class="govuk-grid-row">
+  <section class="govuk-grid-column-two-thirds">
+    <h2 class="govuk-heading-m govuk-!-font-size-27">Contributions</h2>
+    <p class="govuk-body">The code for stream-unzip is public and contributions are welcome though the <a class="govuk-link govuk-!-font-weight-bold" href="https://github.com/uktrade/stream-zip">stream-zip repository on GitHub</a>.</p>
+  </section>
+</div>
diff --git a/docs/input-examples.md b/docs/input-examples.md
@@ -1,4 +1,9 @@
-# Input examples
+---
+layout: sub-navigation
+order: 5
+title: Input examples
+---
+
 
 This page contains examples to show how files from different sources can be compressed into a ZIP using stream-zip. It is likely they will have to be modified for your use case.
 
@@ -32,4 +37,4 @@ zipped_chunks = stream_zip(local_files(names))
 
 ## Submit your own
 
-Pull requests (PRs) that propose changes to this page are especially welcome. PRs can be made at the [source of this page](https://github.com/uktrade/stream-zip/blob/main/docs/recipes.md). Submitting a PR requires a [GitHub account](https://github.com/join) and knowledge of the [GitHub fork and PR process](https://docs.github.com/en/pull-requests).
+Pull requests (PRs) that propose changes to this page are especially welcome. PRs can be made at the [source of this page](https://github.com/uktrade/stream-zip/blob/main/docs/input-examples.md). Submitting a PR requires a [GitHub account](https://github.com/join) and knowledge of the [GitHub fork and PR process](https://docs.github.com/en/pull-requests).
diff --git a/docs/methods.md b/docs/methods.md
@@ -1,4 +1,9 @@
-# Methods
+---
+layout: sub-navigation
+order: 4
+title: Methods
+---
+
 
 Each member file of a ZIP is compressed with one of the below methods.
 

diff --git a/docs/output-examples.md b/docs/output-examples.md
@@ -1,8 +1,13 @@
-# Output examples
+---
+layout: sub-navigation
+order: 6
+title: Output examples
+---
+
 
 This page contains examples to show how the output of stream-zip can be manipulated for different cases. It is likely you will have to modifiy them for your specific needs.
 
-The examples assume an iterable of member files is available in the variable `member_files`. See [Getting started](getting-started.md) for a guide on how to construct this, or [Input examples](input-examples.md) for examples.
+The examples assume an iterable of member files is available in the variable `member_files`. See [Get started](/get-started/) for a guide on how to construct this, or [Input examples](/input-examples/) for examples.
 
 
 ## Local file
@@ -83,4 +88,4 @@ s3.upload_fileobj(
 
 ## Submit your own
 
-Pull requests (PRs) that propose changes to this page are especially welcome. PRs can be made at the [source of this page](https://github.com/uktrade/stream-zip/blob/main/docs/recipes.md). Submitting a PR requires a [GitHub account](https://github.com/join) and knowledge of the [GitHub fork and PR process](https://docs.github.com/en/pull-requests).
+Pull requests (PRs) that propose changes to this page are especially welcome. PRs can be made at the [source of this page](https://github.com/uktrade/stream-zip/blob/main/docs/output-examples.md). Submitting a PR requires a [GitHub account](https://github.com/join) and knowledge of the [GitHub fork and PR process](https://docs.github.com/en/pull-requests).
diff --git a/docs/publish-a-release.md b/docs/publish-a-release.md
@@ -1,4 +1,9 @@
-# How to publish a release
+---
+layout: sub-navigation
+order: 11
+title: How to publish a release
+---
+
 
 Only mambers of the uktrade GitHub organisation may publish a release. If you are a member of the uktrade GitHub organsation: