Parallelize build of documentation. (apache#15062)

This is far more complex than it should be because of
autoapi problems with parallel execution. Unfortunately autoapi
does not cope well when several autoapis are run in parallel on
the same code - even if they are run in separate processes and
for different packages. Autoapi uses common _doctree and _api
directories generated in the source code and they override
each other if two or more of them run in parallel.

The solution in this PR is mostly applicable for CI environment.
In this case we have docker images that have been already built
using current sources so we can safely run separate docker
containers without mapping the sources and run generation
of documentation separtely and independently in each container.

This seems to work really well, speeding up docs generation
2x in public GitHub runners and 8x in self-hosted runners.

Public runners:

* 27m -> 15m

Self-hosted runners:

* 27m -> < 8m

(cherry picked from commit 741a545)

Author: potiuk

.github/workflows/ci.yml

@@ -463,12 +463,26 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
     env:
       RUNS_ON: ${{ fromJson(needs.build-info.outputs.runsOn) }}
       GITHUB_REGISTRY: ${{ needs.ci-images.outputs.githubRegistry }}
+      PYTHON_MAJOR_MINOR_VERSION: ${{needs.build-info.outputs.defaultPythonVersion}}
     steps:
       - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
         uses: actions/checkout@v2
         with:
           persist-credentials: false
           submodules: recursive
+      - name: "Setup python"
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{needs.build-info.outputs.defaultPythonVersion}}
+      - uses: actions/cache@v2
+        id: cache-venv-docs
+        with:
+          path: ./.docs-venv/
+          key: ${{ runner.os }}-docs-venv-${{ hashFiles('setup.py', 'setup.cfg') }}
+          restore-keys: |
+            ${{ runner.os }}-docs-venv-
+      - name: "Free space"
+        run: ./scripts/ci/tools/ci_free_space_on_ci.sh
       - name: "Prepare CI image ${{env.PYTHON_MAJOR_MINOR_VERSION}}:${{ env.GITHUB_REGISTRY_PULL_IMAGE_TAG }}"
         run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
       - name: "Build docs"

.gitignore

@@ -215,3 +215,8 @@ Chart.lock
 pip-wheel-metadata
 
 .pypirc
+/.docs-venv
+
+# Dev files
+/dev/packages.txt
+/dev/Dockerfile.pmc