Update docs to support markdown

docs/conf.py

@@ -33,11 +33,18 @@
 extensions = [
     "sphinx.ext.autodoc",  # for generating documentation from the docstrings in our code
     "sphinx.ext.napoleon",  # for parsing Numpy and Google stye docstrings
+    "myst_parser",  # For parsing markdown
 ]
 
 templates_path = ["_templates"]
 exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
+source_suffix = {
+    ".rst": "restructuredtext",
+    ".txt": "markdown",
+    ".md": "markdown",
+}
+
 
 # -- Options for HTML output -------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

docs/index.md

@@ -0,0 +1,168 @@
+IonBeam
+==========
+
+<p align="center">
+    <img alt="IonBeam logo, showing a stylized scientific instrument with ion streams." src="_static/ionbeam.png">
+</p>
+
+<p align="center">
+    <em>⚡️ A library for IoT data wrangling ⚡️</em>
+</p>
+
+---
+
+
+### 📡 What is IonBeam?
+
+**IonBeam** is a toolkit for **stream based processing of IoT observations**. It allows observational IoT data to be 
+1. 🔗 Ingested from many sources: REST APIs, MQTT brokers, file servers etc.
+2. 🔄 Transformed, cleaned, split, combined and anything else.
+3. 💾 Output into multiple storage formats and databases.
+
+Ionbeam uses a **message / action architecture**. Chunks of observations are represented by messages. Messages are routed through a series of actions which perform processing stesps before the final data is written out.
+
+---
+
+### 🚀 Features
+
+- Sources: REST APIs, MQTT brokers, file servers etc.
+- Support for both polling and event based data ingestion.
+- 
+- Transform **streams of messages** with actions before they reach their final destination.
+- Compatible with databases or external services for flexible deployment.
+
+---
+
+### 🛠️ Deployment Options
+
+IonBeam can be used in three ways:
+
+1. **Locally**, as a command line tool. See [Command Line Usage](#).
+2. **Docker-compose** setup, for local testing and development.
+3. (🔜 **Coming Soon**): Deployment via a Heml Chart to Kubernetes cluster.
+
+---
+
+## 💻 Dev Installation
+
+Install from source
+
+```sh
+git clone github.com/ecmwf-projects/IonBeam
+cd ionbeam
+```
+
+Create a conda or mamba environment, venv or similar
+
+```sh
+conda env create --name ionbeam ipykernel
+conda activate ionbeam
+pip install --editable ".[dev]"
+```
+
+Recommended: Install pre-commit hooks that run ruff, black, isort, flake8, etc on the code before you commit
+
+```sh
+pre-commit install
+```
+
+Create a `secrets.yaml` file containing the access credentials for the various sources of data, use `example_secrets.yaml` as a template. `secrets.yaml` is in the gitignore to lower the risk that it accidentally gets committed to git.
+
+## 📚 Documentation
+
+The documentation will be available at on readthedocs.io once this repo is made public. For now you can build it with sphinx.
+
+## 📜 License
+
+[Apache License 2.0](LICENSE) In applying this license, ECMWF does not waive the privileges and immunities
+granted to it by virtue of its status as an intergovernmental organisation nor does it submit to any jurisdiction.
+
+## ✅ Testing
+
+To run the default battery of smoke tests, just run pytest:
+
+```sh
+$ pytest
+```
+
+To run against live APIs use:
+```sh
+
+$ pytest -m network
+```
+
+## 📓 Running the Jupyter notebooks
+
+Setting up a jupyer lab server from scratch using conda or mamba:
+
+```sh
+# Make an environment for IonBeam
+# Do this at the root of the repository pwd=something/IonBeam/
+conda env create --name ionbeam ipykernel
+conda activate ionbeam
+pip install --editable ".[dev]"
+
+# Make an environment to run jupyter from, using separate ones is best practice
+conda env create --name jupyter jupyter nb_conda_kernels 
+conda activate jupyter
+jupyter lab
+```
+
+
+## 🖥️ Command Line Usage
+Currently the main way to interact with the pipeline is through the command line interface.
+```bash
+% python -m ionbeam --help
+usage: IonBeam [-h] [--validate-config] [-v] [-j [NUMBER]]
+                                       [--finish-after [NUMBER]]
+                                       config_file
+
+Put IOT data into the FDB
+
+positional arguments:
+  config_file           Path to a yaml config file.
+
+options:
+  -h, --help            show this help message and exit
+  --validate-config     Just parse the config and do nothing else.
+  -v, --verbose         Set the logging level, default is warnings only, -v and -vv
+                        give increasing verbosity
+  -j [NUMBER], --parallel [NUMBER]
+                        Engage parallel mode. Optionally specify how many parallel
+                        workers to use, defaults to the number of CPUs + the number of
+                        srcs in the pipeline
+  --finish-after [NUMBER]
+                        If present, limit the number of processed messages to 1 or the
+                        given integer
+```
+The `-vv` and `--finish-after` options are useful for debugging runs.
+
+See [this notebook](examples/notebooks/run_the_pipeline_manually.ipynb) for a walkthrough of assembling the pipeline from various components and then running it. This is useful for debugging the output at various stages in the pipeline.
+
+
+
+Some example heading
+========================
+
+```python
+def function(args, **kwargs) -> int:
+    return 4
+```
+
+
+.. toctree::
+   :maxdepth: 1
+   :glob:
+   :caption: Table of Contents:
+
+   quickstart
+   api_docs
+   examples/*
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/local_dev.sh

@@ -0,0 +1 @@
+sphinx-autobuild . _build/html

setup.cfg

@@ -65,6 +65,9 @@ dev =
 docs = 
     sphinx
     sphinx-rtd-theme
+    myst-parser
+    sphinx-autobuild
+
 
 [options.packages.find]
 where = src