Docs/SK-924 | Update local development guide (#657)

* Reorganize menu, remove local dev instructions from readme in git repo

* Rename distributed.rst to developer.rst

* Improved quickstart

* Updated project page

* Improved readibility a bit

* Remove Studio from the index (keep the file), most info is now in quickstart

* Make the APIClient page Studio-centric

* Less detailed instructions for setting up the Studio project

* Add instructions to clone the repo.

* Clarify that we only use the example from the local clone of fedn

* Fix path

* Remove Docker client instructions from quickstart

* Fixed typos and some broken refs

* Updated example readme in the github repo

* Fix subheadings in developer guide

* Remove local sandbox instructions from all examples.

---------

Co-authored-by: Andreas Hellander <[email protected]>

docs/developer.rst

@@ -1,14 +1,14 @@
 .. _developer-label:
 
-Local development
-=================
+Local development and deployment
+================================
 
 .. note::
    These instructions are for users wanting to set up a local development deployment of FEDn (i.e. without FEDn Studio).
    This requires practical knowledge of Docker and docker-compose. 
 
 Running the FEDn development sandbox (docker-compose)
-=====================================================
+------------------------------------------------------
 
 During development on FEDn, and when working on own aggregators/helpers, it is 
 useful to have a local development setup of the core FEDn services (controller, combiner, database, object store). 
@@ -45,8 +45,7 @@ To connect a native FEDn client, you need to make sure that the combiner service
 One way to achieve this is to edit your '/etc/hosts' and add a line '127.0.0.1  	combiner'. 
 
 Access message logs and validation data from MongoDB  
-====================================================
-
+------------------------------------------------------
 You can access and download event logs and validation data via the API, and you can also as a developer obtain 
 the MongoDB backend data using pymongo or via the MongoExpress interface: 
 
@@ -55,7 +54,7 @@ the MongoDB backend data using pymongo or via the MongoExpress interface:
 Username and password are found in 'docker-compose.yaml'. 
 
 Access global models   
-====================
+------------------------------------------------------
 
 You can obtain global model updates from the 'fedn-models' bucket in Minio: 
 
@@ -64,13 +63,13 @@ You can obtain global model updates from the 'fedn-models' bucket in Minio:
 Username and password are found in 'docker-compose.yaml'. 
 
 Reset the FEDn deployment   
-=========================
+------------------------------------------------------
 
 To purge all data from a deployment incuding all session and round data, access the MongoExpress UI interface and 
 delete the entire ``fedn-network`` collection. Then restart all services. 
 
 Clean up
-========
+------------------------------------------------------
 You can clean up by running 
 
 .. code-block::
@@ -79,7 +78,7 @@ You can clean up by running
 
 
 Connecting clients using Docker:
-================================
+------------------------------------------------------
 
 For convenience, we distribute a Docker image hosted on ghrc.io with FEDn preinstalled. For example, to start a client for the MNIST PyTorch example using Docker
 and FEDN 0.10.0, run this from the example folder:   
@@ -95,7 +94,7 @@ and FEDN 0.10.0, run this from the example folder:
 
 
 Self-managed distributed deployment
-===================================
+------------------------------------------------------
 
 You can use different hosts for the various FEDn services. These instructions shows how to set up FEDn on a **local network** using a single workstation or laptop as 
 the host for the servier-side components, and other hosts or devices as clients. 
@@ -160,9 +159,3 @@ Alternatively updating the `/etc/hosts` file, appending the following lines for
 
    <host local ip>      api-server
    <host local ip>      combiner
-
-
-Start a training session
-------------------------
-
-After connecting with your clients, you are ready to start training sessions from the host machine.

docs/projects.rst

@@ -4,7 +4,7 @@ Develop your own project
 ================================================
 
 This guide explains how a FEDn project is structured, and details how to develop your own
-projects for your own use-cases. 
+projects. 
 
 A FEDn project is a convention for packaging/wrapping machine learning code to be used for federated learning with FEDn. At the core, 
 a project is a directory of files (often a Git repository), containing your machine learning code, FEDn entry points, and a specification 
@@ -71,11 +71,12 @@ to specify the environment:
     1. Provide a ``python_env`` in the ``fedn.yaml`` file. In this case, FEDn will create an isolated virtual environment and install the project dependencies into it before starting up the client. FEDn currently supports Virtualenv environments, with packages on PyPI. 
     2. Manage the environment manually. Here you have several options, such as managing your own virtualenv, running in a Docker container, etc. Remove the ``python_env`` tag from ``fedn.yaml`` to handle the environment manually.  
 
-**Entry Points**
+Entry Points
+-------------
 
 There are up to four Entry Points to be specified.
 
-**Build Entrypoint (build, optional):**
+**build (optional):**
 
 This entrypoint is intended to be called **once** for building artifacts such as initial seed models. However, it not limited to artifacts, and can be used for any kind of setup that needs to be done before the client starts up.
 
@@ -85,16 +86,14 @@ To invoke the build entrypoint using the CLI:
     fedn build --
 
 
-
-**Startup Entrypoint (startup, optional):**
-
+**startup (optional):**
 
 This entrypoint is called **once**, immediately after the client starts up and the environment has been initalized. 
 It can be used to do runtime configurations of the local execution environment. For example, in the quickstart tutorial example, 
 the startup entrypoint invokes a script that downloads the MNIST dataset and creates a partition to be used by that client. 
 This is a convenience useful for automation of experiments and not all clients will specify such a script. 
 
-**Training Entrypoint (train, mandatory):** 
+**train (mandatory):** 
 
 This entrypoint is invoked every time the client recieves a new model update request. The training entry point must be a single-input single-output (SISO) program. It will be invoked by FEDn as such: 
 
@@ -105,7 +104,7 @@ This entrypoint is invoked every time the client recieves a new model update req
 where 'model_in' is the file containing the current global model to be updated, and 'model_out' is a path to write the new model update to.
 Download and upload of these files are handled automatically by the FEDn client, the user only specifies how to read and parse the data contained in them (see examples) . 
 
-**Validation Entrypoint (validate, optional):** 
+**validate (optional):** 
 
 The validation entry point works in a similar was as the trainig entrypoint. It can be used to specify how a client should validate the current global
 model on local test/validation data. It should read a model update from file, validate it (in any way suitable to the user), and write  a **json file** containing validation data:

examples/FedSimSiam/README.rst

@@ -68,4 +68,4 @@ In the figure below we can see that the kNN accuracy increases over the training
 indicating that the training of FedSimSiam is proceeding as intended. 
 
 .. image:: figs/fedsimsiam_monitoring.png
-   :width: 50%
+   :width: 50%

examples/flower-client/README.rst

@@ -47,10 +47,10 @@ a FEDn network. Here you have two main options: using FEDn Studio
 (recommended for new users), or a self-managed pseudo-distributed deployment
 on your own machine. 
 
-If you are using FEDn Studio (recommended):
+Using FEDn Studio:
 -------------------------------------------
 
-Follow instructions here to register for Studio and start a project: https://fedn.readthedocs.io/en/stable/studio.html.
+Follow instructions here to register for Studio and start a project: https://fedn.readthedocs.io/en/stable/quickstart.html.
 
 In your Studio project: 
 
@@ -73,47 +73,13 @@ Or, if you prefer to use Docker (this might take a long time):
    -v $PWD/client.yaml:/app/client.yaml \
    -e CLIENT_NUMBER=0 \
    -e FEDN_PACKAGE_EXTRACT_DIR=package \
-   ghcr.io/scaleoutsystems/fedn/fedn:0.9.0 run client -in client.yaml --secure=True --force-ssl
-
-
-If you are running FEDn in local development mode:
---------------------------------------------------
-
-Deploy a FEDn network on local host (see `https://fedn.readthedocs.io/en/stable/quickstart.html#local-development-deployment-using-docker-compose`). 
-
-Use the FEDn API Client to initalize FEDn with the compute package and seed model: 
-
-.. code-block::
-
-   python init_fedn.py
-
-Create a file 'client.yaml' with the following content: 
-
-.. code-block::
-   
-   network_id: fedn-network
-   discover_host: api-server
-   discover_port: 8092
-   name: myclient
-
-Then start the client (using Docker)
-
-.. code-block::
-
-   docker run \
-   -v $PWD/client.yaml:/app/client.yaml \
-   --network=fedn_default \
-   -e CLIENT_NUMBER=0 \
-   -e FEDN_PACKAGE_EXTRACT_DIR=package \
-   ghcr.io/scaleoutsystems/fedn/fedn:0.9.0 run client -in client.yaml
-
+   ghcr.io/scaleoutsystems/fedn/fedn:0.11.1 run client -in client.yaml --secure=True --force-ssl
 
 Scaling to multiple clients
 ------------------------------------------------------------------
 
-To scale the experiment with additional clients on the same host, execute the run command
-again from another terminal. If running from another host, add another 'client.yaml', install 
-fedn, and execute the run command. In both cases inject a client number as an environment 
+To scale the experiment with additional clients on the same host, generate another 'client.yaml' and execute the run command
+again from another terminal. Inject a client number as an environment 
 varible which is used for distributing data (see 'flwr_task.py').
 
 For Unix Operating Systems:

examples/monai-2D-mednist/README.rst

@@ -19,11 +19,6 @@ Using FEDn Studio:
 -  `Python 3.8, 3.9, 3.10 or 3.11 <https://www.python.org/downloads>`__
 -  `A FEDn Studio account <https://fedn.scaleoutsystems.com/signup>`__
 
-If using pseudo-distributed mode with docker-compose:
-
--  `Docker <https://docs.docker.com/get-docker>`__
--  `Docker Compose <https://docs.docker.com/compose/install>`__
-
 
 Creating the compute package and seed model
 -------------------------------------------
@@ -74,11 +69,10 @@ below command we divide the dataset into 10 parts.
     python prepare_data.py 10
 
 
-
 Using FEDn Studio
 -----------------
 
-Follow the guide here to set up your FEDn Studio project and learn how to connect clients (using token authentication): `Studio guide <https://fedn.readthedocs.io/en/stable/studio.html>`__.
+Follow the guide here to set up your FEDn Studio project and learn how to connect clients (using token authentication): `Studio guide <https://fedn.readthedocs.io/en/stable/quickstart.html>`__.
 On the step "Upload Files", upload 'package.tgz' and 'seed.npz' created above.
 
 Connecting clients:
@@ -110,83 +104,4 @@ For convenience, there is a Docker image hosted on ghrc.io with fedn preinstalle
      -e FEDN_DATA_PATH=/app/data/ \
      -e FEDN_CLIENT_SETTINGS_PATH=/app/client_settings.yaml \
      -e FEDN_DATA_SPLIT_INDEX=0 \
-     ghcr.io/scaleoutsystems/fedn/fedn:0.9.0 run client -in client.yaml --force-ssl --secure=True
-
-
-**NOTE: The following instructions are only for SDK-based client communication and for local development environments using Docker.**
-
-
-Local development mode using Docker/docker compose
---------------------------------------------------
-
-Follow the steps above to install FEDn, generate 'package.tgz' and 'seed.tgz'.
-
-Start a pseudo-distributed FEDn network using docker-compose:
-
-.. code-block::
-
-   docker compose \
-    -f ../../docker-compose.yaml \
-    -f docker-compose.override.yaml \
-    up
-
-This starts up local services for MongoDB, Minio, the API Server, one Combiner and two clients.
-You can verify the deployment using these urls:
-
-- API Server: http://localhost:8092/get_controller_status
-- Minio: http://localhost:9000
-- Mongo Express: http://localhost:8081
-
-Upload the package and seed model to FEDn controller using the APIClient. In Python:
-
-.. code-block::
-
-   from fedn import APIClient
-   client = APIClient(host="localhost", port=8092)
-   client.set_active_package("package.tgz", helper="numpyhelper")
-   client.set_active_model("seed.npz")
-
-You can now start a training session with 5 rounds (default):
-
-.. code-block::
-
-   client.start_session()
-
-Automate experimentation with several clients
-=============================================
-
-If you want to scale the number of clients, you can do so by modifying ``docker-compose.override.yaml``. For example,
-in order to run with 3 clients, change the environment variable ``FEDN_NUM_DATA_SPLITS`` to 3, and add one more client
-by copying ``client1``.
-
-
-Access message logs and validation data from MongoDB  
-====================================================
-
-You can access and download event logs and validation data via the API, and you can also as a developer obtain 
-the MongoDB backend data using pymongo or via the MongoExpress interface: 
-
-- http://localhost:8081/db/fedn-network/ 
-
-The credentials are as set in docker-compose.yaml in the root of the repository. 
-
-Access global models   
-====================
-
-You can obtain global model updates from the 'fedn-models' bucket in Minio: 
-
-- http://localhost:9000
-
-Reset the FEDn deployment   
-=========================
-
-To purge all data from a deployment incuding all session and round data, access the MongoExpress UI interface and 
-delete the entire ``fedn-network`` collection. Then restart all services. 
-
-Clean up
-========
-You can clean up by running 
-
-.. code-block::
-
-   docker-compose -f ../../docker-compose.yaml -f docker-compose.override.yaml down -v
+     ghcr.io/scaleoutsystems/fedn/fedn:0.11.1 run client -in client.yaml --force-ssl --secure=True