Skip to content

Commit

Permalink
Merge pull request #87 from maurerle/fix_lint
Browse files Browse the repository at this point in the history
Improve codebase by running linter checks
  • Loading branch information
rcschrg authored Sep 9, 2024
2 parents e0dffc8 + 8ea0832 commit 883d9cd
Show file tree
Hide file tree
Showing 67 changed files with 280 additions and 279 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/publish-mango.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,4 @@ jobs:
- name: Publish package
uses: pypa/gh-action-pypi-publish@v1
with:
password: ${{ secrets.PYPI_API_TOKEN }}
password: ${{ secrets.PYPI_API_TOKEN }}
4 changes: 2 additions & 2 deletions .github/workflows/test-mango.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ on: [push]
permissions:
contents: read

jobs:
jobs:
build:
runs-on: ubuntu-latest
strategy:
Expand Down Expand Up @@ -36,4 +36,4 @@ jobs:
run: |
source venv/bin/activate
coverage run -m pytest
coverage report
coverage report
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -105,4 +105,4 @@ docs/build/
docs/source/_build

# Git
*.orig
*.orig
2 changes: 1 addition & 1 deletion .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@ formats:
python:
version: "3.7"
install:
- requirements: docs/requirements.txt
- requirements: docs/requirements.txt
2 changes: 1 addition & 1 deletion docs/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ imagesize==0.7.1
Jinja2==2.9
MarkupSafe==0.23
Sphinx==4.0.2
sphinx-rtd-theme==0.5.2
sphinx-rtd-theme==0.5.2
1 change: 0 additions & 1 deletion docs/source/ACL messages.rst
Original file line number Diff line number Diff line change
@@ -1 +0,0 @@

8 changes: 4 additions & 4 deletions docs/source/agents-container.rst
Original file line number Diff line number Diff line change
Expand Up @@ -60,15 +60,15 @@ provided when :py:meth:`__init__()` of an agent is called.
Custom agents that inherit from the ``Agent`` class have to call ``super().__init__(container, suggested_aid: str = None)__``
on initialization.
This will register the agent at the provided container instance and will assign a unique agent id
(``self.aid``) to the agent. However, it is possible to suggest an aid by setting the variable ``suggested_aid`` to your aid wish.
The aid is granted if there is no other agent with this id, and if the aid doesn't interfere with the default aid pattern, otherwise
(``self.aid``) to the agent. However, it is possible to suggest an aid by setting the variable ``suggested_aid`` to your aid wish.
The aid is granted if there is no other agent with this id, and if the aid doesn't interfere with the default aid pattern, otherwise
the generated aid will be used. To check if the aid is available beforehand, you can use ``container.is_aid_available``.
It will also create the task to check for incoming messages.

***************
agent process
***************
To improve multicore utilization, mango provides a way to distribute agents to processes. For this, it is necessary to create and
To improve multicore utilization, mango provides a way to distribute agents to processes. For this, it is necessary to create and
register the agent in a slightly different way.

.. code-block:: python3
Expand All @@ -82,7 +82,7 @@ The process_handle is awaitable and will finish exactly when the process is full

Note that after the creation, the agent lives in a mirror container in another process. Therefore, it is not possible to interact
with the agent directly from the main process. If you want to interact with the agent after the creation, it is possible to
dispatch a task in the agent process using `dispatch_to_agent_process`.
dispatch a task in the agent process using `dispatch_to_agent_process`.

.. code-block:: python3
main_container.dispatch_to_agent_process(
Expand Down
21 changes: 9 additions & 12 deletions docs/source/codecs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ Codecs
Most of the codec related code is taken and adapted from aiomas:
https://gitlab.com/sscherfke/aiomas/

Codecs enable the container to encode and decode known data types to send them as messages.
Codecs enable the container to encode and decode known data types to send them as messages.
Mango already contains two codecs: A json serializer that can (recursively) handle any json serializable object and a protobuf codec
that will wrap an object into a generic protobuf message. Other codecs can be implemented by inheriting
from the ``Codec`` base class and implementing its ``encode`` and ``decode`` methods.
Codecs will only handle types explicitely known to them.
New known types can be added to a codec with the ``add_serializer`` method.
that will wrap an object into a generic protobuf message. Other codecs can be implemented by inheriting
from the ``Codec`` base class and implementing its ``encode`` and ``decode`` methods.
Codecs will only handle types explicitely known to them.
New known types can be added to a codec with the ``add_serializer`` method.
This method expects a type together with a serialization method and a deserialization method that translate the object into a format
the codec can handle (for example a json-serializable string for the json codec).

Expand Down Expand Up @@ -137,7 +137,7 @@ All that is left to do now is to pass our codec to the container. This is done d
**@json_serializable decorator**

In the above example we explicitely defined methods to (de)serialize our class. For simple classes, especially data classes,
we can achieve the same result (for json codecs) via the ``@json_serializable`` decorator. This creates the ``__asdict__``,
we can achieve the same result (for json codecs) via the ``@json_serializable`` decorator. This creates the ``__asdict__``,
``__fromdict__`` and ``__serializer__`` functions in the class:

.. code-block:: python3
Expand Down Expand Up @@ -178,10 +178,10 @@ provides the `codecs.FastJson` codec. This codec usese `msgspec` and does not pr
proto codec and ACLMessage
##########################

Serialization methods for the proto codec are expected to encode the object into a protobuf message object with the ``SerializeToString``
Serialization methods for the proto codec are expected to encode the object into a protobuf message object with the ``SerializeToString``
method.
The codec then wraps the message into a generic message wrapper, containing the serialized
protobuf message object and a type id.
The codec then wraps the message into a generic message wrapper, containing the serialized
protobuf message object and a type id.
This is necessary because in general the original type of a protobuf message can not be infered
from its serialized form.

Expand Down Expand Up @@ -280,6 +280,3 @@ process by making the proto type known to the codec using the ``register_proto_t
b'\x08\x01\x12\x0c\x12\nsome_bytes'
content: "some_bytes"
29 changes: 14 additions & 15 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,20 @@
#
import os
import sys
import sphinx_rtd_theme

# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
sys.path.insert(0, os.path.abspath('../../mango'))
sys.path.insert(0, os.path.abspath("../../mango"))

# -- Project information -----------------------------------------------------

project = 'mango'
copyright = '2023, mango team'
author = 'mango team'
project = "mango"
copyright = "2023, mango team"
author = "mango team"

# The full version, including alpha/beta/rc tags
release = '0.1'
release = "0.1"


# -- General configuration ---------------------------------------------------
Expand All @@ -35,17 +34,17 @@
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.doctest',
'sphinx.ext.intersphinx',
'sphinx.ext.graphviz',
'sphinx.ext.imgmath',
'sphinx_rtd_theme',
"sphinx.ext.autodoc",
"sphinx.ext.doctest",
"sphinx.ext.intersphinx",
"sphinx.ext.graphviz",
"sphinx.ext.imgmath",
"sphinx_rtd_theme",
]


# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
templates_path = ["_templates"]

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
Expand All @@ -58,9 +57,9 @@
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
html_theme = "sphinx_rtd_theme"

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
html_static_path = ["_static"]
2 changes: 1 addition & 1 deletion docs/source/customizing-container.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ A mango container can be customized regarding its way it connects to other conta

***************
connection type
***************
***************
12 changes: 6 additions & 6 deletions docs/source/development.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
Development Guidelines
======================

As we mainly work in a critical domain, we set great value on code quality not only to ensure correctness, but also to improve readability and maintainability. To reach this goal we have to set some standards regarding the development process and the test quality.
As we mainly work in a critical domain, we set great value on code quality not only to ensure correctness, but also to improve readability and maintainability. To reach this goal we have to set some standards regarding the development process and the test quality.

Quickstart
##########

In mango it is not possible to directly push on the branches *development* or *master*. Both branches are protected and changes can only be merged using a github pull-request. So when you work on a feature, the typical process would be to create a feature-branch. When you are finished you just have to create a merge-request, pass the CI/CD pipeline, make a maintainer review the changes, and you are ready to merge!
In mango it is not possible to directly push on the branches *development* or *master*. Both branches are protected and changes can only be merged using a github pull-request. So when you work on a feature, the typical process would be to create a feature-branch. When you are finished you just have to create a merge-request, pass the CI/CD pipeline, make a maintainer review the changes, and you are ready to merge!

CI/CD
#####
Expand All @@ -32,7 +32,7 @@ A unit test is a test for the smallest possible testable part of the code. This
Integration Tests
*****************

An integration test is everything what aims to test more than one unit.
An integration test is everything what aims to test more than one unit.

Coverage
*****************
Expand All @@ -42,12 +42,12 @@ We aim to reach a code coverage of > 90%. Currently, we measure the **statement*
Reviews
*****************

Tests are great but do not lead to better readability and maintainability. One part that will is the review process. In mango we came to the understanding that we want to review **every** change, which should be merged into the development branch. There are no exceptions to this. The idea is not only to check the code for errors, bad smells and security flaws, its part of generating a common understanding of good coding.
Tests are great but do not lead to better readability and maintainability. One part that will is the review process. In mango we came to the understanding that we want to review **every** change, which should be merged into the development branch. There are no exceptions to this. The idea is not only to check the code for errors, bad smells and security flaws, its part of generating a common understanding of good coding.

Linting
*****************

Another approach to improve the code quality is static code analysis, or better known as linting. Linting is an easy to set up possibility to make sure that a certain code standard is fulfilled. There are many useful rules, which can be checked automatically, so we have another line of defense and spare some time when reviewing.
Another approach to improve the code quality is static code analysis, or better known as linting. Linting is an easy to set up possibility to make sure that a certain code standard is fulfilled. There are many useful rules, which can be checked automatically, so we have another line of defense and spare some time when reviewing.

Formatting
*****************
Expand All @@ -57,4 +57,4 @@ The project is formatted using black + isort with default settings.
```bash
isort mango examples tests
black mango examples tests
```
```
2 changes: 0 additions & 2 deletions docs/source/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,5 +48,3 @@ __ https://mosquitto.org/
The protobuf codec is an optional feature that you need to explicity install if you need it.
**TODO: make protobuf optional**


2 changes: 1 addition & 1 deletion docs/source/migration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ mango 0.4.0 to 1.0.0
* The parameters create_acl and acl_metadata from 'send_message' has been removed: use send_acl_message instead
* The parameter mqtt_kwargs from 'send_message' has been removed: use kwargs instead
* The DateTimeScheduledTask has been removed: use TimestampScheduledTask instead
* The context and scheduler of an agent are no longer public: use the convenience methods for sending/scheduling or _context, _scheduler from within the agent
* The context and scheduler of an agent are no longer public: use the convenience methods for sending/scheduling or _context, _scheduler from within the agent
3 changes: 1 addition & 2 deletions docs/source/scheduling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ When using the scheduling another feature becomes available: suspendable tasks.
Dispatch Tasks to other Process
*******************************

As asyncio does not provide real parallelism to utilize multiple cores and agents may have tasks, which need a lot computational power, the need to dispatch certain tasks to other processes appear. Handling inter process communication manually is quite exhausting and having multiple process pools across different roles or agents leads to inefficient resource allocations. As a result mango offers a way to dispatch tasks, based on coroutine-functions, to other processes, managed by the framework.
As asyncio does not provide real parallelism to utilize multiple cores and agents may have tasks, which need a lot computational power, the need to dispatch certain tasks to other processes appear. Handling inter process communication manually is quite exhausting and having multiple process pools across different roles or agents leads to inefficient resource allocations. As a result mango offers a way to dispatch tasks, based on coroutine-functions, to other processes, managed by the framework.

Analogues to the normal API there are two different ways, first you create a ScheduledProcessTask and call ``schedule_process_task``, second you invoke the convnience methods with "process" in the name. These methods exists on any Agent, the RoleContext and the Scheduler.
In mango the following process tasks are available:
Expand Down Expand Up @@ -190,4 +190,3 @@ If you comment in the ExternalClock and change your main() as follows, the progr
clock.set_time(clock.time + 5)
await receiver.wait_for_reply
await c.shutdown()
24 changes: 12 additions & 12 deletions docs/source/tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ Introduction

This tutorial gives an overview of the basic functions of mango agents and containers. It consists of four
parts building a scenario of two PV plants, operated by their respective agents being directed by a remote
controller.
controller.

Each part comes with a standalone executable file. Subsequent parts either extend the functionality or simplify
some concept in the previous part.
Each part comes with a standalone executable file. Subsequent parts either extend the functionality or simplify
some concept in the previous part.

As a whole, this tutorial covers:
- container and agent creation
Expand Down Expand Up @@ -92,10 +92,10 @@ Now we can create our agents. Agents always live inside a container and this con
pv_agent_0 = PVAgent(pv_container)
pv_agent_1 = PVAgent(pv_container)
For now, our agents are purely passive entities. To make them do something, we need to send them a message. Messages are
For now, our agents are purely passive entities. To make them do something, we need to send them a message. Messages are
passed by the container via the ``send_message`` function always at least expects some content and a target address.
To send a message directly to an agent, we also need to provide its agent id which is set by the container when the agent
is created.
is created.

.. code-block:: python
Expand All @@ -120,7 +120,7 @@ This concludes the first part of our tutorial. If you run this code, you should
| Hello I am a PV agent! My id is agent0.
| Hello I am a PV agent! My id is agent1.
| Received message with content: Hello, this is a simple message. and meta {'network_protocol': 'tcp', 'priority': 0}.

.. raw:: html

Expand Down Expand Up @@ -170,7 +170,7 @@ The replies to feed_in requests and later the acknowledgements that a new maximu
We use the ``performative`` field of the ACL message to do this. We set the ``performative`` field to ``inform``
for feed_in replies and to ``accept_proposal`` for feed_in change acknowledgements. All messages between containers
are expected to be ACL Messages (or implement the split_content_and_meta function). Since we do not want to create
the full ACL object ourselves every time, within this example we always use the convenience method
the full ACL object ourselves every time, within this example we always use the convenience method
``container.send_acl_message``, which also supports setting the acl metadata.

.. code-block:: python
Expand Down Expand Up @@ -270,7 +270,7 @@ perform its active actions. We do this by implementing a ``run`` function with t
- send a feed_in request to each known pv agent
- wait for all pv agents to answer
- find the minimum reported feed_in
- send a maximum feed_in setpoint of this minimum to each pv agent
- send a maximum feed_in setpoint of this minimum to each pv agent
- again, wait for all pv agents to reply
- terminate

Expand Down Expand Up @@ -406,7 +406,7 @@ This example covers:
<summary><a>step by step</a></summary>

We want to use the types of custom message objects as the new mechanism for message typing. We define these
as simple data classes. For simple classes like this, we can use the ``json_serializable`` decorator to
as simple data classes. For simple classes like this, we can use the ``json_serializable`` decorator to
provide us with the serialization functionality.

.. code-block:: python
Expand Down Expand Up @@ -454,7 +454,7 @@ Next, we need to create a codec, make our message objects known to it, and pass
)
Any time the content of a message matches one of these types now the corresponding serialize and deserialize
functions are called. Of course, you can also create your own serialization and deserialization functions with
functions are called. Of course, you can also create your own serialization and deserialization functions with
more sophisticated behaviours and pass them to the codec. For more details refer to the ``codecs`` section of
the documentation.

Expand Down Expand Up @@ -507,7 +507,7 @@ you should receive the same output as in part 2:
Corresponding file: `v4_scheduling_and_roles.py`

In example 3, you restructured your code to use codecs for easier handling of typed message objects.
Now it is time to expand the functionality of our controller. In addition to setting the maximum feed_in
Now it is time to expand the functionality of our controller. In addition to setting the maximum feed_in
of the pv agents, the controller should now also periodically check if the pv agents are still reachable.

To achieve this, the controller should send a regular "ping" message to each pv agent that is in turn answered
Expand All @@ -527,7 +527,7 @@ Thus, things like message handlers that require container knowledge are introduc

This example covers:
- role API basics
- scheduling and periodic tasks
- scheduling and periodic tasks

.. raw:: html

Expand Down
2 changes: 1 addition & 1 deletion examples/distributed_clock/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,4 +13,4 @@ If the `clock_agent`s routine takes longer, the manager will wait for it to fini

Caution: it is needed, that all agents are connected before starting the manager

This example is tested with MQTT as well as by using TCP connection
This example is tested with MQTT as well as by using TCP connection
1 change: 0 additions & 1 deletion examples/distributed_clock/clock_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
from typing import TypedDict

import numpy as np

from mango import Role, RoleAgent, create_container
from mango.messages.message import Performatives
from mango.util.clock import ExternalClock
Expand Down
2 changes: 1 addition & 1 deletion examples/distributed_clock/clock_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ async def main(start):
if isinstance(clock, ExternalClock):
for i in tqdm(range(30)):
next_event = await clock_agent.distribute_time()
logger.info(f"current step: {clock.time}")
logger.info("current step: %s", clock.time)
await tasks_complete_or_sleeping(c)
# comment next line to see that the first message is not received in correct timings
# also comment sleep(0) in termination_detection to see other timing problems
Expand Down
Loading

0 comments on commit 883d9cd

Please sign in to comment.