Merge pull request #104 from OFFIS-DAI/feature-api-transfer

Draft: API overhaul, mango 2.0.0

.github/workflows/test-mango.yml

@@ -18,7 +18,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.10", "3.11", "3.12"]
     steps:
     - uses: actions/checkout@v4
     - name: Set up Python
@@ -32,6 +32,8 @@ jobs:
         pip install virtualenv
         virtualenv venv
         source venv/bin/activate
+        pip3 install -U sphinx
+        pip3 install -r docs/requirements.txt
         pip3 install -r requirements.txt
         pip3 install -e .
         sudo apt update
@@ -44,6 +46,10 @@ jobs:
         source venv/bin/activate
         ruff check .
         ruff format --check .
+    - name: Doctests
+      run: |
+        source venv/bin/activate
+        make -C docs doctest
     - name: Test+Coverage
       run: |
         source venv/bin/activate

docs/source/agents-container.rst

@@ -12,58 +12,153 @@ serialization and deserialization of messages.
 Container also help to to speed up message exchange between agents that run on the same physical hardware,
 as data that is exchanged between such agents will not have to be sent through the network.
 
-In mango, a container is created using the classmethod ``mango.create_container``:
+In mango, a container is created using factory methods:
+
+* :meth:`mango.create_tcp_container`
+* :meth:`mango.create_mqtt_container`
+* :meth:`mango.create_ec_container`
+
+Most of the time, the tcp container should be the default choice if you wanna create simulations, which run in real time using
+a simple but fast network protocol.
 
 .. code-block:: python3
 
-    @classmethod
-    async def create_container(cls, *, connection_type: str = 'tcp', codec: Codec = None, clock: Clock = None,
-                      addr: Optional[Union[str, Tuple[str, int]]] = None,
-                      proto_msgs_module=None,
-                      **kwargs):
+    def create_tcp_container(
+        addr: str | tuple[str, int],
+        codec: Codec = None,
+        clock: Clock = None,
+        copy_internal_messages: bool = False,
+        **kwargs: dict[str, Any],
+    ) -> Container:
+
+The factory methods are asyncio-free, meaning most of the time you can create containers without a running asyncio loop.
 
-The factory method is a coroutine, so it has to be scheduled within a running asyncio loop.
 A simple container, that uses plain tcp for message exchange can be created as follows:
 
-.. code-block:: python3
+.. testcode::
 
     import asyncio
-    from mango import create_container
+    from mango import create_tcp_container
 
-    async def get_simple_container():
-        container = await create_container(addr=('localhost', 5555))
+    def get_simple_container():
+        container = create_tcp_container(addr=('localhost', 5555))
         return container
 
-    simple_container = asyncio.run(get_simple_container()))
+    print(get_simple_container().addr)
+
+.. testoutput::
+
+    ('localhost', 5555)
+
+The container type depends totally on the factory method you invoke. Every supported type has its own class backing
+the functionality.
 
-A container can be parametrized regarding its connection type ('tcp' or 'MQTT') and
-regarding the codec that is used for message serialization.
 The default codec is JSON (see section :doc:`codecs` for more information). It is also possible to
-define the clock that an agents scheduler should use (see section scheduling).
+define the clock that an agents scheduler should use (see page :doc:`scheduling`).
+
+Note, that container creation is different from container starting. Before you can work with a container
+you will want to register Agents, and then start (or activate) the container. This shall be done using an
+asynchronous context manager, which we provide by invoking :meth:`mango.activate`.
+
+.. testcode::
+
+    import asyncio
+    from mango import create_tcp_container, activate
+
+    async def start_container():
+        container = create_tcp_container(addr=('localhost', 5555))
+
+        async with activate(container) as c:
+            print("The container is activated now!")
+            await asyncio.sleep(0.1) # activate the container for 0.1 seconds, most of the time you want to include e.g. a condition to await
+        print("The container is automatically shut down, even on exceptions!")
+
+    asyncio.run(start_container())
+
+.. testoutput::
 
-After a container is created, it is waiting for incoming messages on the given address.
-As soon as the container has some agents, it will distribute incoming messages
-to the corresponding agents and allow agents to send messages to other agents.
+    The container is activated now!
+    The container is automatically shut down, even on exceptions!
 
-At the end of its lifetime, a ``container`` should be shutdown by using the method ``shutdown()``.
-It will then shutdown all agents that are still running
-in this container and cancel running tasks.
+At the end of its lifetime, a ``container`` the container will shutdown. This will be done by the context manager, so no need for the
+user to worry about it. This will also shutdown all agents that are still running in this container and cancel running tasks.
 
 ***************
 mango agents
 ***************
 mango agents can be implemented by inheriting from the abstract class ``mango.Agent``.
-This class provides basic functionality such as to register the agent at the container or
-to constantly check the inbox for incoming messages.
-Every agent lives in exactly one container and therefore an instance of a container has to be
-provided when :py:meth:`__init__()` of an agent is called.
-Custom agents that inherit from the ``Agent`` class have to call ``super().__init__(container, suggested_aid: str = None)__``
-on initialization.
-This will register the agent at the provided container instance and will assign a unique agent id
-(``self.aid``) to the agent. However, it is possible to suggest an aid by setting the variable ``suggested_aid`` to your aid wish.
+This class provides basic functionality such as to scheduling convenience methods or to constantly check the inbox for incoming messages.
+Every agent can live in exactly one container, to register an agent the method :meth:`mango.Container.register` can be used. This method will assign
+the agent a generated agent id (aid) and enables the agent scheduling feature.
+
+However, it is possible to suggest an aid by setting the parameter ``suggested_aid`` of :meth:`mango.Container.register` to your aid wish.
 The aid is granted if there is no other agent with this id, and if the aid doesn't interfere with the default aid pattern, otherwise
 the generated aid will be used. To check if the aid is available beforehand, you can use ``container.is_aid_available``.
-It will also create the task to check for incoming messages.
+
+Note that, custom agents that inherit from the ``Agent`` class have to call ``super().__init__()__`` on initialization.
+
+.. testcode::
+
+    from mango import Agent, create_tcp_container
+
+    class MyAgent(Agent):
+        pass
+
+    async def create_and_register_agent():
+        container = create_tcp_container(addr=('localhost', 5555))
+
+        agent = container.register(MyAgent(), suggested_aid="CustomAgent")
+        return agent
+
+    print(asyncio.run(create_and_register_agent()).aid)
+
+.. testoutput::
+
+    CustomAgent
+
+Further there are some important lifecycle methods you often want to implement:
+
+* :meth:`mango.Agent.on_ready`
+   * Called when all containers have been activated during the activate call, which started the container the agent is registered in.
+   * At this point all relevant containers have been started and the agent is already registered. This is the correct method for starting to send messages, even to other containers.
+* :meth:`mango.Agent.on_register`
+   * Called when the Agent just has been registered.
+   * At this point the scheduler is initialized and the agent address is known, but no communication can happen yet.
+* :meth:`mango.Agent.on_start`
+   * Called when the container of the agent has been started during activation.
+   * At this point internal communication is possible and depending on your setup external communication could be done too.
+
+Besides the lifecycle, one of the main functions implemented in Agents are message exchange function. For this part read :doc:`/message exchange`.
+
+*********************************
+Express setup of mango simulation
+*********************************
+
+It is not necessary to create the container all by yourself, as you often want to just distribute some agents evenly to a number of containers. This can be done
+with an asynchronous context manager created by :meth:`mango.run_with_tcp` (:meth:`mango.run_with_mqtt` for MQTT protocol). This method just expects the number of containers
+you want to start and the agents, which shall run in these containers.
+
+With this method sending a message to an agent in another container looks like this:
+
+.. testcode::
+
+    import asyncio
+    from mango import PrintingAgent, run_with_tcp
+
+    async def run_with_tcp_example():
+        agent_tuple = (PrintingAgent(), dict(aid="MyAgent"))
+        single_agent = PrintingAgent()
+
+        async with run_with_tcp(2, agent_tuple, single_agent) as cl:
+            # cl is the list of containers, which are created internally
+            await agent_tuple[0].send_message("Hello, print me!", single_agent.addr)
+            await asyncio.sleep(0.1)
+
+    asyncio.run(run_with_tcp_example())
+
+.. testoutput::
+
+    Received: Hello, print me! with {'sender_id': 'MyAgent', 'sender_addr': ['127.0.0.1', 5555], 'receiver_id': 'agent0', 'network_protocol': 'tcp', 'priority': 0}
 
 ***************
 agent process
@@ -74,12 +169,10 @@ register the agent in a slightly different way.
 .. code-block:: python3
 
     process_handle = await main_container.as_agent_process(
-        agent_creator=lambda sub_container: TestAgent(
-            container, aid_main_agent, suggested_aid=f"process_agent1"
-        )
+        agent_creator=lambda sub_container: sub_container.register(MyAgent(), suggested_aid=f"process_agent1")
     )
 
-The process_handle is awaitable and will finish exactly when the process is fully set up. Further, it contains the pid `process_handle.pid`.
+The ``process_handle`` is awaitable and will finish exactly when the process is fully set up. Further, it contains the pid ``process_handle.pid``.
 
 Note that after the creation, the agent lives in a mirror container in another process. Therefore, it is not possible to interact
 with the agent directly from the main process. If you want to interact with the agent after the creation, it is possible to
@@ -89,6 +182,6 @@ dispatch a task in the agent process using `dispatch_to_agent_process`.
 
     main_container.dispatch_to_agent_process(
         pid,
-        your_function, # will be called with the mirror container + x as arguments
+        your_function, # will be called with the mirror container + varargs as arguments
         ... # varargs, additional arguments you want to pass to your_function
     )

docs/source/api_ref/index.rst

@@ -2,17 +2,30 @@
 API reference
 =============
 
-The API reference provides detailed descriptions of mango's classes and
+The API reference provides detailed descriptions of the mango's classes and
 functions.
 
-Subpackages
------------
+.. automodule:: mango
+   :members:
+   :undoc-members:
+   :imported-members:
+   :inherited-members:
+
+
+.. note::
+   Note that, most classes and functions described in the API reference
+   should be imported using `from mango import ...`, as the stable and public API
+   generally will be available by using `mango` and the internal module structure
+   might change, even in minor releases.
+
+By subpackages
+---------------
 
 .. toctree::
    :maxdepth: 2
 
+   mango.express
    mango.agent
    mango.container
    mango.messages
-   mango.modules
    mango.util

docs/source/api_ref/mango.agent.rst

@@ -1,15 +1,15 @@
-mango.agent package
+Agents API
 ===================
 
-mango.agent.core module
+Agent core
 -----------------------
 
 .. automodule:: mango.agent.core
    :members:
    :undoc-members:
    :show-inheritance:
 
-mango.agent.role module
+Roles
 -----------------------
 
 .. automodule:: mango.agent.role