Architecture Backend
The backend consists of multiple components:
- Flask
- Workload Generator
- Database Manager
- InfluxDB
All these components are running in separate Processes and are communicating via interprocess communication with each other. The workload generator implements the logic for creating a configurable number of queries per second from a predefined query set. By default, all queries of a workload are chosen with the same probability. Afterward, the workload generator passes the queries to the database objects (managed by the database manager), which maintain query queues (implemented with Python’s multiprocessing library) and handle the actual query sending with psycopg2. We utilize ZeroMQ’s publisher-subscriber pattern for efficient inter-process communication. Flask handles the communication between the backend and frontend. Furthermore, the time-series database InfluxDB is used to store the relevant metrics permanently.
Before having a look at the code you should have basic knowledge in python. The backend uses the following important libraries:
- zeromq: https://zguide.zeromq.org/docs/chapter1/ (chapter 1 to 3)
- psycopg: https://www.psycopg.org/
- flask: https://flask.palletsprojects.com/en/1.1.x/
- flask Restx: https://flask-restx.readthedocs.io/en/latest/
- marshmallow: https://marshmallow.readthedocs.io/en/stable/
- flask_accepts: https://github.com/apryor6/flask_accepts
- Python and InfluxDB: https://www.influxdata.com/blog/getting-started-python-influxdb/
The flask component uses for every namespace (for example /metrics) the following files:
- controller.py : The controller is responsible for coordinating Flask routes, services, and schemas. On top, the controller is also responsible to provide the documentation for Swagger. To provide the documentation for swagger the controller uses the flask_accepts library in connection with schemas (also used to serialize and deserialize objects) from marshmallow. The controller uses also models (The model is where the entity itself is defined in a Python representation)for type safety. All the requests are delegated to the service.
- model.py: A model is used for type safety and to represent an entity itself as a Python representation (object). If we for example getting a response from the database manager or influxdb we convert this response, which is often a JSON, to a Python object. These objects are defined by the models. These objects are often composed of other objects.
- schema.py: A schema is responsible for the serialization/deserialization of a given entity. An entity can be a response from the influxdb or the database manager. This response can then be deserialized (load) into a python entity (model). On the other hand a python entity (model) can be serialized (dump) to a JSON-encoded string. For the deserialization, the post_load decorator is used. For the schemas, we are using the marshmallow library. The schemas are also used by the controller for documentation.
- service.py : The service is responsible for interacting with the entity. This includes fetching data from the influx or database manager. The data is then if needed deserialized into a Python entity (model) by using the corresponding schemas.
The main libraries used are:
- marshmallow (for schemas) (https://marshmallow.readthedocs.io/en/stable/quickstart.html)
- flask restx (https://flask-restx.readthedocs.io/en/latest/)
- flask_accepts (amongst other things for creating documentation from schemas) (https://github.com/apryor6/flask_accepts)
The Database manager manages the database objects. Every database object represents a Hyrise instance and only this object communicates with it. If the manager gets a request from the Flask application, for example, to activate a plugin, it delegates this request to the right database object. The database object itself then sends a request to the Hyrise database.
The workers are the central part to run a workload on the Hyrise databases. There are two kinds of worker:
- queue_worker
- task_worker
Every database object has one worker pool consisting of one queue worker and n task workers. Every worker runs in its own Process. Since the python interpreter can only execute one Thread per time, it relies on Processes for parallel code execution (https://stackoverflow.com/questions/3044580/multiprocessing-vs-threading-python). The queue worker subscribes to the Zeromq socket of the workload generator (publisher) (https://zguide.zeromq.org/docs/chapter2/#Multipart-Messages). If it gets some tasks (list of SQL queries) from the publisher it places them in a thread save multiprocessing queue (https://docs.python.org/3.8/library/multiprocessing.html). The task workers are then extracting a query from the multiprocessing queue and executing it. The task worker is then saving the latency of the queries every second to the influx DB.
For all connections to the influx DB and the Hyries database, the cockpit is using self-defined cursor objects hyrisecockpit/database_manager/cursor.py. These are just wrappers around psycopg2 and InfluxDBClient objects. cursor.py also provides factories for these cursors. The database object database.py initializes a ConnectionFactory (for the Hyrise database) and a StorageConnectionFactory for the influx DB. All jobs can then use these factories to create a cursor when they need one. All connection information is saved in the factory object from the database object (database.py).
Every database object needs to communicate with its Hyrise instance. To get meta information and to change the state of the Hyrise (for example load tables). The communication logic is defined by jobs (hyrisecockpit/database_manager/job/). We have three kinds of job handlers that are executing these jobs:
- jobs that need to be continuously in the background (for example: get the system information) are executed by the
continuously_job_handler - jobs that will be done asynchronously (for example: loading the tables) are handled by the
asynchronously_job_handler - jobs that are done synchronously (for example: sending a SQL query over the SQL endpoint) are handled by the
synchronous_job_handler
The Flask app is communicating with the database manager and the workload generator via a simple REQ/REP pattern (https://zguide.zeromq.org/docs/chapter1/#Ask-and-Ye-Shall-Receive). Since the Flask app runs multi-threaded, every thread needs to create its own socket connection. For that, it can use the connection manager (hyrisecockpit/api/connection_manager.py). The flask app only fulfills the role of the client in the communication. The workload generator and the database manager are fulfilling the role of the server. For that, they are instantiating a server object (hyrisecockpit/server.py). It is just a wrapper around a zeromq server. The workload generator and database manager are initializing the server with a dictionary that maps to the handle function defined in the workload generator (hyrisecockpit/workload_generator/generator.py) and on for the database manager in (hyrisecockpit/database_manager/manager.py). The request for the server needs to map a key in the dictionary. The value of this key is a reference to a function that will handle this request.
The workload generator and the workers are communicating with the PUB/SUB pattern (https://zguide.zeromq.org/docs/chapter2/#Multipart-Messages). For that, the workload generator initializes a publisher zeromq socket (hyrisecockpit/workload_generator/generator.py). The worker then simply subscribes to this socket.
To support the Cockpit on MacOSX, the use of an adjusted multiprocessing queue (https://docs.python.org/3.8/library/multiprocessing.html) is needed. On Mac, the queue does not support the qzise property. For that, a wrapper is implemented in hyrisecockpit/cross_platform_support/multiprocessing_support.py. All modules that need a multiprocess queue need to import it from this module.
Besides the multiprocess queue, on Mac the default implementation of a MagicMock is not threadsafe. To fix this issue a wrapper around the MagicMock solves this problem. In all tests that are testing the multithreaded behavior of the cockpit, an import of hyrisecockpit/cross_platform_support/testing_support.py is needed.
You can find all the needed information in the Implementing a Driver section https://github.com/hyrise/Cockpit/wiki/Implementing-a-Driver .
TODO
(time range (flask)) (continuous queries)