Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement production ready setup with reader/writer worker split #30

Open
pharr117 opened this issue Oct 29, 2023 · 1 comment
Open
Labels
enhancement New feature or request

Comments

@pharr117
Copy link
Collaborator

The application is currently reliant on the Flask development mode setup since data is gathered and stored in single-process available Python global maps.

Flask is meant to be run behind some server type (sometimesa WSGI, gunicorn being one example). These servers spin up multiple instances of the Python program to help with load balancing and app runs.

The application data system is not prepared for this. Each gunicorn process would have its own global data store, meaning the app would be bashing RPC servers on a per-worker basis.

The application needs to be moved to a more production-ready state. One way would be to implement application modes for reader/writer process splitting such as:

  1. Reader mode -> a Flask application that reads from an external data store and responds to requests
  2. Write mode -> a Python application that writes to an external data store
  3. Data store layer -> A data storage location that both reader and writer interface with

A production deployment would then look like:

  1. A gunicorn server that spins up multiple Reader mode versions of the application
  2. A writer process that writes to the data store
  3. A shared data storage layer
@pharr117 pharr117 added the enhancement New feature or request label Oct 29, 2023
@pharr117 pharr117 linked a pull request Oct 29, 2023 that will close this issue
@pharr117 pharr117 removed a link to a pull request Oct 29, 2023
@pharr117
Copy link
Collaborator Author

Work had started here: https://github.com/DefiantLabs/cosmos-upgrades/tree/feat/production-ready-build

This was a very quick v1 of a possible reader/writer split that was working with a shared a FileSystem cache between the two workers with an example Docker Compose build linking the 2.

A more complex setup with an underlying database layer replacing the FileSystem cache may be a better move.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: No status
Development

No branches or pull requests

1 participant