-
Notifications
You must be signed in to change notification settings - Fork 1
Home
Oliver edited this page Apr 1, 2021
·
15 revisions
Kiosc is a web-based platform to run Docker containers that expose graphical web user interfaces.
- later one we might want want to look whether the command starts with a
#!and then- write the command contents into a temporary file
- bind-mount it read-only into the container, chmod +x
- run it
- Portainer
- different scope, more general, we don't need any/all of their repositories, for example
- https://github.com/portainer/portainer
- might be worth looking into for learning about their management of docker container state
- Admiral
- again much broader scope
- https://vmware.github.io/
- might be worth looking into for learning about their management of docker container state
Kiosc allows to:
- run Docker containers from given (a) image names, (b) environment variables, (c) commands/scripts in the container a. in isolated network environments b. applying restrictions on memory, CPU usage etc. using Docker features
- offer project-based access to containers by being SODAR-core based
- work as a SODAR target site
- limit access to containers to authenticated users by providing a reverse proxy that checks permissions
- use of web sockets at predefined URLs, e.g.,
./websocket
- investigate Docker log output in the web interface
- manage Docker container state using an easy-to-use user web interface
- state is updated between user-defined "should" and Docker "is" state, e.g., start and stop containers on user interaction
- state is also updated between Docker "is" state and user-defined "current" state, e.g., reflect a crashed container in the user UI
- expose the management functionality also via a REST API
- a command line program
kiosc-cliis provided similar tovarfish-clithat can be integrated intocubi-tk, e.g., for creating appropriate containers for a new single-cell data set (cubi-tkbrings the knowledge about the Bioinformatics application)
- a command line program
- manage container "templates" at different stages:
- site-wide definition of templates
- project-wide definition of templates that are optionally linked to a site-wide template
- containers can be created from either
- provide a site-wide view for admins
- of all containers defined in Kiosc with their state in Docker (can be "not found/deleted")
- of all containers running in Docker that are not represented in Kiosc
- of all other Docker entities and their links to containers in Kiosc (e.g., networks and volumes)
- provide site-wide admin commands to
- cleanup unused entities in Docker
- remove containers residing as "stopped" in Kiosc for time span X
- stop and clean everything in Docker known to Kiosc and reflect that status in Kiosc as well
- regular maintenance tasks to perform cleanup and stopping as described in the first two points
-
ContainerTemplate- allows to define templates for containers, which can either be project-wide or site-wide (if
project==None) - allows to define a template for a container (e.g., with preset for maximal memory usage or shell and environment variables)
- environment variables can be set as required
- an optional link/foreign key to a site template exists when created from a site template
- default values from container site templates are copied but a user interface is provided to compare site and project template, show differences and select values to copy over from site to project template
- environment variables must be markable as "secret" and can only be set but not read (e.g., for
- allows to define templates for containers, which can either be project-wide or site-wide (if
-
Container- a container in stopped, running, etc. state; needs to be able to reflect a container deleted in Docker
- log information stored related to the container
- environment variables must be markable as "secret" and can only be set but not read (e.g., for tokens/keys)
- define a "container port" on which a HTTP server must listen and a HTTP path
- when HTTP connections to
http://container:$container_port/$path/return a 2xx or 3xx code then the container app is marked as "available" - optionally, a heartbeat URL can be given that is checked instead
- a "timeout" interval defines how long a container can be started but not ready yet
- container is marked as "timeout exceeded" and "should" state is stopped and stop is applied
All changes, whether automatic or manual should be reflected in the SODAR core timeline.
- superusers can create site-wide container templates
- site-wide templates can be copied to others but this loses the link
- project contributors and above can "copy" site-wide templates to project
- this links them to the project with all values but setting a foreign key to the site-wide template
- project-wide templates can also be copied to other project-wide ones, optional foreign key to side-wide template remains at side-wide template
- configuration can be copied from site-wide template to project-wide template or template to container by project contributor (or above) of the receiving entity
- UI works as follows
- display "there are changes between this and upstream"
- display button "compare & apply"
- button brings user to a view that displays the changed values side-by-side
- checkboxes allow selecting which to copy
- apply button changes the receiving entity
- if containers are involved then they are restarted if necessary (or flagged as "needs restart because of config change")
- at a later point, a UI way should be provided to copy from one template to another that the user has read/write access to as necessary
- secrets can never be copied
- same as Superuser creates site-wide template but for project contributors or above
- project contributors (or above) creates a container
- environment variables and settings can be defined manually or
- optionally, user can select a template to start from
- project contributors (or above) can later update container parameters
- copying values from the used template (if set) is implement similar to copying values from template to template
- secrets can never be copied
- containers are marked for restart if currently running, if user does not confirm then marked as "restart pending because of config change'
- project contributors can flag container's "should" state (e.g., "running" or "stopped")
- asynchronous container change state is triggered immediately
- project contributors (or above) can see container state and current log
- LATER: when running, most current log should be updated automatically from server via JSON
- project guest (or above) can "open" a container, e.g. click on "view" button
- if container is not running then set container "should" state as "running"
- redirect to "container reverse proxy lobby"
- an "activity loading" indicator is displayed
- page is automatically reloaded (or updated with server using AJAX)
- if startup fails, this is displayed
- if startup succeeds, user is redirected to reverse proxy for the given container
- every time a user accesses the reverse proxy, this is logged in the database
- if there is no log entry in the last time span X (container/template parameter, site-wide maximum through environment variable)
- the container's should state is set to "stop"
- asynchronous task for change in container state is started immediately
- Similar to Containers are shutdown automatically after time span X, the entity in Docker corresponding to the Container record in Kiosc is cleaned up.
The following should be a global environment switch in Kiosc. Any change requires a Kiosc restart. Any restart where a change of this variable is detected (implies that it must be stored somewhere in the database) triggers a restart of all containers.
- this is also useful when developing
- the prototype implementation worked as follows and we can reuse this
- Docker containers do not go into any special Docker network
- each container needs a definition of the "container" port where the server running inside of the container is listening on
- each container gets a unique, auto-generated high-port where the service is exposed on the host of the docker container
- reverse proxy knows this and forwards
http://localhost:$host_port/... - this implies that access to any high port has to be blocked on the kiosc server
- kiosc has to run inside a container
- it is possible to have the code and everything (e.g., miniconda environment) on the host computer
- use a docker container having the same OS as the host/development compuer
- bind-mount these locations read-write into the container, e.g., to
/opt/miniconda3and/usr/app/src - launch a
docker run -it ubuntu:20.04 bash -i - then navigate into
/usr/app/srcand callsource /opt/miniconda3/bin/activate && python manage.py runserver
- a unique Docker network is created for each container
- kiosc is attached to each such network and can now access the hosts using
http://$container_id:$container_port/... - on cleanup of containers the private bridge networks have to be removed as well
- this will require a change in network allocation as described here
- this completely isolates the individual docker containers which is a good thing
- define milestones
- features
- tasks
- break into tickets
- start out implementing first draft of basic SODAR core app with models
- work on tickets
Start with fresh SODAR core 0.9.1. Start and stop containers and provide access via reverse proxy. No state handling, logging, tests yet.
- Project contributor (or above) creates container in project
- Create SODAR core 0.9.1 code base
- Create models:
ContainerTemplate-
Container- when HTTP connections to http://container:$container_port/$path/ return a 2xx or 3xx code then the container app is marked as "available". optionally, a heartbeat URL can be given that is checked instead
- a "timeout" interval defines how long a container can be started but not ready yet. container is marked as "timeout exceeded" and "should" state is stopped and stop is applied
- Fields:
-
state: stopped, running, etc (which docker states exist?) + deleted! -
date_created: models.DateTimeField(auto_now_add=True, help_text="DateTime of creation") -
date_modified: models.DateTimeField(auto_now=True, help_text="DateTime of last modification") -
date_last_logs: models.DateTimeField(auto_now_add=True, help_text="DateTime of last log pull") -
sodar_uuid: = models.UUIDField(default=uuid_object.uuid4, unique=True, help_text="Process SODAR UUID") -
project: models.ForeignKey(Project, help_text="Project in which this objects belongs") -
container_id: models.CharField(max_length=128, help_text="Container ID", blank=True, null=True) -
internal_port: models.IntegerField(default=80, help_text="Server port within the container", blank=False, null=False) -
internal_path: models.CharField() -
host_port: models.IntegerField(help_text="The port of the container on the host", blank=False, null=False, unique=True) -
do_run: models.BooleanField(default=False) -
state: models.CharField(max_length=64) -
environment: JSONField(help_text="The environment variables to use.") must be markable as "secret" and can only be set but not read (e.g., for tokens/keys) -
environment_secret: bool -
command: models.CharField(max_length=64000, help_text="The command to execute", blank=True, null=True)
-
-
LogEntryContainer-
date_created: models.DateTimeField(auto_now_add=True, help_text="DateTime of creation") -
level: models.CharField(max_length=32) -
text: models.TextField() -
container: models.ForeignKey(Container, related_name="log_entries", blank=False, null=False)
-
BackgroundJobContainer
- Container creation and listing
- View and form to submit container
- View to delete container
- View to update container
- View to list containers
- Start/Stop container
- Add task to pull image
- Add task to run container
- Implement simple state machine that only knows start stop. No sync of docker state. No AJAX yet. Async is required though.
- View to start background job to start container (contributor+)
- View to stop container (contributor+)
- Reverse proxy
- Implement (copy) reverse proxy to access containers
Add tests and mock the docker calls.
- Test views
- Test tasks
- Test models
Add logging with timeline. Fetch logs from Docker container.
- Project contributor (or above) inspects container config logs
- Add logging with timeline to tasks
- Add logging with timeline to views
- Fetch logs from Docker container and store in
LogEntryContainer
Implement (or copy from PoC) state machine. Use BackgroundJobContainer to store state.
- Project contributor (or above) manages container state
- Project guest (or above) "opens" container
- Extend
Containermodel to contain states reported to user (these are the possible states of a docker container)- States from PoC:
initial,idle,starting,running,stopping,failed,pulling,deleting - Actual Docker container states:
created,restarting,running,paused,exited,dead- Extend by:
deleting+deleted - What about
pulling?
- Extend by:
- States from PoC:
- Extend state machine to react to the states
- Extend permissions: contributor+ can change state
- Extend permissions: guest can do something that I didn't understand yet. Q: what is a proxy lobby?
- project guest (or above) can "open" a container, e.g. click on "view" button
- if container is not running then set container "should" state as "running"
- redirect to "container reverse proxy lobby"
- an "activity loading" indicator is displayed
- page is automatically reloaded (or updated with server using AJAX)
- if startup fails, this is displayed
- if startup succeeds, user is redirected to reverse proxy for the given container
Introduce templates. Enable starting containers from templates.
- Superuser creates site-wide template
- Project contributor (or above) copies site-wide template to project
- Project contributor (or above) applies changes from template to template or container
- Project contributor (or above) creates project-wide template
- Project contributor (or above) manages container parameters
- Create view and form for site-wide template creation. (superuser)
- Create view to copy site-wide templates (superuser)
- No foreignkey to original template
- Create view and form for project-wide template creation. (contributor+)
- Create view to copy site-wide template to project or template from other projects. (contributor+)
- Set foreingkey to site-wide template
- Extend container creation view to select from defined templates
- Should the user still be able to pull an image from an individually defined string?
- Create view to copy configuration from site-wide template to project-wide template (contributor+)
- display "there are changes between this and upstream"
- display button "compare & apply"
- button brings user to a view that displays the changed values side-by-side
- checkboxes allow selecting which to copy
- apply button changes the receiving entity
- if containers are involved then they are restarted if necessary (or flagged as "needs restart because of config change")
- at a later point, a UI way should be provided to copy from one template to another that the user has read/write access to as necessary
- secrets can never be copied
- Create view to copy configuration from template to container (contributor+)
- Create view to update configuration of container (contributor+)
- secrets can never be copied
- containers are marked for restart if currently running, if user does not confirm then marked as "restart pending because of config change'
Develop views for the rest API. This parts needs to be defined in detail.
- View for creating containers ... from templates? TDB
- View for deleting containers
- View for start/stopping containers
Add views for the admin to manage the docker service and all containers/images.
- Containers are shutdown automatically after time span X
- Containers are cleaned up automatically if currently shutdown after time span X
- Admin views
- List of containers with their state in Docker (can be "not found/deleted")
- List of containers running in Docker that are not represented in Kiosc
- List of other Docker entities and their links to containers in Kiosc (e.g., networks and volumes)
- Admin management commands (names are preliminary, can be changed!)
-
cleanup_unused: cleanup unused entities in Docker (Q: what is unused?) -
remove_stopped: remove containers residing as "stopped" in Kiosc for time span X -
stop_clean: stop and clean everything in Docker known to Kiosc and reflect that status in Kiosc as well
-
- Admin periodic tasks:
- Run
cleanup_unusedandremove_stopped - Shut down unused containers:
- every time a user accesses the reverse proxy, this is logged in the database
- if there is no log entry in the last time span X (container/template parameter, site-wide maximum through environment variable)
- the container's should state is set to "stop"
- asynchronous task for change in container state is started immediately
- Run
AJAX and Vue?
- Mach et otze
Improve scripts that can be passed to the command field of a container
- Check if scripts are passed
- Check if text starts with a #!
- write the command contents into a temporary file
- bind-mount it read-only into the container, chmod +x
- run it