Here is a quick reference to sections that may be of interest to you:
- How to manage an outage - Things are broken, and I need to fix them now
- Deployment Considerations - What do I need to configure to deploy this?
- Resource Usage - How much resources do I expect this to take?
- Troubleshooting - How can I tell what is and isn't working easily?
This purpose of this first section is the "its 3am, what should I do?"
This system is the following containers:
tfserving
config_manager
remote_model_puller
master
All of which have healthchecks and resolve their own state.
First determine if its an outage.
- Are the containers up and healthy? Not an outage
- Is there just a couple restarts? Not an outage
- Are we seeing containers down or cascading restarts? Maybe
The only container that can cause issues short term is the tfserving
container.
If any of the following containers are causing issues:
config_manager
remote_model_puller
master
it is not considered a true outage. Those containers being restarted/failing/stopped is fine. Stop them if they are causing you issues and recommendations will investigate in the morning
If the tfserving
container is restarting on a single node. Then consider removing that node from
swarm. We can have 1 of many containers down with minimal impact.
If the tfserving
container is cascading/failing on multiple/all nodes. There is most likely an issue
with the container that will require manual intervention. Please escalate the problem.
If tfserving
is causing system wide issues. You can aways stop the service in docker swarm. This will
cause a lot of logging from the search side and cause recommendations alerts to go off.
Stopping the service in docker-swarm will cause increased load on elasticsearch for search, and will send an error alert to Recommendations every 15 minutes. This should be considered a last resort policy.
Contacts in order:
- Carl McGraw
- Zhihong Duan
- Max Anger
- Kip Obenauf
- Inga Evenchik
During business hours in order:
- use the MS Teams Channel, "@" the appropriate people
- use the email [email protected]
After hours in order:
- Send a message to the MS Teams Channel If no response in 5 minutes
- Use phone contact information on confluence page
When an outage is occurring, meaning tfserving
containers are not available. The following events are expected:
- Calls from search -> Elasticsearch will attempt to call
tfserving
via gRPC (15-45 per request) - These calls will hit a 600ms initial connection timeout (thus blocking in ES for 600ms, and tying up resources)
- Once more than 25% of inbound requests fail, ES will flip a killswitch stopping all requests from trying for 60 seconds
- After 60seconds ES will attempt again. The killswitch will flip immediately if requests are still failing
- Recommendations alerts will trigger on each unique killswitch in a 15 minute interval
- Search will fallback when timeouts or killswitches occur. Serving less than optimal results to users.
During the killswitch the load on ES is relatively low (but higher than normal). When the killswitch isn't active, but
there is an outage in tfserving
containers, ES will tie threads up and cause search to lock apache threads too.
This resource stealing of ES and search can cause wide sweeping effects. The killswitch is designed to let the system take a break, but it will still cause substantial resource holding inbetween killswitch activations.
The purpose of this section is to give some deployment considerations regarding the system. I assume in this section that you have some familiarity with what services are running this project. They are:
tfserving
config_manager
remote_model_puller
master
For more indepth details please visit the swarm-deploy.yml
at the root of this project.
I am enumerating a list for each service on its shared dependencies. You can consider this a deployment checklist of things that need to be configured before the service is deployable and operational.
called by Elasticsearch via gRPC connections. It represents the machine learning model
- container serves to Elasticsearch servers on a gRPC connection. Ensure that the ES servers in the environment can access the container through DNS entry
logginrec.mlmodels.icfsys.com
- uses gRPC connection on port
8500
for talking to Elasticsearch. Ensure that8500
is mounted to the host - uses
/local_saved_models
internally to store model state. Ensure that the host volume is mounted in the container at the correct location. Should be referenced by envvar$LOCAL_MODEL_DIRECTORY
- uses
/serving_config/models.config
internally to store config state. Ensure that the host volume is mounted in the container in the correct location. Should be referenced by envvar$TENSORFLOW_SERVING_CONFIG_FILE
- Should run in global mode on each node. We shouldn't have a
tfserving
instance running without aremote_model_puller
orconfig_manager
also running on the same node! - Should have MOST of the node resources available to it.
remote_model_puller
andconfig_manager
should be taking up just a small share of resources. The large share of resources should go to this container.
tracks what tfserving
knows about, and keeps it up to date with available models, also removes models that are no longer valid to clean up disk space.
- should be able to write to
/app/
dir on container filesystem to save down healthcheck information between threads - should have the
$HOSTNAME
envvar defined and passed to it. This envvar will let it register with themaster
node. Registration is the main way formaster
to track this container. It will reference it directly by its hostname to avoid load balancing - should have port
8002:8002
mapped to the host container. This is for communication withmaster
. Themaster
container will call it to initiate admin calls. - should have
$LOCAL_MODEL_DIRECTORY
as a volume mount on the host machine. It will share this withtfserving
andremote_model_puller
. This will allowconfig_manager
to track what models are available - should have
$TENSORFLOW_SERVING_CONFIG_FILE
as a volume mount on the host machine. It will share this withtfserving
. This will allowconfig_manager
to updatetfserving
models that are available - should have
$TENSORFLOW_SERVING_GRPC_TARGET
available internal to the container. This will allow theconfig_manager
to communicate withtfserving
and determine which models it currently knows about
tracks what models are available on a defined gcs bucket. Then pulls the most recent/current models down to local file system. This local file system is shared with config_manager
- should be able to write to
/app/
dir on container filesystem to save down healthcheck information between threads - should have the
$HOSTNAME
envvar defined and passed to it. This envvar will let it register with themaster
node. Registration is the main way formaster
to track this container. It will reference it directly by its hostname to avoid load balancing - should have port
8001:8001
mapped to the host container. This is for communication withmaster
. Themaster
container will call it to initiate admin calls - should have access to gcp credentials at
$CLOUDSDK_CONFIG/application_default_credentials.json
- should have
$REMOTE_MODEL_DIRECTORY
as a volume mount on the host machine. It will use a remote GCS bucket at$REMOTE_MODEL_DIRECTORY
to know current remote state. - should have
$LOCAL_MODEL_DIRECTORY
as a volume mount on the host machine. It will share this withtfserving
andremote_model_puller
. This will allowremote_model_puller
to pull availalbe/valid models locally and share them with other containers
Allows administrative operations to occur on the cluster. The master
container keeps a list of registered config_manager
and remote_model_puller
nodes. It allows users to perform operations across the entire cluster, and to read/change remote model state as well.
- should be able to write to
/app/
dir on container file system to save down registration information between thread - should be available on a DNS
mlmodelmanager.icfsys.com
, serving port80
- should be running on only one node
- should have access to gcp credentials at
$CLOUDSDK_CONFIG/application_default_credentials.json
We anticipate the following resources used per container:
service | mem | cpu | net io | disk io |
---|---|---|---|---|
tfserving |
~12gb | high | medium | low |
config_manager |
500mb | low | low | low |
remote_model_puller |
500mb | low | medium | medium |
master |
500mb | low | low | low |
We anticipate the following request loads
service | type | reqps | connections | external to docker? |
---|---|---|---|---|
tfserving |
grpc | 5k> | ~20 | yes via DNS entry / docker load balancer |
config_manager |
http | <1 | 1 | no |
remote_model_puller |
http | <1 | 1 | no |
master |
http | <1 | 1 | yes - internal to ATG via DNS entry / docker load balancer |
How to check if everything is or isn't working.
All containers carry healthchecks on them that check their state for internal consistency. If the container fails it healthcheck then it will eventually restart and thus shouldn't cause an issue.