Cluster Has Inconsistent View of Existing Docker Containers

[claykirk]

It appears that the "clusterSpecificAgentInstances" in the "DockerPlugin" class becomes stale and inconsistent over time. When a SERVER_PING request is sent from the server it provides a list of agents in the request which can be referenced using the "pluginRequest.listAgents()" method. The plugin iterates through this list to cleanup the agents. During cleanup, a fetch is made against the "clusterSpecificAgentInstances" using the "instancesCreatedAfterTimeout" method of the "DockerContainers" class when the "ServerPingRequestExecutor" calls the "performCleanupForACluster" method here. The agents in question never get cleaned up or disabled because the "DockerContainers" instance used is stale and hasn't been refreshed since the plugin was instantiated. The code to add these agents to the list of agents to disable exists here.

To address the inconsistent view of the "clusterSpecificAgentInstances" I've introduced a periodic force refresh of the "clusterSpecificAgentInstances" by scheduling a task each hour to reset the "refreshed" variable to false. This seems to help the problem but I'm not sure if there is a better solution for the root cause.

[ghost]

All committers have signed the CLA.