What happens to the original processes? how to capture a whole service?

[MartinEmrich]

Quite new to mirrord, and I just got it working. But I still have to wrap my head around this. And strangly I found no previous discussions about this (IMHO very obvious) topic.

The default behaviour (mirroring traffic) seems confiusing. If an incoming request arrives at the (remote) platform, it then would be processed twice: once on the original service, and once on the local, mirrored process. Imagine a "purchase request" in the typical "hello-world pet shop application" being duplicated...

With the "steal" mode, that's "fixed", but what's with the original processes on the remote cluster? It is still running. So it still has connections to databases, message queues, etc., an thus will still interact with the platform.

If it's not just one pod (like in my first use case on a single-pod StatefulSet), it gets more confusing. As I see it, mirrord runs on a pod level, so if I have a deployment with multiple replicas, I can only put mirrord over one of the pods, ending up with a mixture of the original code running on the cluster, and my local process randomly getting a subset of requests from kube-proxy.

So can I completely, temporarily replace a service (Either in the Kubernetes Service sense, but also in the
Deployment/StatefulSet/Replicaset sense) with my local process via mirrord, without altering Ressources on the remote cluster?

[eyalb181]

Hey @MartinEmrich, there's the pause feature, which pauses the target pod while mirrord is running. Does that address the problem you're describing?

Also, note that multi-replica deployments are supported as part of mirrord for Teams.

[MartinEmrich]

@eyalb181 thanks!

That pause function seems to work. I assume it uses SIGSTOP and SIGCONT? This could be a bit "brutal", i.e. freezing in-flight DB transactions and all sorts of connections. But this will mostly work for me.

mirrord for Teams seems a bit overkill for me, but I'll keep it on my radar!

[eyalb181]

We don't explicitly send signals to pause the target, but rather use the API of the container runtime in use (containerd, Docker and cri-o are supported). I don't know if there is a runtime that uses SIGSTOP in order to pause containers, but for example Docker internally uses the cgroup freezer in order to suspend all processes in a container.

[MartinEmrich]

I guess the effect is similar, the kernel will no longer schedule that process (group) on the CPU. from the orginal process tree view, it's like time gets frozen, and after the mirrord redirect is lifted, it's a time-jump forward, instantly timing out any in-flight connections/transactions.

At least the logs of my application's original pod look that way.