criu with k8s, how to?

[ashish-billore]

First, this is not an issue, just a query, so please feel free to action it accordingly.

Context:
So far I am able to make criu checkpoint and restore work well for my application (user_sctp based sctp application) with runc directly (no management/orchestration layer above), all thanks to support from criu community and @adrianreber 👍

Now my goal is to somehow make this checkpoint and restore work with k8s pods.

---

Findings so far:

I see following pre-reqs for this:

• k8s container runtime support:
  docker (though latest is broken) has experimental support for criu; cri-o seem to have patches under progress to make it work; containerd has criu patches too.

• k8s support:
  There are KEPs and patches, some active ones are:
  KEP: Add Forensic Container Checkpointing KEP kubernetes/enhancements#1990
  Pod lifecycle checkpointing kubernetes/kubernetes#3949

---

While the community process is underway to have checkpoint restore support available in k8s (which will take sometime), I want to try it out in the meantime.

• Has anyone tried making checkpoint/restore work with recent versions of k8s (v1.18 onwards)?
• Do anyone has any recommended way to try it?

I could not find any recent working reference of criu + k8s.
If anyone knows, please share.

[adrianreber]

I am not aware of any other public discussions around Kubernetes and container migration.

[ashish-billore]

@adrianreber thanks for the response.

Sure, as I understand the above community activities will take a while to have checkpoint and restore available for pods with k8s, so I was looking for a working PoC possibly with workarounds, just to try it out with k8s pods.

Thanks!

[vutuong]

@ashish-billore You should check @schrej works here: kubernetes/kubernetes#3949. He has done a very wonderful job. He also published the document for testing pod migration in K8s ( in the thread) https://docs.google.com/document/u/1/d/1E5p_FOHDGAp5YEQ23dCi9I8wPnMzd4aOazxI4uO_AMo/edit
I also tested his work and it can work well. I hope that this very useful function will be worked or merged with K8S as soon as possible.

[ashish-billore]

@vutuong Thanks for the useful info.
I was aware of @schrej work and the related discussion in the KEP (in my original query above), however, didn't try the patches yet.
I'll try these patches and wish this feature lands in k8s soon.
Thanks!

[vutuong]

I'll try these patches and wish this feature lands in k8s soon.
Thanks!
@ashish-billore I'm trying to gather information about why it needs to support podmigration in K8s as my research requirements. So could you please tell me your use-case or the reasons for that? Especially, the comparisons between podmigration and StatefulSets. Because we can also use StatefulSets to support for stateful applications without podmigration?

[adrianreber]

@vutuong It seems you have another definition of a stateful application than I do. How do you define a stateful application?

[vutuong]

@vutuong How do you define a stateful application?
To answer your question, I think it will be more clear if we consider what the state is? To my best knowledge, the state that relates to the StatefulSet, for example in :https://www.youtube.com/watch?v=zj6r_EEhv6s&ab_channel=ThatDevOpsGuy, is something like a login session. For that, the states can be decoupled from the webserver process and store in other data storage (eg. Memcache, Redis, database). Then, the StatefulSet is introduced to deal with it.
So sometimes it makes me confuse. What is the difference between using Pod/container migration and StatefulSet since they seem both to try to prevent loss of application state?
Until now, I just found that, in comparison with StatefulSet, Pod/Container migration can speed up the initiation time for services that have a long startup time.
Any other reasons? I would be glad to hear your opinion? Thanks.

[adrianreber]

When talking about stateful, from CRIU's point of view, it is about the actual state of all involved processes. Meaning everything the container processes have loaded in memory will still be loaded into memory after migration. Any kind of memory backed database for example. Or processes which need a long time to start and to initialize.

Not sure what a StatefulSet exactly is in Kubernetes, but it sounds like it is talking about state concerned to storage. If the container, even in a StatefulSet, is started again it still needs to load all the data from storage. CRIU enables you have a stateful migration including everything which the process has loaded in memory.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[teo-tsou]

Hello after all this long time, is there any feature supporting SCTP socket maintenance after migration in CRIU? If not is there any way @ashish-billore to explain how did he manage to overcome it? Thank you :)