k8s logtail

[asaiacai]

Closes #3429

This swaps the container sleep with a logtailing process so that pod logs now show task logs. These are retrievable via kubectl logs <POD_NAME> and also written to /var/log/pods/*/*/*.log on the node disk, so it should integrate with most k8s logging solutions out of the box (OpenTelemetry, Fluentd, Filebeats, etc.)

Implemented as a watcher process that does tail -f if there's an open writing file handle on the task log by checking lsof and closes the tail -f program if there are no such writers.

Tested (run the relevant ones):

• Code formatting: bash format.sh
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_container_logs_multinode_kubernetes
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_container_logs_two_jobs_kubernetes
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_container_logs_two_simultaneous_jobs_kubernetes

[asaiacai]

@romilbhardwaj I finished running the smoke tests on my end if you have a spare cycle to take a look

[romilbhardwaj]

Thanks @asaiacai!

[asaiacai]

@romilbhardwaj reran the smoke tests and manually checked kubectl logs again but let me know and I can draft up some additional tests.

[romilbhardwaj]

Thanks @asaiacai. Pushed some changes to reduce sleep (which was causing delays in pod termination). lmk what you think.

BTW, tests seem to be failing for me:

$ pytest tests/test_smoke.py::test_container_logs_two_simultaneous_jobs_kubernetes --kubernetes

Logs: https://gist.github.com/romilbhardwaj/4065c409f089760fb11415f0bf186262

Looks like logs are correctly tailed on the first sky launch, but logs from any subsequent sky exec are not tailed.

This was on a GKE cluster. Interestingly, the manual repro seems to work on my local sky local up cluster, but the test still fails at kubectl get pods | grep t-container-logs-two-5n-e5 | grep head | awk '{print $1}' | xargs -I {} kubectl logs {} | wc -l | grep 18.

[asaiacai]

hey @romilbhardwaj thanks for the edits! seems like there was variable delay between the task stdout/stderr being written to /var/log/pods which was occasionally longer than 5 seconds. Increasing the sleep time seemed to reliably get rid of it and i was able to run it 19 times in a row without fail on GKE. Let me know if you also see this reflected on your end.

[romilbhardwaj]

Thanks @asaiacai! Tested it, works nicely. LGTM.