Long term logs on PipelineRuns and Taskruns

[vdemeester]

Initiating this to discuss long-term storage for logs for PipelineRun and Taskruns. As of today, logs on PipelineRun and TaskRun are only available for a limited amount of time and, most importantly are tied to the underlying pods lifecycle. Once the pods are gone, the logs are gone too. With the rise of garbage collection, described in TEPs like this one, we need to figure out if (and how) we want to propose solutions to store logs in the log term (long after the pods is gone, most likely also long after the PipelineRun/TaskRun object are gone).

• Should we define an API for this ? (that would fetch logs from external resources like elastic, …)
• Should we have a set of converntion (annotation, …) that tooling can use to know where/how to get the logs ?

Tools to consider during the discussions

• tkn
• dashboard
• results
• … probably more.

There is already tooling in the kubernetes ecosystem to gather logs from pods, we should aim to not re-invent the wheel here, but only find ways to easily hook-up into those options.

[bobcatfish]

this is probably relevant to @wlynch 's interests - also note that we (primarily @sbwsg ) did do a lot of investigation into this and decided to leave long term log storage to other k8s tools

[vdemeester]

Yeah this is also our approach, from the tektoncd/pipeline perspective. But I feel there is room for enhancement, documentation, integration, with the tools we ship as part of tekton (tkn and dashboard are relevant here).

[wlynch]

+1 to @vdemeester's idea for an annotation / other data that would tells us how to look up the data later.

This is kinda how users can fetch long term logs for GKE with Stackdriver today - we can construct a query with the information we need from a given TaskRun (cloud project, cluster name, namespace, pod name, timestamps).
The question is how can we generalize this to support other log locations. 🤔