PipelineRun Status bloat and excessive updates

[skaegi]

In our production systems we are seeing large PR objects frequently over 1MB in size when serialized as JSON with nearly all the information in status pipelineSpec and taskruns. This might seem big now but I would bet this is really just the beginning and as we see more Tekton usage things are going to get a lot larger. In addition as a pipeline runs the PRs get updated along with changes to status.taskruns. This is triggering a huge amount of traffic for listening systems like the dashboard and our other systems that react to events in cluster. We're already hitting internal cluster limitations and have had to trim the PRs significantly to avoid overwhelming our messaging systems.

Looking at how the core Kubernetes API use status, it seems limited to very basic information and avoiding redundant information already captured in child or related objects. How painful would it be to re-work status to be lightweight?

[imjasonh]

To me this seems to be closely related to #454 -- if we had an off-cluster system to store *Run statuses, we wouldn't be bound by etcd's storage and performance issues around l arge objects. (That being said, we would be bound by whatever that system's limitations are 😄 )

As an alternative, we could phase out reporting all of each sub-*Runs' statuses in the PipelineRun, and just require clients to look up all the constituent *Runs' statuses in order to display all that information. Or we could only report sub-Runs' .status.conditions to signal success/failure/ongoing, and stop reporting per-step information, timing, etc. These would need to be carefully communicated and staged slowly to avoid breaking clients that may rely on this information today.

[bobcatfish]

i believe the chains folks will have some thoughts here as well @dlorenc

[bobcatfish]

Also more info in #279

The PRs that made these changes are #2444 and #2489

In our production systems we are seeing large PR objects frequently over 1MB in size when serialized as JSON with nearly all the information in status pipelineSpec and taskruns.

one thought: maybe dereferencing all the taskruns into the pipelinerun is too far?

[imjasonh]

one thought: maybe dereferencing all the taskruns into the pipelinerun is too far?

I think it makes sense to report each TaskRun's .status.conditions (success/failure) and results, and maybe a few more things, but per-step information might be overkill, and probably contributes the most to storage bloat. Updating the PipelineRun each time a step starts/finishes also means a lot more updates.

If Chains needs some complete record of each internal state I assume they could do the TaskRun lookups themselves, or consult a Results API which has done all the lookups, when that exists.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.