Prow: stop using volumes and use local disk instead

[tuminoid]

With the new private infrastructure servers deployed, Prow should stop using slow volumes and use local disk instead.

[tuminoid]

@lentzi90 as discussed offline.

[lentzi90]

Let's just confirm that we have capacity first. It would not be nice to reserve local disk for prow if that makes the jenkins tests starved for resources.

[tuminoid]

For the reference, can you summarize how the disk space was spent right now? @lentzi90

[lentzi90]

Sure! Prow is currently using 100 GiB volumes. The cluster consists of 1 control plane node and minimum 3 worker nodes at the moment. The workers can scale up to a total of 6. The bastion host does not use any volume.

In total, that can be maximum 700 GiB. Most of the time we are at 400 GiB.

(Outside of this is 1 volume for ghproxy, also 100 GiB. This is attached to a pod so using local disk for it may not be a good idea. It would lose all state every image bump or eviction.)

[tuminoid]

Thank you. I think we are safe to do the change next week. Leave ghproxy with a volume.

[tuminoid]

/triage accepted