Introduce staging binary cache to reduce our binary cache size

[Mic92]

The idea is to have buckets to test VCL configuration on them.
In particular I want to create a new S3 bucket that we would start pushing to with hydra.
The VCL rules in fastly would check both buckets for the migration period.
After we have migrated the part of the binary cache that we are interested,
we make the VCL rules only hit the new bucket.
After that we add an archiving rule to transition the old bucket to aws glacier: https://aws.amazon.com/blogs/aws/archive-s3-to-glacier/

[edolstra]

What's this migration all about?

[Mic92]

@edolstra to reduce AWS costs on our binary cache.

[edolstra]

@Mic92 Yeah but why do we need a separate bucket for that? The way I imagine doing GC is that the GC process would set a tag on objects that we consider garbage, and then we can have a lifecycle rule that moves them to Glacier. As an intermediate step, we could have an access control rule that only makes them inaccessible (so if we screw up, we can restore them more cheaply than from Glacier).

[Mic92]

@edolstra sounds easier actually. I thought it might be also expensive to run api requests on all objects. But it looks like it's still doable:

Let's say we need 1-2 requests per object it would be ({1,2} * 889587727) / (1e6) = 889.5 -> 1779.1 Dollar
What would be the best way to find all the objects to keep?