Fastly reads from >= two locations

[RaitoBezarius]

Is your feature request related to a problem? Please describe.

In preparations to experiment other stores of data than our main S3 location, we need to make Fastly read from >= 2 locations.

Describe the solution you'd like

(1) Design a solution using a Fastly personal account, mimicking Fastly configuration of cache.nixos.org
(2) Show smooth fallback between the two locations in case of 404
(3) Suggest a Terraform configuration for this infrastructure

Describe alternatives you've considered

Additional context

[delroth]

(2) Show smooth fallback between the two locations in case of 404

A few more scenarios that I think should be tested:

• What happens if the primary origin is down / very slow?
• Depending on how the origin selection is done:
  • If it relies on always trying a primary read first: what's the latency cost for stuff served from the secondary?
  • If it relies on having some kind of dispatch/redirect service with state of what's in the primary origin: what happens if that redirect/dispatch service is down? What happens if the state is wrong?

Depending on the exact specifics of (1) I can imagine there's plenty of new "fun" reliability issues that will crop up in there that will need some consideration.

[RaitoBezarius]

cc @edef1c @flokli we discussed that a long time ago IIRC.

[vcunat]

I don't think we can afford for the primary origin to be very slow. Or at the very least not for common-ish *.narinfo paths.

[flokli]

Let's discuss this first for the NAR paths, not NARInfo, which is far less latency-sensitive, and contains more data.