feat(libstore): curl-based s3

[lovesegfault]

Motivation

#13084

Context

---

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[xokdvium]

Would it be conceivable to just outright replace aws-sdk-cpp with this? I don't think we'd want to support it if we land this eventually.

[lovesegfault]

I was unsure about just removing the existing implementation outright, but I can do that if folks are OK with it?

[xokdvium]

but I can do that if folks are OK with it?

Can't speak for everyone, but I feel ripping out aws-sdk-cpp would be very welcome if the new thing is more lightweight and reuses the curl infrastructure with feature parity. That would definitely be better than supporting both implementations.

[lovesegfault]

@xokdvium I'm going to get this in a fully working state, and after I'm happy with it I'll tack on a final commit removing the old impl, so if there's tension we can discuss that point separately, or I can make it a separate PR, etc.

I'm with you that if this works well, we're better off just tossing the old impl, though that would mean folks building with old curl would lose the ability to talk to s3, but idk if we care to support people building non-standard configurations like that.

[Mic92]

I want to definitely test this with our hydra staging instance to ensure it still can upload stuff to s3 buckets before merging.

[lovesegfault]

AFAICT this is ready for (very) preliminary testing; at least I was able to pull and push to an S3 store and auth just worked, like magic :)

[lovesegfault]

Nevermind, I messed something up, buckets outside of us-east-1 don't work :P

[xokdvium]

Some preliminary comments

[Ericson2314]

Now that I see the callers, we can actually go back to void, but different than before.

They all throw exceptions with a slightly different message.

Search around for the "catch, add trace, rethrow" pattern. This method can throw a nix error, and then the callers can do that to add contextual information.