ops: stdcm requests can't be saved and reproduced anymore

[eckter]

The current way to save and rerun past requests is to save editoast input payloads in datadog.

That relies on the input being smaller than 1024 characters. Which was fine up to now, but adding the list of allowed tracks to this payload brings it to a few hundreds kbs.

---

Possible fix 1

Object storage: #13640

It would take some time to set it up, but it would be straightforward, efficient, and reliable.

Possible fix 2

1. Copy the input payload and remove the list of tracks from the copy, to keep its size under the threshold. Log that in datadog instead.
2. Figure out which search environment was used in the request, and log that. May require an API change.
3. Adapt the script that builds and save a payload
   a. Identify the used search environment
   b. Query that search environment to get the tracks. May require a new endpoint, or iterating through all the envs and their tracks.
   c. Patch those tracks into the request to be sent to editoast.

While this delays the need for an object storage for a little longer, I don't think it would take less time / effort / complexity than option 1.

[eckter]

Notes for today's meeting:

• S3 isn't necessarily exposed: we may need to go through the gateway and/or an editoast endpoint
• We may go through admin interfaces, with an ADM account, but that requires some initial setup
• We want to delete timetables from the DB after a pretty short time (~10 days). If we setup an object storage, we could also store past timetable there for a longer time (maybe 2 months).
• We don't know precisely how much we'd need to pay per MB with an s3.
• We can directly access the s3 storage.

-> Option 1

TODO:

• Setup an S3:
  • Update terraform, config access rights
  • Update the "DAT"
  • Configure something equivalent for local setups? Either that or skip it entirely.
• Core:
  • config s3 access
  • Save the payloads
  • Save the timetables, but only once. Or at least once per stdcm worker start
  • Check on valkey whether the timetable has been saved to the s3. Depends on: ops: add a more persistent cache for core timetable #13908
• Compare file sizes, including with compression
• Update reproducibility script