add max_workers experiment

[spencerschrock]

Summary

This benchmarking experiment evaluates the effect of the max_workers count on sharded hashing. The worker counts were picked to fall around vCPU counts for common VMs.

Local experiments with a large model (to ignore the effects of OS disk caching) showed a diminishing effect beyond 8 workers, indicating the disk became the bottleneck. By default, we defer to the value selected by the concurrent.futures library (cpu_count + 4, up to a maximum of 32).

We likely don't need to increase our value beyond the max of 32, but we may want to advocate for users to set the value lower if disk usage is a concern.

python3 benchmarks/exp_worker.py ~/models/Llama-3.1-70B
1:    1064.7986
2:     516.6226
4:     276.0468
8:     223.2531
12:    232.7897
16:    222.8352
24:    222.3078
32:    222.9888
48:    222.4046
64:    222.3379
80:    222.1991
96:    222.3327
128:   222.3440

Checklist

• All commits are signed-off, using DCO
• All new code has docstrings and type annotations
• All new code is covered by tests. Aim for at least 90% coverage. CI is configured to highlight lines not covered by tests.
• Public facing changes are paired with documentation changes
• Release note has been added to CHANGELOG.md if needed