Unexpected blocking IO when configuring TLS with tls-native-roots enabled

[fmonjalet]

Bug Report

Version

0.12.3 (feature tls-native-roots enabled)

Platform

Linux

Description

Hi! I was investigating tokio scheduling lags in a service I work on, and ended up noticing that TLS configuration can take several milliseconds (~20ms). This can pile up and lead to tokio stalls of hundreds of milliseconds, blocking all other processing in the meantime. Our process is part of a data intensive distributed system and communicates with hundreds or thousands of nodes at any point, and connections end up being regularly refreshed (for example because of nodes coming and going away).
Another aggravating factor is that we use a custom load balancer that will configure TLS for new discovered endpoints (and maybe that's an anti-pattern, happy to be guided there).

When looking a bit deeper into it, I noticed that when the tls-native-roots is enabled, configuring TLS will lead to calling this function, that (if I understood correctly) will read the certificate chain from disk, at least on Linux.

The code we use is basically something like:

endpoint.tls_config(ClientTlsConfig::new().with_enabled_roots())

Is this a known issue? If yes, what is usually the best practice?
I imagine it would be fine for us to have a thread refresh the native certificate chain in the background and just use this to initialize TLS, but I am not sure how to proceed (and want to avoid over-engineering if there is a better alternative).

Thanks!