feat: improve engine caching and fix bugs

[zewenli98]

Description

As I requested, TensorRT 10.14 added an argument trt.SerializationFlag.INCLUDE_REFIT to allow refitted engines to keep refittable. That means engines can be refitted multiple times. Based on the capability, this PR enhances the existing engine caching and refitting features as follows:

1. To save hard disk space, engine caching will only save weight-stripped engines on disk regardless of compilation_settings.strip_engine_weights. Then, when users pull out the cached engine, it will be automatically refitted and kept refittable.
2. Compiled TRT modules can be refitted multiple times with refit_module_weights(). e.g.:

for _ in range(3):
    trt_gm = refit_module_weights(trt_gm, exp_program)

3. Due to some changes, the insertion and pulling of cached engines are located in different places, which causes 🐛 [Bug] Engine cache failed on torch.compile backend=tensorrt #3909. This PR unified the insertion and pulling in _conversion.py.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Checklist:

• My code follows the style guidelines of this project (You can use the linters)
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas and hacks
• I have made corresponding changes to the documentation
• I have added tests to verify my fix or my feature
• New and existing unit tests pass locally with my changes
• I have added the relevant labels to my PR in so that relevant reviewers are notified

[narendasan]

@cehongwang please take a pass so we have multiple eyes on this PR

[narendasan]

When would a torch.compile use try to use strip weights?

[zewenli98]

added the warning back. Not sure why strip_engine_weights arg doesn't work for torch.compile()

[narendasan]

It just doenst make sense. Torch compile is not serializable. So why would you ever want a callable that doesnt have the weights in it

[zewenli98]

My understanding is that for torch.compile, _pretraced_backend() calls compile_module(), which is the same as the AOT to build TRT engines. If we pass in settings.strip_engine_weights, doesn't it tell the builder to build a weight-stripped engine?