Parakeet 0.6B Int8 model consuming 1.2GB RAM

[sunilpro]

Tested loading parakeet-tdt-0.6b-v3-int8 on iOS 16, XCode shows 1.23GB RAM usage. Isn't it supposed to occupy around 600MB?

[dosubot]

Hi @sunilpro! I'm Dosu and I’m helping the sherpa-onnx team.

The model files for parakeet-tdt-0.6b-v3-int8 total about 640MB on disk, but there isn't official documentation specifying expected RAM usage on iOS or other platforms. It's common for RAM usage to exceed the on-disk size due to factors like ONNX Runtime overhead, memory mapping, and iOS-specific allocations. Seeing 1.23GB RAM usage is not necessarily abnormal, even for an Int8 model, given these factors. If you want to investigate further, profiling memory allocations or adjusting ONNX Runtime settings may help, but there are no documented benchmarks or known issues about this for Parakeet models on iOS yet reference.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[csukuangfj]

If you don't do any thing and just load the model into memory, it consumes already around 600MB of RAM.

Can you describe why it occupies only around 600 MB of RAM?

Maybe you assume that computation won't take up memory.

[sunilpro]

I'd assume computations wouldn't take 600MB on a 600MB model! I maybe wrong though

Here's the fun part:
config.model_config.provider = "coreml";

When I changed provider from "cpu" to "coreml" the memory usage shot up to 2.9GB before the app crashed due to OOM.

I'm testing this on iPhone 15, if that helps. @dosu