Support --tokenizer-path to load custom tokenizer (local or remote) #92
Description
Currently, the tokenizer path is hardcoded and cannot be overridden, which causes issues when evaluating local or private models.
I suggest adding a --tokenizer-path argument to support loading tokenizers from either a local directory or a remote Hugging Face model.
This would require changes in /token_benchmark_ray.py and /src/llmperf/utils.py.
I’m happy to submit a PR for this if it sounds good.