-
Notifications
You must be signed in to change notification settings - Fork 0
Labels
enhancementNew feature or requestNew feature or request
Description
In audeering/audmodel#14 we have seen that we can speed up downloading large files at least with MinIO by using multiple workers.
I suggest we add a num_workers and chunk_size argument to Backend.get_file() and Backend.get_archive(). A backend implementation may then implement parallel download or simply ignore the arguments if not applicable. The question is what should be the default parameters. For num_workers it is straight-forward, we can simply set it to 1. For chunk_size it is not that obvious because the best choice may depend on the backend . Hence, I suggest we set None and - in that case we let the backend choose the chunk size.
/cc @hagenw
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request