Skip to content

Speed up file download by using multiple workers #264

@frankenjoe

Description

@frankenjoe

In audeering/audmodel#14 we have seen that we can speed up downloading large files at least with MinIO by using multiple workers.

I suggest we add a num_workers and chunk_size argument to Backend.get_file() and Backend.get_archive(). A backend implementation may then implement parallel download or simply ignore the arguments if not applicable. The question is what should be the default parameters. For num_workers it is straight-forward, we can simply set it to 1. For chunk_size it is not that obvious because the best choice may depend on the backend . Hence, I suggest we set None and - in that case we let the backend choose the chunk size.

/cc @hagenw

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions