Support HTTP 301 and 307 Redirects with Cache-Control Logic in Backend HTTP Client #4096
Description
Description:
We need to enhance the backend HTTP client in Dragonfly to support HTTP 301 (Moved Permanently) and 307 (Temporary Redirect) redirects while adhering to the Cache-Control header logic. This improvement is crucial for optimizing blob downloads from Harbor.
Currently, when Dragonfly downloads blobs from Harbor using the provided Blobs URL, it initiates Range requests based on the Piece Length. However, Harbor's Blobs URL often redirects to an object storage URL, resulting in frequent 307 (Temporary Redirect) responses. Without proper handling of the Cache-Control header, Dragonfly repeatedly sends requests to Harbor, leading to increased load and inefficiency.
Problem:
Harbor Blobs URLs trigger multiple 307 redirects to object storage URLs.
The backend HTTP client does not currently respect Cache-Control directives, which could otherwise reduce redundant requests to Harbor by caching redirect responses or respecting no-cache directives.
Proposed Solution:
Implement support for HTTP 301 and 307 redirects in Dragonfly's backend HTTP client.
Ensure the client adheres to the Cache-Control header logic during redirects, such as caching redirect locations when allowed or avoiding caching when instructed (e.g., no-cache or no-store).
Optimize Range requests to minimize unnecessary interactions with Harbor by leveraging cached redirect information when appropriate.
Expected Benefits:
Reduced number of requests to Harbor due to proper handling of cached redirect responses.
Improved download performance and efficiency for blobs redirected to object storage.
Better compliance with HTTP standards for redirects and caching.
Additional Context:
This feature is particularly important for scenarios involving Harbor and object storage integrations, where redirects are common.
The implementation should handle edge cases such as invalid or missing Cache-Control headers and ensure that Range requests continue to function correctly after redirects.
Task:
Update the HTTP client in Dragonfly to handle 301 and 307 redirects.
Implement logic to parse and respect Cache-Control headers during redirects.
Add relevant tests to verify correct behavior for redirect caching and Range requests.
Document the behavior and any configuration options for users.