Using curl content downloader still relies on DeliveryOptimization DO-client for rootkey download

[AndreRicardo-Zoetis]

We are trying to test OTA in Docker containers but noticed that even though there is an entry to use libcurl_content_downloader

iot-hub-device-update/scripts/docker/build_docker_image.sh

Line 197 in 35de328

create_content_downloader_registration 'libcurl_content_downloader'

The root key workflow, only seems to work with "DO", am I missing something?

iot-hub-device-update/src/rootkey_workflow/src/rootkey_workflow.c

Lines 46 to 50 in 35de328

	ADUC_RootKeyPkgDownloaderInfo rootkey_downloader_info = {
	.name = "DO", // DeliveryOptimization
	.downloadFn = DownloadRootKeyPkg_DO,
	.downloadBaseDir = workFolder,
	};

Modified /var/lib/adu/extensions/content_downloader/extension.json to use libcurl_content_downloader.so

{
   "fileName":"/var/lib/adu/extensions/sources/libcurl_content_downloader.so",
   "sizeInBytes":0,
   "hashes": {
        "sha256":"4M06Q8dj1JNNtcSDnftZgwPeIvHycqYJphiGsjfKpwo="
   }
}

In the end because deliveryoptimization-agent is not running on the container, the OTA fails with:

2025-03-25T13:43:14.3848Z 38423[38423] [D] Attempting download of 'http://[redacted].microsoft.com/WestUS2/rootkeypackages/rootkeypackage-1.json' using 'DO' [ADUC_RootKeyPackageUtils_DownloadPackage:114]
2025-03-25T13:43:14.3848Z 38423[38423] [I] Downloading File 'http://[redacted].microsoft.com/WestUS2/rootkeypackages/rootkeypackage-1.json' to '/var/lib/adu/downloads/a8ab014a-82ac-49f7-b405-8f5ce2c9f2d0/rootkeypackage-1.json' [DownloadRootKeyPkg_DO:22]
2025-03-25T13:43:25.3935Z 38423[38423] [E] DO error, msg: unrecognized error, code: 0x80d01001, timeout? 0 [DownloadRootKeyPkg_DO:36]
2025-03-25T13:43:25.3937Z 38423[38423] [I] Download rc: 0, erc: 0xa0101001 [DownloadRootKeyPkg_DO:51]
2025-03-25T13:43:25.3937Z 38423[38423] [E] Unable to download root key pkg. [ADUC_RootKeyPackageUtils_DownloadPackage:120]
2025-03-25T13:43:25.3938Z 38423[38423] [E] Failed download of rootkey package, ERC: 0xa0101001 [RootKeyWorkflow_UpdateRootKeys:66]
2025-03-25T13:43:25.3938Z 38423[38423] [E] Fail update root keys, ERC 0xa0101001 [RootKeyWorkflow_UpdateRootKeys:178]
2025-03-25T13:43:25.3938Z 38423[38423] [E] Update Rootkey failed, 0xa0101001. Deployment cannot proceed. [OrchestratorUpdateCallback:398]

Do we need to write the equivalent of src/utils/rootkeypackage_utils/src/rootkeypackage_do_download.cpp but for curl?

[jw-msft]

Yes, or it's probably better to change it to lookup the registered content downloader and use it like when downloading payloads.

[jw-msft]

@AndreRicardo-Zoetis
Draft PR: #717
Will change it to non-draft when done testing.

[jw-msft]

Tested with curl for root key download.
curl content downloader for payloads is not regressed and working.
DO for rootkey pkg download and payloads is also not regressed and working.
set to non-draft state.

[AndreRicardo-Zoetis]

@jw-msft can you please also have a look at the two comments I've made on your closed PR? Not sure you get notified for those.

#717 (comment)

#717 (comment)

[jw-msft]

@jw-msft can you please also have a look at the two comments I've made on your closed PR? Not sure you get notified for those.

#717 (comment)

#717 (comment)

@AndreRicardo-Zoetis somehow missed these. See #736 for your 2 comments.

[AndreRicardo-Zoetis]

@jw-msft thank you for the changes above.

Something I've suggested to @eshashah-msft this week regarding these changes.

To us it was very surprising that changing the contentDownloader to curl, did not change the root key package download to use curl as well, and still required do-client to be installed and running.

AducIotAgent -l 2 --extension-type contentDownloader --register-extension /var/lib/adu/extensions/sources/libcurl_content_downloader.so

Would it be possible to change the root key package download to align with contentDownloader setting? Make it an install/runtime setting rather than having to decide at compile time with ADUC_ROOTKEY_PKG_DOWNLOAD_WITH_CURL ?

PS: This also aligns with your preference to not have to distribute multiple .deb packages.

[jw-msft]

@AndreRicardo-Zoetis I think this makes sense. ideally it would rev the content downloader contract 1.1 (with the .so) to add an export fn that can download arbitrary url without needing to provide FileEntity, etc, although it would have to handle the case when the registered content downloader is 1.0 contract.

Another option is to just key off of the registered content downloader (and using the existing rootkey downloaders) but then it would need to add the concept of being able to query for "well-known" 1st party content downloaders. I think this latter idea is not as good as 1st idea.

Probably another nuance here is maybe it should just have 1stparty supported downloaders as provided so the internal APIs on them are more fluid and not make them content downloader extensions (as they are today), which would then only be used to override. This would allow avoiding the contract api stuff for 1st party content downloaders (DO and curl).

Then a runtime config would be added to switch between curl and DO--have to decide if it makes sense to have orthogonal settings for rootkeypkg and update payloads. It probably makes sense to only allow all DO or all curl, but keeping the flexibility for other combinations might make sense.

Also, I think it would make sense to use libcurl directly instead of exec of curl CLI process, but that is an orthogonal improvement.

(To be clear, the above would not make it into the 1.2.0 release but could be part of a follow-up minor release)

[AndreRicardo-Zoetis]

"Also, I think it would make sense to use libcurl directly instead of exec of curl CLI process, but that is an orthogonal improvement."

Using libcurl with progress would be of great value to us because we have customers with slow internet speeds and our support field agents have requested for a download progress.
Currently we only give feedback that the download started but sometimes the process is interrupted out of frustration, partially due to the lack of feedback on the download.

[jw-msft]

Using libcurl with progress would be of great value to us because we have customers with slow internet speeds and our support field agents have requested for a download progress.
Currently we only give feedback that the download started but sometimes the process is interrupted out of frustration, partially due to the lack of feedback on the download.

@AndreRicardo-Zoetis How would the progress reports make it to your customers and how is it currently knowing about the download starts?
Would it need to send it on a separate channel of communication (e.g. mqtt server, az append blob, etc)?
(Reporting it as a twin PATCH would quickly eat the message quota)

[AndreRicardo-Zoetis]

Using libcurl with progress would be of great value to us because we have customers with slow internet speeds and our support field agents have requested for a download progress.
Currently we only give feedback that the download started but sometimes the process is interrupted out of frustration, partially due to the lack of feedback on the download.

@AndreRicardo-Zoetis How would the progress reports make it to your customers and how is it currently knowing about the download starts? Would it need to send it on a separate channel of communication (e.g. mqtt server, az append blob, etc)? (Reporting it as a twin PATCH would quickly eat the message quota)

Progress is reported in the device screen, in the UI, not sent to Azure IoT.

Currently we extended AducIotAgent to use NNG and we expose several IPC sockets to allow other processes to know about the AducIotAgent status and also send telemetry to Azure IoT Hub.