Unable to download root key pkg on Ubuntu 20.04

[draco961]

I have deployed the following packages on my gateway running Ubuntu 20.04

deliveryoptimization-agent_1.0.0_armhf.deb
deviceupdate-agent_1.1.0_armhf.deb
libdeliveryoptimization_1.0.0_armhf.deb

The state is the following

• There is not IoT Edge runtime running.
• I use a simple connection string to communicate with IoT Hub set up in the "/etc/adu/du-config.json"
• The "adu" download folder has the right right.
• I am able create an import manifest.
• I can deploy the update to the gateway from IoT Hub
• The gateway see there is a new update. and try to install

Device Update agent fails on download a root key pkg. Any idea why ?

Logs:

-- Logs begin at Wed 2025-04-30 14:38:18 CEST, end at Wed 2025-04-30 15:09:58 CEST. --
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: <- 15:07:56 CONNACK | SESSION_PRESENT: true | RETURN_CODE: 0x0
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.1109Z 26176[26176] [D] IotHub connection status: 0, reason: 6 [IoTHub_CommunicationManager_ConnectionStatus_Callback:212]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: -> 15:07:56 SUBSCRIBE | PACKET_ID: 2 | TOPIC_NAME: $iothub/twin/res/# | QOS: 0
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: <- 15:07:56 SUBACK | PACKET_ID: 2 | RETURN_CODE: 0
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: -> 15:07:56 PUBLISH | IS_DUP: false | RETAIN: 0 | QOS: DELIVER_AT_MOST_ONCE | TOPIC_NAME: $iothub/twin/GET/?$rid=3
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: <- 15:07:56 PUBLISH | IS_DUP: false | RETAIN: 0 | QOS: DELIVER_AT_MOST_ONCE | TOPIC_NAME: $iothub/twin/res/200/?$rid=3 | PAYLOAD_LEN: 4399
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.5164Z 26176[26176] [D] ComponentName:deviceUpdate, propertyName:service [ADUC_PnP_ComponentClient_PropertyUpdate_Callback:536]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.5176Z 26176[26176] [D] Update Action info string ({"fileUrls":null,"rootKeyPackageUrl":"http:\/\/iot-eu-projectname-dev--iot-eu-projectname-dev.b.nlu.dl.adu.microsoft.com\/WestEurope\/rootkeypackages\/rootkeypackage-2.json","updateManifest":"{\"manifestVersion\":\"5\",\"updateId\":{\"provider\":\"rotarex\",\"name\":\"x2000\",\"version\":\"1.0.1\"},\"compatibility\":[{\"manufacturer\":\"projectname\",\"model\":\"x2000\"}],\"instructions\":{\"steps\":[{\"handler\":\"microsoft\/script:1\",\"files\":[\"f0532a4eb60e6d462\"],\"handlerProperties\":{}}]},\"files\":{\"f0532a4eb60e6d462\":{\"fileName\":\"firmware-deployment-script.sh\",\"sizeInBytes\":62,\"hashes\":{\"sha256\":\"C6gzu\/Wmd1TGl49QIUbZ7MMjPyprg6zkMu8udEC1r7g=\"}}},\"createdDateTime\":\"2025-04-30T06:58:21Z\"}","updateManifestSignature":null,"workflow":{"action":3,"id":"06ed69af-f5a4-4bdb-a64f-b91130efd1fa"}}), property version (12) [OrchestratorUpdateCallback:433]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.5299Z 26176[26176] [D] Processing deployment 06ed69af-f5a4-4bdb-a64f-b91130efd1fa ... [OrchestratorUpdateCallback:446]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.5303Z 26176[26176] [D] Queueing message (t:0, c:0xb6fce9e8, m:{"deviceUpdate":{"__t":"c","agent":{"lastInstallResult":{"stepResults":null,"resultCode":1,"extendedResultCodes":"00000000","resultDetails":null},"state":6,"workflow":{"action":3,"id":"06ed69af-f5a4-4bdb-a64f-b91130efd1fa"}}}}) [ADUC_D2C_Message_SendAsync:526]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.5307Z 26176[26176] [E] stat path '/var/lib/adu/downloads/06ed69af-f5a4-4bdb-a64f-b91130efd1fa/rootkeypackage-2.json' failed: 2 [SystemUtils_IsFile:734]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.5309Z 26176[26176] [D] Attempting download of 'http://iot-eu-projectname-dev--iot-eu-projectname-dev.b.nlu.dl.adu.microsoft.com/WestEurope/rootkeypackages/rootkeypackage-2.json' using 'DO' [ADUC_RootKeyPackageUtils_DownloadPackage:109]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.5309Z 26176[26176] [I] Downloading File 'http://iot-eu-projectname-dev--iot-eu-projectname-dev.b.nlu.dl.adu.microsoft.com/WestEurope/rootkeypackages/rootkeypackage-2.json' to '/var/lib/adu/downloads/06ed69af-f5a4-4bdb-a64f-b91130efd1fa/rootkeypackage-2.json' [DownloadRootKeyPkg_DO:22]
*** BEGIN ERROR *** 
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7975Z 26176[26176] [E] DO error, msg: unrecognized error, code: 0x8007023e, timeout? 0 [DownloadRootKeyPkg_DO:36]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7978Z 26176[26176] [E] Unable to download root key pkg. [ADUC_RootKeyPackageUtils_DownloadPackage:115]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7980Z 26176[26176] [E] Fail update root keys, ERC 0xa017023e [RootKeyWorkflow_UpdateRootKeys:168]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7981Z 26176[26176] [E] Update Rootkey failed, 0xa017023e. Deployment cannot proceed. [OrchestratorUpdateCallback:466]
*** END ERROR *** 
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7978Z 26176[26176] [I] Download rc: 0, erc: 0xa017023e [DownloadRootKeyPkg_DO:51]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7987Z 26176[26176] [D] Replacing existing pending message. (t:0, s:{"deviceUpdate":{"__t":"c","agent":{"lastInstallResult":{"stepResults":null,"resultCode":1,"extendedResultCodes":"00000000","resultDetails":null},"state":6,"workflow":{"action":3,"id":"06ed69af-f5a4-4bdb-a64f-b91130efd1fa"}}}}) [ADUC_D2C_Message_SendAsync:521]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7989Z 26176[26176] [D] Send message completed (status:5) [OnUpdateResultD2CMessageCompleted:64]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7989Z 26176[26176] [D] Queueing message (t:0, c:0xb6fbfdc8, m:{"deviceUpdate":{"__t":"c","agent":{"lastInstallResult":{"resultCode":0,"extendedResultCodes":"A017023E","resultDetails":null},"state":255,"workflow":{"action":3,"id":"06ed69af-f5a4-4bdb-a64f-b91130efd1fa"}}}}) [ADUC_D2C_Message_SendAsync:526]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7992Z 26176[26176] [I] OrchestratorPropertyUpdateCallback ended [OrchestratorUpdateCallback:517]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7993Z 26176[26176] [I] Processing existing Device Twin data after agent started. [ADUC_PnPDeviceTwin_Callback:647]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7993Z 26176[26176] [D] Notifies components that all callback are subscribed. [ADUC_PnPDeviceTwin_Callback:650]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7994Z 26176[26176] [I] Perform startup tasks. [ADUC_Workflow_HandleStartupWorkflowData:355]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7994Z 26176[26176] [I] There's no update actions in current workflow (first time connected to IoT Hub). [ADUC_Workflow_HandleStartupWorkflowData:360]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7995Z 26176[26176] [D] cancellationType(1) => Normal [ADUC_Workflow_HandleUpdateAction:628]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7995Z 26176[26176] [W] Setting 'OperationCancelled' when there's no active workflow. (value:0) [workflow_set_operation_cancel_requested:2096]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7995Z 26176[26176] [I] Cancel received with no operation in progress - returning to Idle state [ADUC_Workflow_HandleUpdateAction:658]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.7998Z 26176[26176] [D] Queueing message (t:5, c:0xb6fbd2e8, m:{"deviceUpdate":{"__t":"c","agent":{"deviceProperties":{"manufacturer":"projectname","model":"x2000","interfaceId":null,"contractModelId":"dtmi:azure:iot:deviceUpdateContractModel;3","aduVer":"DU;agent\/1.1.0"},"compatPropertyNames":"manufacturer,model"}}}) [ADUC_D2C_Message_SendAsync:526]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8001Z 26176[26176] [I] DeviceInformation component is ready - reporting properties [DeviceInfoInterface_Connected:166]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8002Z 26176[26176] [I] Property manufacturer changed to projectname [RefreshDeviceInfoInterfaceData:131]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8003Z 26176[26176] [I] Property model changed to x2000 [RefreshDeviceInfoInterfaceData:131]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8013Z 26176[26176] [I] Property osName changed to Ubuntu [RefreshDeviceInfoInterfaceData:131]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8016Z 26176[26176] [I] Property swVersion changed to 20.04.2 LTS (Focal Fossa) [RefreshDeviceInfoInterfaceData:131]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8017Z 26176[26176] [I] Property processorArchitecture changed to armv7l [RefreshDeviceInfoInterfaceData:131]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8401Z 26176[26176] [I] Property processorManufacturer changed to ARM [RefreshDeviceInfoInterfaceData:131]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8403Z 26176[26176] [I] Property totalMemory changed to 1004236 [RefreshDeviceInfoInterfaceData:131]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8404Z 26176[26176] [I] Property totalStorage changed to 2392752 [RefreshDeviceInfoInterfaceData:131]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8407Z 26176[26176] [D] Queueing message (t:2, c:0xb6fbd410, m:{"deviceInformation":{"__t":"c","manufacturer":"projectname","model":"x2000","osName":"Ubuntu","swVersion":"20.04.2 LTS (Focal Fossa)","processorArchitecture":"armv7l","processorManufacturer":"ARM","totalMemory":1004236,"totalStorage":2392752}}) [ADUC_D2C_Message_SendAsync:526]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8411Z 26176[26176] [I] DiagnosticsInterface is connected [DiagnosticsInterface_Connected:97]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8412Z 26176[26176] [D] Sending D2C message (t:0, retries:0). [ProcessMessage:390]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8413Z 26176[26176] [D] Sending D2C message:
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: {"deviceUpdate":{"__t":"c","agent":{"lastInstallResult":{"resultCode":0,"extendedResultCodes":"A017023E","resultDetails":null},"state":255,"workflow":{"action":3,"id":"06ed69af-f5a4-4bdb-a64f-b91130efd1fa"}}}} [ADUC_D2C_Default_Message_Transport_Function:580]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8415Z 26176[26176] [D] Sending D2C message (t:2, retries:0). [ProcessMessage:390]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8415Z 26176[26176] [D] Sending D2C message:
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: {"deviceInformation":{"__t":"c","manufacturer":"projectname","model":"x2000","osName":"Ubuntu","swVersion":"20.04.2 LTS (Focal Fossa)","processorArchitecture":"armv7l","processorManufacturer":"ARM","totalMemory":1004236,"totalStorage":2392752}} [ADUC_D2C_Default_Message_Transport_Function:580]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8417Z 26176[26176] [D] Sending D2C message (t:5, retries:0). [ProcessMessage:390]
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: 2025-04-30T13:07:56.8418Z 26176[26176] [D] Sending D2C message:
Apr 30 15:07:56 ubuntu AducIotAgent[26176]: {"deviceUpdate":{"__t":"c","agent":{"deviceProperties":{"manufacturer":"projectname","model":"x2000","interfaceId":null,"contractModelId":"dtmi:azure:iot:deviceUpdateContractModel;3","aduVer":"DU;agent\/1.1.0"},"compatPropertyNames":"manufacturer,model"}}} [ADUC_D2C_Default_Message_Transport_Function:580]

[jw-msft]

If applicable, I would suggest opening a SR ticket as per https://github.com/Azure/iot-hub-device-update/blob/develop/SUPPORT.md:

Customers with an Azure support plan can open an Azure support ticket. We recommend this option if your problem requires immediate attention.

If not, are there any interesting details for the following?

grep -B10 -A10 /var/log/deliveryoptimization-agent/do-agent.*.log rootkeypackage

do-client v1.1.0 would be preferred but it seems only debian10 (not ubunut) arm32 artifact is available:
https://github.com/microsoft/do-client/releases/tag/v1.1.0

Would debian10 be an option, or you must use ubuntu 20.04 on armhf?

[jw-msft]

Searching the DO code, I did not see 0x8007023e.

It would be good to open an issue about that error code here at microsoft/do-client repo for this: https://github.com/microsoft/do-client/issues.

I would also suggest a tcpdump or wireshark trace for traffic on port 80 when reproing and/or for the DNS resolutions for host iot-eu-projectname-dev--iot-eu-projectname-dev.b.nlu.dl.adu.microsoft.com

There is this related issue on do-client repo:
microsoft/do-client#119

[draco961]

I have opened a ticket at Azure

Here is the content of the /var/log/deliveryoptimization-agent/do-agent. * file
There might be a problem with the admin.config.json file but I don't know what to put inside, mine is empty.

2025-05-12T11:12:35.0947380Z 641   641   info     {Run} Started, DU;deliveryoptimization-agent/v1.0.0+20221028.174510.b61de2d (HEAD)
2025-05-12T11:13:08.9574610Z 641   642   info     {_TryRefresh} Read json config file /etc/deliveryoptimization-agent/admin-config.json
******* BEGIN ERROR ******
2025-05-12T11:13:08.9603490Z 641   642   error    (hr:8007023E) std::exception: getentropy [/code/client-lite/src/ipc/rest_api_request.cpp, 81]
2025-05-12T11:14:21.6028790Z 641   642   info     {_TryRefresh} Read json config file /etc/deliveryoptimization-agent/admin-config.json
2025-05-12T11:14:21.6033630Z 641   642   error    (hr:8007023E) std::exception: getentropy [/code/client-lite/src/ipc/rest_api_request.cpp, 81]
2025-05-12T11:14:24.6455880Z 641   642   warn     {_OnData} Socket receive error: 2, End of file
2025-05-12T11:14:26.4191790Z 641   642   warn     {_OnData} Socket receive error: 2, End of file
2025-05-12T11:14:33.9445310Z 641   642   error    (hr:8007023E) std::exception: getentropy [/code/client-lite/src/ipc/rest_api_request.cpp, 81]
2025-05-12T12:02:35.4606860Z 641   642   info     {_TryRefresh} Read json config file /etc/deliveryoptimization-agent/admin-config.json
2025-05-12T12:02:35.4611780Z 641   642   error    (hr:8007023E) std::exception: getentropy [/code/client-lite/src/ipc/rest_api_request.cpp, 81]

Migrate to debian 10 is unfortunately not an option :(

I tried to search with the keyword 8007023e and I found something someone with similar problem, it appears it is a problem of rights on that thread, I tried what they propose but nothing seems to work

I tried the dump file but I don't know how to do that correctly, I continue investiguation

[jw-msft]

@draco961 Would you be able to try the curl downloader to evaluate if it works when DO is not in the equation? If so, this is how to switch to curl content downloader and how to switch back to DO downloader if needed:

# Register Curl content downloader
sudo /usr/bin/AducIotAgent -l0 -extension-type contentDownloader --register-extension /var/lib/adu/extensions/sources/libcurl_content_downloader.so

# Register DO content downloader
sudo /usr/bin/AducIotAgent -l0 -extension-type contentDownloader --register-extension /var/lib/adu/extensions/sources/libdeliveryoptimization_content_downloader.so

[draco961]

When I run sudo /usr/bin/AducIotAgent -l0 -extension-type contentDownloader --register-extension /var/lib/adu/extensions/sources/libcurl_content_downloader.so

I have the error message Unsupported short argument -x.

[jw-msft]

@draco961 where did you get the following DO packages from?
deliveryoptimization-agent_1.0.0_armhf.deb
libdeliveryoptimization_1.0.0_armhf.deb

I did not see them on github releases page or in packages.microsoft.com for ubuntu 20.04.

Are they by chance the debian10 armhf .deb packages such as
https://packages.microsoft.com/debian/10/prod/pool/main/d/deliveryoptimization-agent/deliveryoptimization-agent_1.0.0_armhf.deb
and
https://packages.microsoft.com/debian/10/prod/pool/main/libd/libdeliveryoptimization/libdeliveryoptimization_1.0.0_arm64.deb
?

Similarly, there is no deviceupdate-agent armhf .deb pkg for ubuntu 20.04, but there is one for debian 10. Therefore, did you use this .deb package by chance?

https://packages.microsoft.com/debian/10/prod/pool/main/d/deviceupdate-agent/deviceupdate-agent_1.1.0_armhf.deb

I have this ubuntu 20.04.05 server image,
https://cdimage.ubuntu.com/releases/focal/release/ubuntu-20.04.5-preinstalled-server-armhf+raspi.img.xz
, running on a Raspberry Pi 3B+ and am now trying the debian 10 armhf .deb packages on it.

Is this the same ubuntu 20.04.05 image you used?

[jw-msft]

When I run sudo /usr/bin/AducIotAgent -l0 -extension-type contentDownloader --register-extension /var/lib/adu/extensions/sources/libcurl_content_downloader.so

I have the error message Unsupported short argument -x.

There was a typo. It should be two dashes in front of extension-type. Here is the corrected command (NOTE the --extension-type instead of -extension-type :

sudo /usr/bin/AducIotAgent -l0 --extension-type contentDownloader --register-extension /var/lib/adu/extensions/sources/libcurl_content_downloader.so

[draco961]

I downloaded the packages from here
https://packages.microsoft.com/repos/microsoft-debian-bullseye-prod/pool/main/...
and you ask me from here
https://packages.microsoft.com/debian/10/prod/pool/main/...

Is there any difference ?
I tried with the packages from your link and same issue :(

I tried with

sudo /usr/bin/AducIotAgent -l0 --extension-type contentDownloader --register-extension /var/lib/adu/extensions/sources/libcurl_content_downloader.so

Something is happening, it looks like it register the downloader but shutdown deviceupdate-agent. If I restart it, DO is used again

[jw-msft]

that cmd is for registering content downloader extension that agent will use and the logs show it did so successfully.

To start the agent, use systemctl start deviceupdate-agent or to get logs on the console it can be run standalone via sudo /usr/bin/AducIotAgent as before.

[draco961]

Is this feature working with deviceudpate-agent version 1.1.0 ?

[jw-msft]

For now, to enable downloading of the rootkey package with curl, one must rebuild the agent from sources:

git clone https://github.com/azure/iot-hub-device-update
cd iot-hub-device-update
./scripts/build.sh --rootkeypkg-curl --build-packages
ls -l out/*.deb

The approach given above would work in a future minor release (the registration only affects downloading update payload content currently) once additional work is done to enable it so that content downloader contract allows for downloading rootkey package in addition to update payloads.