HSDS azure kubernetes lockup

[Leonard-Mueller]

Hi, I have been running hsds on an azure kubernetes service cluster for a couple months and it has been very solid so far. Recently I ran into some issues with the hsds cluster suddenly becoming unresponsive. It says it is still operational when curling the /about endpoint.
Running the hsinfo endpoint sometimes works and sometimes it returns this error:

username: admin (admin)
password: ********************************
ERROR 2025-05-12 13:02:47,642 got <class 'requests.exceptions.RetryError'> exception: HTTPConnectionPool(host='<hostip>', port=80): Max retries exceeded with url: /?domain=%2Fhome (Caused by ResponseError('too many 503 error responses'))
home: NO ACCESS
server version: 0.9.1
node count: 4
up: 20 hours 21 min 47 sec
h5pyd version: 0.19.0

running the hsls command also returns an error but it managed to run it once in the broken state:

                                                    folder    /
admin                                               folder   2025-01-14 13:40:43 /home/
ERROR 2025-05-12 13:02:58,239 got <class 'requests.exceptions.RetryError'> exception: HTTPConnectionPool(host=<hostip>, port=80): Max retries exceeded with url: /?domain=%2Fhome (Caused by ResponseError('too many 503 error responses'))

What is noticable is that all the pods say they were only started recently, a time which correlates with the first not all dn_ids found erros. It was no manual restart but since also the cluster nodes say they were created at that time this suggests an automatic update of the node operating system or something similar (unfortunately was not able to verify within the aks logs). It seems like the automatic restarting of the pods does not work properly. I had this issue occure in two different aks clusters where I host hsds.

When I manually restart all pods of the deployment via kubectl rollout restart deployment hsds it runs smoothly again.

Some logs of the hsds pods:

Defaulted container "sn" out of: sn, dn
2025-05-11T14:41:01.188972556Z WARN> ClientError: Cannot connect to host 10.244.1.8:6101 ssl:default [Connect call failed ('10.244.1.8', 6101)]
2025-05-11T14:41:01.189338640Z ERROR> k8s_get_dn_urls - Exception: Internal Server Error from /info request
2025-05-11T14:41:01.192585103Z WARN> scaling - got 3 dn_ids expected 4
2025-05-11T14:41:01.192715000Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:11.255341427Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:21.343061680Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:31.374071584Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:41.420208288Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:51.469857952Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:01.517216379Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:11.580419795Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:21.643042332Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:31.704066186Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:41.755255130Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:51.810123758Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:43:01.915280706Z WARN> scaling - node_numbers not consecutive - got: [2, 3]
2025-05-11T17:45:27.859414157Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T04:12:21.586426975Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T05:11:29.196915404Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T05:13:49.556992518Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T07:07:20.263299333Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T10:09:10.710862955Z WARN> select param is 620 characters long
2025-05-12T10:09:10.721442661Z WARN> select param is 721 characters long
2025-05-12T10:09:10.722004151Z WARN> select param is 721 characters long
2025-05-12T10:09:10.722612712Z WARN> select param is 721 characters long
2025-05-12T10:09:10.723117726Z WARN> select param is 721 characters long
2025-05-12T10:09:10.757419197Z WARN> select param is 720 characters long
2025-05-12T10:09:20.973273745Z WARN> select param is 620 characters long
2025-05-12T10:09:20.980846193Z WARN> select param is 620 characters long
2025-05-12T10:09:31.121470759Z WARN> select param is 703 characters long
2025-05-12T10:09:31.131655977Z WARN> select param is 620 characters long
2025-05-12T10:09:31.144136849Z WARN> select param is 620 characters long
2025-05-12T10:17:58.077461606Z WARN> select param is 620 characters long
2025-05-12T10:17:58.081012981Z WARN> select param is 703 characters long
2025-05-12T10:17:58.107836748Z WARN> select param is 512 characters long
2025-05-12T10:18:08.295661050Z WARN> select param is 620 characters long
2025-05-12T10:18:08.296238855Z WARN> select param is 620 characters long
2025-05-12T10:18:08.304142039Z WARN> select param is 620 characters long
2025-05-12T10:18:08.318738679Z WARN> select param is 512 characters long
2025-05-12T10:18:08.338515108Z WARN> select param is 821 characters long
2025-05-12T10:18:18.439841846Z WARN> select param is 620 characters long
2025-05-12T10:18:18.449873429Z WARN> select param is 620 characters long
2025-05-12T10:18:18.452095883Z WARN> select param is 620 characters long
2025-05-12T11:26:03.116042296Z WARN> Object: http://10.244.1.8:6101/domains not found
2025-05-12T11:26:03.116308489Z WARN> domain: hsdscontainer/db not found
2025-05-12T11:26:03.116316421Z WARN> fetch result - not found error for: /db
2025-05-12T11:26:03.120221251Z WARN> get_domains - domain: /db not found in crawler dict
2025-05-12T11:26:03.204039312Z WARN> Object: http://10.244.1.8:6101/domains not found
2025-05-12T11:26:03.204239955Z WARN> domain: hsdscontainer/db not found
2025-05-12T11:26:03.204248148Z WARN> fetch result - not found error for: /db
2025-05-12T11:26:03.204285204Z WARN> get_domains - domain: /db not found in crawler dict

2025-05-11T14:41:01.132076863Z WARN> ClientError: Cannot connect to host 10.244.1.8:6101 ssl:default [Connect call failed ('10.244.1.8', 6101)]
2025-05-11T14:41:01.132455476Z ERROR> k8s_get_dn_urls - Exception: Internal Server Error from /info request
2025-05-11T14:41:01.135939658Z WARN> scaling - got 3 dn_ids expected 4
2025-05-11T14:41:01.136062093Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:11.183341071Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:21.234603968Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:31.281539794Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:41.328286034Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:41:51.402948947Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:01.474730805Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:11.556206076Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:21.643824831Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:31.718458517Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:41.774393702Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:42:51.836871434Z WARN> not all dn_ids found, got: ['dn-ecbed', 'dn-7811b', 'dn-90954']
2025-05-11T14:43:01.917405594Z WARN> scaling - node_numbers not consecutive - got: [2, 3]
2025-05-11T15:54:13.376909288Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-11T16:11:18.436844293Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-11T16:45:18.576590760Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-11T17:50:58.147116960Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-11T22:49:08.902611889Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T00:36:51.453031159Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T00:46:34.141227642Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T00:58:27.314176357Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T02:55:42.823295632Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T03:56:51.112958156Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T10:09:10.726935246Z WARN> select param is 620 characters long
2025-05-12T10:09:10.748468746Z WARN> select param is 620 characters long
2025-05-12T10:09:20.956553496Z WARN> select param is 620 characters long
2025-05-12T10:09:20.959156595Z WARN> select param is 620 characters long
2025-05-12T10:09:20.978175382Z WARN> select param is 620 characters long
2025-05-12T10:09:20.992750976Z WARN> select param is 821 characters long
2025-05-12T10:09:31.113686692Z WARN> select param is 721 characters long
2025-05-12T10:09:31.114547264Z WARN> select param is 721 characters long
2025-05-12T10:09:31.115527327Z WARN> select param is 721 characters long
2025-05-12T10:09:31.116513298Z WARN> select param is 721 characters long
2025-05-12T10:09:31.117506281Z WARN> select param is 620 characters long
2025-05-12T10:09:31.127903420Z WARN> select param is 512 characters long
2025-05-12T10:18:18.439652434Z WARN> select param is 731 characters long
2025-05-12T10:18:18.441464792Z WARN> select param is 731 characters long
2025-05-12T10:18:18.457957505Z WARN> select param is 620 characters long
2025-05-12T11:02:50.994200826Z WARN> Object: http://10.244.1.8:6101/domains not found
2025-05-12T11:02:50.994490955Z WARN> domain: hsdscontainer/db not found
2025-05-12T11:02:50.994498176Z WARN> fetch result - not found error for: /db
2025-05-12T11:02:50.994586971Z WARN> get_domains - domain: /db not found in crawler dict
2025-05-12T11:02:51.057151852Z WARN> Object: http://10.244.1.8:6101/domains not found
2025-05-12T11:02:51.057360348Z WARN> domain: hsdscontainer/db not found
2025-05-12T11:02:51.057366878Z WARN> fetch result - not found error for: /db
2025-05-12T11:02:51.057444105Z WARN> get_domains - domain: /db not found in crawler dict

Defaulted container "sn" out of: sn, dn
2025-05-11T14:44:19.683711106Z WARN> scaling - node_numbers not consecutive - got: [-1, -1, 0, 1]
2025-05-11T17:05:08.308980409Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-11T20:46:56.847148063Z ERROR> Unexpected HTTPInternalServerError exception in doHealthCheck: Internal Server Error
2025-05-12T10:08:51.956746127Z WARN> Object: http://10.244.1.8:6101/domains not found
2025-05-12T10:08:51.956946269Z WARN> domain: hsdscontainer/db not found
2025-05-12T10:08:51.956953290Z WARN> fetch result - not found error for: /db
2025-05-12T10:08:51.957536942Z WARN> get_domains - domain: /db not found in crawler dict
2025-05-12T10:08:52.038433833Z WARN> Object: http://10.244.1.8:6101/domains not found
2025-05-12T10:08:52.038653385Z WARN> domain: hsdscontainer/db not found
2025-05-12T10:08:52.038661127Z WARN> fetch result - not found error for: /db
2025-05-12T10:08:52.038807758Z WARN> get_domains - domain: /db not found in crawler dict
2025-05-12T10:09:10.709447534Z WARN> select param is 731 characters long
2025-05-12T10:09:10.710076395Z WARN> select param is 731 characters long
2025-05-12T10:09:10.721969825Z WARN> select param is 620 characters long
2025-05-12T10:09:10.723074826Z WARN> select param is 703 characters long
2025-05-12T10:09:10.727338693Z WARN> select param is 620 characters long
2025-05-12T10:09:10.727926862Z WARN> select param is 620 characters long
2025-05-12T10:09:10.731749611Z WARN> select param is 620 characters long
2025-05-12T10:09:10.738564935Z WARN> select param is 620 characters long
2025-05-12T10:09:20.951849817Z WARN> select param is 731 characters long
2025-05-12T10:09:20.951907544Z WARN> select param is 731 characters long
2025-05-12T10:09:20.966722196Z WARN> select param is 703 characters long
2025-05-12T10:09:20.969765496Z WARN> select param is 620 characters long
2025-05-12T10:09:20.970208425Z WARN> select param is 620 characters long
2025-05-12T10:09:20.977098671Z WARN> select param is 721 characters long
2025-05-12T10:09:20.977589432Z WARN> select param is 721 characters long
2025-05-12T10:09:20.978174527Z WARN> select param is 721 characters long
2025-05-12T10:09:20.978750968Z WARN> select param is 721 characters long
2025-05-12T10:09:20.979427130Z WARN> select param is 620 characters long
2025-05-12T10:09:31.095829647Z WARN> select param is 731 characters long
2025-05-12T10:09:31.096910061Z WARN> select param is 731 characters long
2025-05-12T10:09:31.105725173Z WARN> select param is 620 characters long
2025-05-12T10:09:31.106109805Z WARN> select param is 620 characters long
2025-05-12T10:09:31.109941155Z WARN> select param is 721 characters long
2025-05-12T10:09:31.110459398Z WARN> select param is 721 characters long
2025-05-12T10:09:31.111022049Z WARN> select param is 721 characters long
2025-05-12T10:09:31.111707535Z WARN> select param is 721 characters long
2025-05-12T10:09:31.122561950Z WARN> select param is 720 characters long
2025-05-12T10:09:31.125992604Z WARN> select param is 620 characters long
2025-05-12T10:09:31.128788470Z WARN> select param is 821 characters long
2025-05-12T10:17:58.071601513Z WARN> select param is 620 characters long
2025-05-12T10:17:58.086751333Z WARN> select param is 721 characters long
2025-05-12T10:17:58.087332322Z WARN> select param is 721 characters long
2025-05-12T10:17:58.087855294Z WARN> select param is 721 characters long
2025-05-12T10:17:58.088446558Z WARN> select param is 721 characters long
2025-05-12T10:17:58.091847233Z WARN> select param is 620 characters long
2025-05-12T10:17:58.093061150Z WARN> select param is 620 characters long
2025-05-12T10:18:08.289839561Z WARN> select param is 731 characters long
2025-05-12T10:18:08.290545409Z WARN> select param is 731 characters long
2025-05-12T10:18:08.314513279Z WARN> select param is 620 characters long
2025-05-12T10:18:08.315536407Z WARN> select param is 620 characters long
2025-05-12T10:18:08.316131788Z WARN> select param is 620 characters long
2025-05-12T10:18:08.318705852Z WARN> select param is 721 characters long
2025-05-12T10:18:08.319222113Z WARN> select param is 721 characters long
2025-05-12T10:18:08.319809522Z WARN> select param is 721 characters long
2025-05-12T10:18:08.320492896Z WARN> select param is 721 characters long
2025-05-12T10:18:08.323175464Z WARN> select param is 620 characters long
2025-05-12T10:18:08.327344392Z WARN> select param is 620 characters long
2025-05-12T10:18:18.448973613Z WARN> select param is 721 characters long
2025-05-12T10:18:18.448991270Z WARN> select param is 721 characters long
2025-05-12T10:18:18.449343232Z WARN> select param is 721 characters long
2025-05-12T10:18:18.449854776Z WARN> select param is 721 characters long
2025-05-12T10:18:18.457936509Z WARN> select param is 512 characters long
2025-05-12T10:18:18.459066187Z WARN> select param is 620 characters long
2025-05-12T10:18:18.465449302Z WARN> select param is 620 characters long

2025-05-12T09:24:29.877099745Z WARN> returning 503 - node_state: WAITING
2025-05-12T09:24:29.878137275Z WARN> returning 503 - node_state: WAITING
2025-05-12T09:24:29.878464891Z WARN> returning 503 - node_state: WAITING
2025-05-12T09:24:29.883460091Z WARN> returning 503 - node_state: WAITING
2025-05-12T09:24:29.893297197Z WARN> returning 503 - node_state: WAITING
2025-05-12T09:24:29.957138787Z WARN> returning 503 - node_state: WAITING
2025-05-12T09:24:29.957742691Z WARN> returning 503 - node_state: WAITING
2025-05-12T09:24:29.966824214Z WARN> returning 503 - node_state: WAITING
2025-05-12T09:24:38.930946205Z WARN> not all dn_ids found, got: ['dn-aeb2d', 'dn-90954', 'dn-f71db']
...
concatinated repetitive warnings
...
2025-05-12T11:39:40.966040865Z WARN> returning 503 - node_state: WAITING
2025-05-12T11:39:40.974103224Z WARN> returning 503 - node_state: WAITING
2025-05-12T11:39:42.976364487Z WARN> returning 503 - node_state: WAITING
2025-05-12T11:39:46.979700681Z WARN> returning 503 - node_state: WAITING
2025-05-12T11:39:48.119343001Z WARN> not all dn_ids found, got: ['dn-aeb2d', 'dn-90954', 'dn-f71db']
2025-05-12T11:39:54.981560289Z WARN> returning 503 - node_state: WAITING
2025-05-12T11:39:58.173053474Z WARN> not all dn_ids found, got: ['dn-aeb2d', 'dn-90954', 'dn-f71db']
2025-05-12T11:39:59.250147390Z WARN> returning 503 - node_state: WAITING
2025-05-12T11:40:08.219730428Z WARN> not all dn_ids found, got: ['dn-aeb2d', 'dn-90954', 'dn-f71db']
2025-05-12T11:40:10.984159278Z WARN> returning 503 - node_state: WAITING

[Leonard-Mueller]

Upon further investigation it seems to me like the last data node was stuck in the WAITING state. I realize that this is a state that should only be temporary and does not directly mean that the node is broken.
Could this temporarily be resolved with a liveness probe that additionally looks for the WAITING state and a less aggressive period and failure threshold?

[jreadey]

So, the pod was running, but the data node container in the pod was in WAITING state? it looks like there was a pod IP being reported by Kubernetes, but when another pod tried sending a health check request to it, it got a Client Error (Cannot connect to host.. error).

HSDS on Kubernetes achieves parallelism by having the SN container in the pod that gets a request to (potentially) distribute actions across all DN containers in all pods. For this to work, everyone has to agree on which pods are running and the ordering of pods (based on IP+port address). When you dynamically change the number of pods using kubectl scale deployment, you'll see the state temporarily go to WAITING before (hopefully) returning to RUNNING again. The delay is caused by the time spent waiting for each pod to agree on the new number of pods and ordering.

Anyway, if you have a chance to try manually scaling the deployment up and down, I'd be curious if you see this issue come up or not.

Also, if you could set the LOG_LEVEL to debug, there might be some additional log lines that would be helpful in understanding what's going on.

Also if the next time this come up, you could do a kubectl describe pod on the pod for the DN in the waiting state, that might be helpful. There might be some additional state that should be inspected so that the pod in question is not included in the list of active pods.

[Leonard-Mueller]

thanks for the clarification. I will have a look at it. I manually scaled the deployment up and down in the past which was no issue.
But manually scaling the underlying node pool has caused a similar issue in the past. I will try to replicate it that way and see if I can get more information.

[Leonard-Mueller]

Hi @jreadey, today I ran into a similar issue. There seems to have been an automated restart of the cluster. I have included the logs of the initial phase in the different files, this time the log level was set to DEBUG:
hsds-78dd9889fc-4dtfw.log
hsds-78dd9889fc-jqdp4.log
hsds-78dd9889fc-jxbh7.log
hsds-78dd9889fc-mzk8x.log

the pod with the jqpd4 suffix was the one having the issues. It was unable to find the data_node id for the jxbh7 pod. I have created a little overview where combinded the ip, dn_id and pod id for easier understanding.

The describe pod command of the pod with the issues had nothing in it:

Name:             hsds-78dd9889fc-jqdp4
Namespace:        default
Priority:         0
Service Account:  default
Node:             aks-hsds8gb-39208979-vmss000000/10.224.0.4
Start Time:       Sun, 15 Jun 2025 10:34:39 +0200
Labels:           app=hsds
                  pod-template-hash=78dd9889fc
Annotations:      <none>
Status:           Running
IP:               10.244.1.8
IPs:
  IP:           10.244.1.8
Controlled By:  ReplicaSet/hsds-78dd9889fc
Containers:
  sn:
    Container ID:   containerd://ba6859b9d25c97c407f4f89eea81f51ba7d192e040559d9b5b7d86c2d313d202
    Image:          hdfgroup/hsds:0.9.1
    Image ID:       docker.io/hdfgroup/hsds@sha256:ce8c8b42703b8cb41a6486d0388ce7333adb523817f14b1b727d20de231f0bfe
    Port:           5101/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 15 Jun 2025 10:34:56 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  1G
    Requests:
      memory:  500M
    Liveness:  http-get http://:5101/info delay=5s timeout=1s period=60s #success=1 #failure=3
    Environment:
      NODE_TYPE:                sn
      HEAD_PORT:                0
      AZURE_CONNECTION_STRING:  <set to the key 'az_conn_str' in secret 'azure-conn-str'>  Optional: false
      POD_IP:                    (v1:status.podIP)
    Mounts:
      /config/config.yml from config (rw,path="config.yml")
      /config/override.yml from override (rw,path="override.yml")
      /config/passwd.txt from accounts (rw,path="passwd.txt")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dxwk8 (ro)
  dn:
    Container ID:   containerd://8271b64eb08c9605994bebb5575aa59feef2e2fb72ebfb741e816e79a2a7211e
    Image:          hdfgroup/hsds:0.9.1
    Image ID:       docker.io/hdfgroup/hsds@sha256:ce8c8b42703b8cb41a6486d0388ce7333adb523817f14b1b727d20de231f0bfe
    Port:           6101/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 15 Jun 2025 10:34:57 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  2G
    Requests:
      memory:  500M
    Liveness:  http-get http://:6101/info delay=5s timeout=1s period=60s #success=1 #failure=3
    Environment:
      NODE_TYPE:                dn
      HEAD_PORT:                0
      AZURE_CONNECTION_STRING:  <set to the key 'az_conn_str' in secret 'azure-conn-str'>  Optional: false
    Mounts:
      /config/config.yml from config (rw,path="config.yml")
      /config/override.yml from override (rw,path="override.yml")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dxwk8 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  accounts:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  user-password
    Optional:    false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      hsds-config
    Optional:  false
  override:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      hsds-override
    Optional:  false
  kube-api-access-dxwk8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Some things I noticed:

• The dn with issues does not report it is in a waiting state.
• The cluster would still try to process a request, The logs of the show that they access data, except for the one that had issues reaching another dn.
• Old pods were present when the cluster restarted. Some are reported as terminating, but some have not started terminating yet. So for a short time the new nodes talk to ones that will be terminated.

I hope this helps with finding the issue :)

[Leonard-Mueller]

Hello, I just wanted to give an update on the issue. It still persists but I was able to reliably trigger it via an aks nodepool upgrade:

az aks nodepool upgrade \
  --resource-group <resource-group> \
  --cluster-name <cluster-name> \
  --name <nodepool-name> \
  --node-image-only

I am working with a rather small nodepool of 2 or 3 nodes, with one extra surge node durning the upgrade process. Suprisingly the cluster runs fine during the upgrade process except when it is in the process of creating and terminating pods (But I think this is expected). Only after all the nodes have been upgraded and the pods are distributed in their final configuration on the new nodes I get the broken state.

I tried this with even and uneven numbers of pods and nodes, with same results.

To fix the cluster I have to run at least one kubectl rollout restart commands on the deployment.

[jreadey]

Thanks for the update!

Do you think the node upgrade was the reason for the first failures you saw? Or is is just a reliable reproducer?

[Leonard-Mueller]

I am quite sure that those upgrades are the reason for all the failures. The issue still persists with the server locking up around every two weeks when azure automatically updates the nodes.

[jreadey]

I'm wondering how I could deliberately trigger this failure to facilitate investigation...

Scaling the number of pods doesn't seem to cause a problem. Does scaling the node pool always trigger the issue?

Feel free to contact me if you'd like to discuss offline.