Healthcheck - best practices

[igolman]

Hi 👋

I was using following HealthCheck configuration for my stackback containers:

  stackback_s3:
    image: 'ghcr.io/lawndoc/stack-back:v1.4.5'
...
    healthcheck:
      test: [ 'CMD', 'rcb', 'status' ]
      interval: 5m
      timeout: 60s
      start_period: 1m
      start_interval: 30s

this worked pretty well, until from some point (I suppose since the deployment schedule became more often) a "stale"-locks start to appear:

/restic-compose-backup # restic list locks
repository fd8b5644 opened (version 2, compression level auto)
3556d98bd04d2cfa04c39fecaf5205003694a04cae899261b01ebf23d60cbbd5
a352ecabe7b4e7b4db3f0966e9e3039d797fab4cc742cc9de6610db445dfb073

/restic-compose-backup # restic  cat lock  3556d98bd04d2cfa04c39fecaf5205003694a04cae899261b01ebf23d60cbbd5
repository fd8b5644 opened (version 2, compression level auto)
{
  "time": "2025-08-19T12:11:28.397181926+02:00",
  "exclusive": false,
  "hostname": "639ac14aba17",
  "username": "root",
  "pid": 18011
}

Those locks do not affect backup, however they do affect the cleanup i.e. the maintenance of the repository:

...
2025-08-21 00:10:18,766 - INFO: Running maintenance tasks
2025-08-21 00:10:18,767 - INFO: Forget outdated snapshots
2025-08-21 00:10:18,767 - DEBUG: cmd: restic -r s3:<my_bucket> forget --group-by paths --keep-daily 14 --keep-weekly 3 --keep-monthly 1 --keep-yearly 0
2025-08-21 00:10:21,169 - ERROR: ---------- stdout ----------
2025-08-21 00:10:21,169 - ERROR: repo already locked, waiting up to 0s for the lock
2025-08-21 00:10:21,169 - ERROR: ----------------------------
2025-08-21 00:10:21,169 - ERROR: ---------- stderr ----------
2025-08-21 00:10:21,169 - ERROR: unable to create lock in backend: repository is already locked by PID 18011 on 639ac14aba17 by root (UID 0, GID 0)
2025-08-21 00:10:21,169 - ERROR: lock was created at 2025-08-19 12:11:28 (35h58m53.058891396s ago)
2025-08-21 00:10:21,169 - ERROR: storage ID dad4a709
2025-08-21 00:10:21,169 - ERROR: the `unlock` command can be used to remove stale locks
2025-08-21 00:10:21,170 - ERROR: ----------------------------
2025-08-21 00:10:21,170 - DEBUG: returncode 11
2025-08-21 00:10:21,170 - INFO: Prune stale data freeing storage space
2025-08-21 00:10:21,170 - DEBUG: cmd: restic -r s3:<my_bucket> prune
2025-08-21 00:10:23,798 - ERROR: ---------- stdout ----------
2025-08-21 00:10:23,798 - ERROR: repo already locked, waiting up to 0s for the lock
2025-08-21 00:10:23,798 - ERROR: ----------------------------
2025-08-21 00:10:23,799 - ERROR: ---------- stderr ----------
2025-08-21 00:10:23,799 - ERROR: unable to create lock in backend: repository is already locked by PID 18011 on 639ac14aba17 by root (UID 0, GID 0)
2025-08-21 00:10:23,799 - ERROR: lock was created at 2025-08-19 12:11:28 (35h58m55.687212606s ago)
2025-08-21 00:10:23,799 - ERROR: storage ID dad4a709
2025-08-21 00:10:23,799 - ERROR: the `unlock` command can be used to remove stale locks
2025-08-21 00:10:23,799 - ERROR: ----------------------------
2025-08-21 00:10:23,799 - DEBUG: returncode 11
2025-08-21 00:10:23,799 - ERROR: Cleanup exit code: 11

since the hostname in the lock file is the same as the running container (not the backup runner) and the timestamp of the Lock file does not correspond the backup schedule, I've started to suspect the HealthCheck .

The rcb status calls the restic cat config command, which by default does create a lock on the repository... There is an --no-lock option, which disables the lock..

I've updated my HealthChecks to use restic cat config --no-lock command instead of rcb status

    healthcheck:
      test: [ 'CMD', 'sh', '-c', 'restic cat config --no-lock >/dev/null 2>&1' ]
      interval: 15m
      timeout: 60s
      start_period: 1m
      start_interval: 30s

It would be nice to receive any thoughts/suggestions on this topic 🙃

Cheers!

[lawndoc]

I've run into this before. The solution for me was to make the health checks less frequent. The locks appeared because the healthcheck is timing out (60s) and cancelling the restic operation before it is finished.

I've had no further issues with the following configuration:

    healthcheck:
      test: ["CMD", "rcb", "status"]
      interval: 30m
      timeout: 5m
      retries: 3
      start_period: 1m