Graceful shutdown of Docker container not respected

[yandodov]

As I posted on Stack Overflow:

My project uses nodemon inside Docker. It runs perfectly, but when I stop the process from my terminal with Ctrl+C, the container doesn't get turned off immediately. Instead, I have to wait for exactly 10 seconds before Docker finally kills it.

My guess is that nodemon fails to trap the SIGINT signal and doesn't act on it.

• Versions: node@v20.9.0, darwin@22.6.0
• nodemon -v: 3.0.2
• Operating system/terminal environment (powershell, gitshell, etc): Docker on macOS Ventura 13.6.3 (22G436)
• Using Docker? What image: node:18-bullseye-slim
• Command you ran: nodemon src/script.js

Expected behaviour

Docker container should immediately exit.

Actual behaviour

Docker container exits after a 10 second delay.

Steps to reproduce

I've made a GitHub repo with the example. Reproduction steps are in the README.

---

If applicable, please append the --dump flag on your command and include the output here ensuring to remove any sensitive/personal details or tokens.

$ npm run dev

> dev
> docker compose up --build --no-log-prefix

[+] Building 2.5s (9/9) FINISHED                                                  docker:desktop-linux
 => [node internal] load build definition from Dockerfile                                         0.0s
 => => transferring dockerfile: 136B                                                              0.0s
 => [node internal] load .dockerignore                                                            0.0s
 => => transferring context: 2B                                                                   0.0s
 => [node internal] load metadata for docker.io/library/node:18-bullseye-slim                     0.0s
 => [node internal] load build context                                                            0.0s
 => => transferring context: 206B                                                                 0.0s
 => [node 1/4] FROM docker.io/library/node:18-bullseye-slim                                       0.0s
 => CACHED [node 2/4] WORKDIR /app                                                                0.0s
 => [node 3/4] COPY package.json .                                                                0.0s
 => [node 4/4] RUN npm i                                                                          2.3s
 => [node] exporting to image                                                                     0.1s
 => => exporting layers                                                                           0.1s
 => => writing image sha256:f10b8543d5d6127ed57669be50b428a99bf75dd3babdc1179589c7d25582f4d1      0.0s
 => => naming to docker.io/library/test-node                                                      0.0s
[+] Running 1/1
 ✔ Container test-node-1  Recreated                                                               0.1s 
Attaching to test-node-1

> start
> nodemon src/script.js --dump

[nodemon] 3.0.2
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
--------------
node: v18.19.0
nodemon: 3.0.2
command: /usr/local/bin/node /app/node_modules/.bin/nodemon src/script.js --dump
cwd: /app
OS: linux x64
--------------
{
  run: false,
  system: { cwd: '/app' },
  required: false,
  dirs: [ '/app' ],
  timeout: 1000,
  options: {
    dump: true,
    ignore: [
      '**/.git/**',
      '**/.nyc_output/**',
      '**/.sass-cache/**',
      '**/bower_components/**',
      '**/coverage/**',
      '**/node_modules/**',
      re: /.*.*\/\.git\/.*.*|.*.*\/\.nyc_output\/.*.*|.*.*\/\.sass\-cache\/.*.*|.*.*\/bower_components\/.*.*|.*.*\/coverage\/.*.*|.*.*\/node_modules\/.*.*/
    ],
    watch: [ '*.*', re: /.*\..*/ ],
    monitor: [
      '*.*',
      '!**/.git/**',
      '!**/.nyc_output/**',
      '!**/.sass-cache/**',
      '!**/bower_components/**',
      '!**/coverage/**',
      '!**/node_modules/**'
    ],
    ignoreRoot: [
      '**/.git/**',
      '**/.nyc_output/**',
      '**/.sass-cache/**',
      '**/bower_components/**',
      '**/coverage/**',
      '**/node_modules/**'
    ],
    restartable: 'rs',
    colours: true,
    execMap: { py: 'python', rb: 'ruby', ts: 'ts-node' },
    stdin: true,
    runOnChangeOnly: false,
    verbose: false,
    signal: 'SIGUSR2',
    stdout: true,
    watchOptions: {},
    execOptions: {
      script: 'src/script.js',
      exec: 'node',
      args: [],
      scriptPosition: 0,
      nodeArgs: undefined,
      execArgs: [],
      ext: 'js,mjs,cjs,json',
      env: {}
    }
  },
  load: [Function (anonymous)],
  reset: [Function: reset],
  lastStarted: 0,
  loaded: [],
  watchInterval: null,
  signal: 'SIGUSR2',
  command: {
    raw: { executable: 'node', args: [ 'src/script.js' ] },
    string: 'node src/script.js'
  }
}
--------------
test-node-1 exited with code 0

[github-actions]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[remy]

My guess is that the sub process isn't shutting down when you send the ctrl+c (which is received by nodemon) so nodemon has a grace period (of 10 seconds polling the sub process) after which it forcibly exits.

Wonder if running your docker instance with env value DEBUG=nodemon,nodemon:* and it might give us insights into what's actually hanging.

[yandodov]

@remy here's the debug output you suggested:

$ npm run dev

> dev
> docker compose up --build --no-log-prefix

[+] Building 0.1s (9/9) FINISHED                                                     docker:desktop-linux
 => [node internal] load .dockerignore                                                               0.0s
 => => transferring context: 2B                                                                      0.0s
 => [node internal] load build definition from Dockerfile                                            0.0s
 => => transferring dockerfile: 136B                                                                 0.0s
 => [node internal] load metadata for docker.io/library/node:18-bullseye-slim                        0.0s
 => [node 1/4] FROM docker.io/library/node:18-bullseye-slim                                          0.0s
 => [node internal] load build context                                                               0.0s
 => => transferring context: 34B                                                                     0.0s
 => CACHED [node 2/4] WORKDIR /app                                                                   0.0s
 => CACHED [node 3/4] COPY package.json .                                                            0.0s
 => CACHED [node 4/4] RUN npm i                                                                      0.0s
 => [node] exporting to image                                                                        0.0s
 => => exporting layers                                                                              0.0s
 => => writing image sha256:deb343f642d0dcd9868a5438fa728fa2cd0ea0ff09921c49b089b51be0855b05         0.0s
 => => naming to docker.io/library/mcve-nodemon-docker-exit-node                                     0.0s
[+] Running 1/0
 ✔ Container mcve-nodemon-docker-exit-node-1  Created                                                0.0s 
Attaching to mcve-nodemon-docker-exit-node-1

> start
> DEBUG=nodemon,nodemon:* nodemon src/script.js

2024-01-04T09:26:24.937Z nodemon bus new listener: reset (0)
2024-01-04T09:26:24.938Z nodemon bus new listener: reset (0)
2024-01-04T09:26:24.942Z nodemon bus new listener: quit (0)
2024-01-04T09:26:24.942Z nodemon bus new listener: quit (0)
2024-01-04T09:26:24.942Z nodemon bus new listener: restart (0)
2024-01-04T09:26:24.942Z nodemon bus new listener: restart (0)
2024-01-04T09:26:24.943Z nodemon bus new listener: boot (0)
2024-01-04T09:26:24.943Z nodemon bus new listener: boot (0)
2024-01-04T09:26:24.944Z nodemon bus new listener: reset (2)
2024-01-04T09:26:24.945Z nodemon bus emit: boot
2024-01-04T09:26:24.946Z nodemon bus emit: reset
2024-01-04T09:26:24.946Z nodemon resetting watchers
2024-01-04T09:26:24.946Z nodemon reset
2024-01-04T09:26:24.964Z nodemon config: dirs [ '/app' ]
[nodemon] 3.0.2
[nodemon] to restart at any time, enter `rs`
2024-01-04T09:26:24.967Z nodemon bus new listener: error (0)
2024-01-04T09:26:24.968Z nodemon bus new listener: error (0)
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node src/script.js`
2024-01-04T09:26:24.977Z nodemon:run fork sh -c node src/script.js
2024-01-04T09:26:24.977Z nodemon bus new listener: exit (0)
2024-01-04T09:26:24.977Z nodemon bus new listener: exit (0)
2024-01-04T09:26:24.978Z nodemon:run start watch on: [ '*.*', re: /.*\..*/ ]
2024-01-04T09:26:24.978Z nodemon start watch on: /app
2024-01-04T09:26:24.979Z nodemon ignored [
  '**/.git/**',
  '**/.nyc_output/**',
  '**/.sass-cache/**',
  '**/bower_components/**',
  '**/coverage/**',
  '**/node_modules/**',
  re: /.*.*\/\.git\/.*.*|.*.*\/\.nyc_output\/.*.*|.*.*\/\.sass\-cache\/.*.*|.*.*\/bower_components\/.*.*|.*.*\/coverage\/.*.*|.*.*\/node_modules\/.*.*/
]
2024-01-04T09:26:24.992Z nodemon:watch chokidar watching: /app/package-lock.json
2024-01-04T09:26:24.992Z nodemon:watch chokidar watching: /app/package.json
2024-01-04T09:26:24.994Z nodemon:watch chokidar watching: /app/src/script.js
2024-01-04T09:26:24.995Z nodemon watch is complete
Hello world!
[nodemon] clean exit - waiting for changes before restart
2024-01-04T09:26:25.011Z nodemon:run bus.emit(exit) via SIGUSR2
2024-01-04T09:26:25.011Z nodemon bus emit: exit
^CGracefully stopping... (press Ctrl+C again to force)
Aborting on container exit...
[+] Stopping 1/1
 ✔ Container mcve-nodemon-docker-exit-node-1  Stopped                                               10.1s 
canceled

I've updated the reproduction repo to include the DEBUG variable, so you can try it on your machine as well.

I don't know if this output is useful because nothing actually gets logged after pressing Ctrl + C (the line beginning with ^C). That's when the delay occurs.

[github-actions]

This issue has been automatically marked as idle and stale because it hasn't had any recent activity. It will be automtically closed if no further activity occurs. If you think this is wrong, or the problem still persists, just pop a reply in the comments and @remy will (try!) to follow up.
Thank you for contributing <3

[yandodov]

@remy I don't think this issue should be closed. I can still reproduce this, even in both Rancher Desktop:

[+] Building 6.8s (9/9) FINISHED                          docker:default
 => [node internal] load build definition from Dockerfile           0.0s
 => => transferring dockerfile: 136B                                0.0s
 => [node internal] load .dockerignore                              0.0s
 => => transferring context: 2B                                     0.0s
 => [node internal] load metadata for docker.io/library/node:18-bu  3.2s
 => [node 1/4] FROM docker.io/library/node:18-bullseye-slim@sha256  0.0s
 => [node internal] load build context                              0.0s
 => => transferring context: 223B                                   0.0s
 => CACHED [node 2/4] WORKDIR /app                                  0.0s
 => [node 3/4] COPY package.json .                                  0.0s
 => [node 4/4] RUN npm i                                            3.4s
 => [node] exporting to image                                       0.1s
 => => exporting layers                                             0.1s
 => => writing image sha256:de8b0b784c881e29f92bab03ea83ca6de2f317  0.0s
 => => naming to docker.io/library/mcve-nodemon-docker-exit-node    0.0s
[+] Running 2/1
 ✔ Network mcve-nodemon-docker-exit_default   Created               0.3s 
 ✔ Container mcve-nodemon-docker-exit-node-1  Created               0.0s 
Attaching to node-1

> start
> DEBUG=nodemon,nodemon:* nodemon src/script.js

2024-01-25T08:34:35.886Z nodemon bus new listener: reset (0)
2024-01-25T08:34:35.887Z nodemon bus new listener: reset (0)
2024-01-25T08:34:35.891Z nodemon bus new listener: quit (0)
2024-01-25T08:34:35.891Z nodemon bus new listener: quit (0)
2024-01-25T08:34:35.891Z nodemon bus new listener: restart (0)
2024-01-25T08:34:35.892Z nodemon bus new listener: restart (0)
2024-01-25T08:34:35.892Z nodemon bus new listener: boot (0)
2024-01-25T08:34:35.892Z nodemon bus new listener: boot (0)
2024-01-25T08:34:35.894Z nodemon bus new listener: reset (2)
2024-01-25T08:34:35.896Z nodemon bus emit: boot
2024-01-25T08:34:35.896Z nodemon bus emit: reset
2024-01-25T08:34:35.896Z nodemon resetting watchers
2024-01-25T08:34:35.896Z nodemon reset
2024-01-25T08:34:35.914Z nodemon config: dirs [ '/app' ]
[nodemon] 3.0.3
[nodemon] to restart at any time, enter `rs`
2024-01-25T08:34:35.920Z nodemon bus new listener: error (0)
2024-01-25T08:34:35.920Z nodemon bus new listener: error (0)
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node src/script.js`
2024-01-25T08:34:35.929Z nodemon:run fork sh -c node src/script.js
2024-01-25T08:34:35.929Z nodemon bus new listener: exit (0)
2024-01-25T08:34:35.930Z nodemon bus new listener: exit (0)
2024-01-25T08:34:35.930Z nodemon:run start watch on: [ '*.*', re: /.*\..*/ ]
2024-01-25T08:34:35.931Z nodemon start watch on: /app
2024-01-25T08:34:35.931Z nodemon ignored [
  '**/.git/**',
  '**/.nyc_output/**',
  '**/.sass-cache/**',
  '**/bower_components/**',
  '**/coverage/**',
  '**/node_modules/**',
  re: /.*.*\/\.git\/.*.*|.*.*\/\.nyc_output\/.*.*|.*.*\/\.sass\-cache\/.*.*|.*.*\/bower_components\/.*.*|.*.*\/coverage\/.*.*|.*.*\/node_modules\/.*.*/
]
2024-01-25T08:34:35.947Z nodemon:watch chokidar watching: /app/package-lock.json
2024-01-25T08:34:35.947Z nodemon:watch chokidar watching: /app/package.json
2024-01-25T08:34:35.955Z nodemon:watch chokidar watching: /app/src/script.js
2024-01-25T08:34:35.955Z nodemon watch is complete
Hello world!
2024-01-25T08:34:35.974Z nodemon:run bus.emit(exit) via SIGUSR2
2024-01-25T08:34:35.974Z nodemon bus emit: exit
[nodemon] clean exit - waiting for changes before restart
^CGracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/1
 ✔ Container mcve-nodemon-docker-exit-node-1  Stopped              10.5s 
node-1 exited with code 0
canceled

…and OrbStack:

[+] Building 0.6s (9/9) FINISHED                        docker:orbstack
 => [node internal] load build definition from Dockerfile          0.0s
 => => transferring dockerfile: 136B                               0.0s
 => [node internal] load .dockerignore                             0.0s
 => => transferring context: 2B                                    0.0s
 => [node internal] load metadata for docker.io/library/node:18-b  0.6s
 => [node 1/4] FROM docker.io/library/node:18-bullseye-slim@sha25  0.0s
 => [node internal] load build context                             0.0s
 => => transferring context: 34B                                   0.0s
 => CACHED [node 2/4] WORKDIR /app                                 0.0s
 => CACHED [node 3/4] COPY package.json .                          0.0s
 => CACHED [node 4/4] RUN npm i                                    0.0s
 => [node] exporting to image                                      0.0s
 => => exporting layers                                            0.0s
 => => writing image sha256:d43b6b2309d5b15cf112821bf56f77dea4337  0.0s
 => => naming to docker.io/library/mcve-nodemon-docker-exit-node   0.0s
[+] Running 1/0
 ✔ Container mcve-nodemon-docker-exit-node-1  Created              0.0s 
Attaching to node-1

> start
> DEBUG=nodemon,nodemon:* nodemon src/script.js

2024-01-25T08:42:00.234Z nodemon bus new listener: reset (0)
2024-01-25T08:42:00.235Z nodemon bus new listener: reset (0)
2024-01-25T08:42:00.239Z nodemon bus new listener: quit (0)
2024-01-25T08:42:00.240Z nodemon bus new listener: quit (0)
2024-01-25T08:42:00.240Z nodemon bus new listener: restart (0)
2024-01-25T08:42:00.240Z nodemon bus new listener: restart (0)
2024-01-25T08:42:00.240Z nodemon bus new listener: boot (0)
2024-01-25T08:42:00.240Z nodemon bus new listener: boot (0)
2024-01-25T08:42:00.241Z nodemon bus new listener: reset (2)
2024-01-25T08:42:00.242Z nodemon bus emit: boot
2024-01-25T08:42:00.242Z nodemon bus emit: reset
2024-01-25T08:42:00.242Z nodemon resetting watchers
2024-01-25T08:42:00.242Z nodemon reset
2024-01-25T08:42:00.261Z nodemon config: dirs [ '/app' ]
[nodemon] 3.0.3
[nodemon] to restart at any time, enter `rs`
2024-01-25T08:42:00.264Z nodemon bus new listener: error (0)
2024-01-25T08:42:00.265Z nodemon bus new listener: error (0)
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node src/script.js`
2024-01-25T08:42:00.272Z nodemon:run fork sh -c node src/script.js
2024-01-25T08:42:00.272Z nodemon bus new listener: exit (0)
2024-01-25T08:42:00.272Z nodemon bus new listener: exit (0)
2024-01-25T08:42:00.272Z nodemon:run start watch on: [ '*.*', re: /.*\..*/ ]
2024-01-25T08:42:00.273Z nodemon start watch on: /app
2024-01-25T08:42:00.273Z nodemon ignored [
  '**/.git/**',
  '**/.nyc_output/**',
  '**/.sass-cache/**',
  '**/bower_components/**',
  '**/coverage/**',
  '**/node_modules/**',
  re: /.*.*\/\.git\/.*.*|.*.*\/\.nyc_output\/.*.*|.*.*\/\.sass\-cache\/.*.*|.*.*\/bower_components\/.*.*|.*.*\/coverage\/.*.*|.*.*\/node_modules\/.*.*/
]
2024-01-25T08:42:00.286Z nodemon:watch chokidar watching: /app/package-lock.json
2024-01-25T08:42:00.286Z nodemon:watch chokidar watching: /app/package.json
2024-01-25T08:42:00.291Z nodemon:watch chokidar watching: /app/src/script.js
2024-01-25T08:42:00.292Z nodemon watch is complete
Hello world!
2024-01-25T08:42:00.304Z nodemon:run bus.emit(exit) via SIGUSR2
2024-01-25T08:42:00.304Z nodemon bus emit: exit
[nodemon] clean exit - waiting for changes before restart
^CGracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/1
 ✔ Container mcve-nodemon-docker-exit-node-1  Stopped             10.1s 
node-1 exited with code 0
canceled

[lyrixderaven]

I'm seeing the same behavior in a standard docker compose setup at the moment. I've also been able to reproduce the issue with @hdodov's mcve repository and tried adapting it to run a minimal express app that also listens for SIGINT and SIGTERM signals during runtime, neither of which seem to reach the application.

[arpanetus]

I'm seeing the same behavior in a standard docker compose setup at the moment. I've also been able to reproduce the issue with @hdodov's mcve repository and tried adapting it to run a minimal express app that also listens for SIGINT and SIGTERM signals during runtime, neither of which seem to reach the application.

Hey @lyrixderaven, I had the same issue, but not with nodemon. The default stop_grace_period (that could be set in docker-compose file) is 10s, you can put any number you'd like to. In my case 10m was enough, it worked well. You can give it a try, if it's still relevant.

[akselerando]

I am having the same issue in my application where the child process listens to SIGINT and SIGTERM, but none is received until it is killed after 10 seconds. Any updates on this @remy @lyrixderaven ?

[remy]

I've mark as not stale, but I've not had a chance to look at this at all (partly work, partly christmas family season). Thank you @hdodov for the repo to replicate, that will help a lot (if anyone else wants to take a look in the meantime, I'd be grateful).

[nathan-muir]

@remy I've had a quick dig, and I think the issue is that the child process is killed twice in succession:

First with child.kill(signal) then 6 lines later with exec(kill -${sig} ${child.pid})

nodemon/lib/monitor/run.js

Lines 428 to 434 in d147fb6

	child.kill(signal);
	pids.sort().forEach((pid) => exec(`kill -${sig} ${pid}`, noop));
	waitForSubProcesses(child.pid, () => {
	// finally kill the main user process
	exec(`kill -${sig} ${child.pid}`, callback);

---

This matches the experience i was having with close-with-grace where my node process would receive two SIGUSR2's about 4ms apart