createZstdCompress throw error if set constants.ZSTD_c_nbWorkers

[wszgrcy]

Version

22.17.0

Platform

Microsoft Windows NT 10.0.19045.0 x64

Subsystem

No response

What steps will reproduce the bug?

use zstd compress file

import fs from 'fs';
import path from 'path';
import { Readable } from 'stream';
import { pipeline } from 'stream/promises';
import { constants, createZstdCompress } from 'zlib';
(async () => {
  function createBlankStream(count: number) {
    return new (class extends Readable {
      readCount = 0;
      constructor() {
        super({ highWaterMark: 8 * 1024 * 1024 });
      }
      override _read(size: number) {
        this.readCount += size;
        const needSize = this.readCount > count ? size - (this.readCount - count) : size;
        this.push(Buffer.alloc(needSize, 0));
        if (needSize !== size) {
          this.push(null);
        }
      }
    })();
  }
  let zstdC = createZstdCompress({
    params: {
      [constants.ZSTD_c_compressionLevel]: 1,
      [constants.ZSTD_c_nbWorkers]: 1,
    },
  });
//If no exceptions are reported, you can try increasing the buffer
  await pipeline(createBlankStream(8 * 1024 * 1024+1), zstdC, fs.createWriteStream(path.join(process.cwd(), '11.tar')));
})();

How often does it reproduce? Is there a required condition?

This bug is 100% reproducible.

What is the expected behavior? Why is that the expected behavior?

work

What do you see instead?

Error [ERR_STREAM_PUSH_AFTER_EOF]: stream.push() after EOF
    at readableAddChunkPushByteMode (node:internal/streams/readable:495:28)
    at Readable.push (node:internal/streams/readable:392:5)
    at ZstdCompress.processCallback (node:zlib:536:32)
Emitted 'error' event on ZstdCompress instance at:
    at emitErrorNT (node:internal/streams/destroy:170:8)
    at emitErrorCloseNT (node:internal/streams/destroy:129:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:90:21) {
  code: 'ERR_STREAM_PUSH_AFTER_EOF'
}

Additional information

No response

[bricss]

If you'll pass count into stream constructor highWaterMark it will solve the issue, e.g.: super({ highWaterMark: count });

[wszgrcy]

If you'll pass count into stream constructor highWaterMark it will solve the issue, e.g.: super({ highWaterMark: count });

createBlankStream is only an auxiliary function for simulating file reading

[bricss]

There is an actual issue within your code example that leads to an error.

The ERR_STREAM_PUSH_AFTER_EOF error happens when a Readable stream pushes data after it has already been ended with push(null).

You can try with real file instead of simulation and there will be no error.

[wszgrcy]

There is an actual issue within your code example that leads to an error.

The ERR_STREAM_PUSH_AFTER_EOF error happens when a Readable stream pushes data after it has already been ended with push(null).

You can try with real file instead of simulation and there will be no error.

I am using real files, but to simplify the feedback issues, I created this code example
In the code above, there will be three pushes of 8388608, 1, and null
So I'm not quite sure what you mean.

[bricss]

This code works like a charm:

import fs from 'node:fs';
import path from 'node:path';
import { pipeline } from 'node:stream/promises';
import zlib from 'node:zlib';

await pipeline(
  fs.createReadStream(path.join(process.cwd(), 'woot.md')),
  zlib.createZstdCompress({
      params: {
          [zlib.constants.ZSTD_c_compressionLevel]: 1,
          [zlib.constants.ZSTD_c_nbWorkers]: 1,
      },
  }),
  fs.createWriteStream(path.join(process.cwd(), 'waat.xz')),
);

The issue lies in highWaterMark opt in stream constructor options.

[wszgrcy]

This code works like a charm:

import fs from 'node:fs';
import path from 'node:path';
import { pipeline } from 'node:stream/promises';
import zlib from 'node:zlib';

await pipeline(
fs.createReadStream(path.join(process.cwd(), 'woot.md')),
zlib.createZstdCompress({
params: {
[zlib.constants.ZSTD_c_compressionLevel]: 1,
[zlib.constants.ZSTD_c_nbWorkers]: 1,
},
}),
fs.createWriteStream(path.join(process.cwd(), 'waat.xz')),
);
The issue lies in highWaterMark opt in stream constructor options.

The file size is too small. Penalties will only be imposed when the size is large enough. That's why I use 8*1024*1024 instead of 1
Since I've come to raise an issue, I have sufficient confidence to prove that there is indeed a problem

Additionally, I have clearly stated in the title that setting ZSTD_c_nbWorkers will result in an error, while leaving it unset will function normally,
That is, if you comment out ZSTD_c_nbWorkers in the example I mentioned, everything will work as expected
I think I've made myself clear .......

[bricss]

How large enough file we are talking about?
I tried to use real file in size of almost 1 GB, and still no errors observed 🔭