It's great that the plugin can decompress on the fly gzip files using the gzip_pattern https://www.elastic.co/docs/reference/logstash/plugins/plugins-inputs-s3#plugins-inputs-s3-gzip_pattern
It would be nice to have a similar feature for zstd compression that is being used by AWS in some use cases. We are using Lambda logs shipping to S3 as described in this tutorial https://docs.aws.amazon.com/lambda/latest/dg/logging-with-s3.html (through a loggroup with DELIVERY log_class) and the logs arrive compressed with zstd https://facebook.github.io/zstd/ On a linux host we can use unzstd from https://packages.debian.org/bookworm/zstd package to unzip it.
It's great that the plugin can decompress on the fly gzip files using the
gzip_patternhttps://www.elastic.co/docs/reference/logstash/plugins/plugins-inputs-s3#plugins-inputs-s3-gzip_patternIt would be nice to have a similar feature for zstd compression that is being used by AWS in some use cases. We are using Lambda logs shipping to S3 as described in this tutorial https://docs.aws.amazon.com/lambda/latest/dg/logging-with-s3.html (through a loggroup with DELIVERY log_class) and the logs arrive compressed with zstd https://facebook.github.io/zstd/ On a linux host we can use
unzstdfrom https://packages.debian.org/bookworm/zstd package to unzip it.