[AWS Firehose]: Preserve Original Event Data in WAF Firehose Integration #16188
Description
Integration Name
Amazon Data Firehose [awsfirehose]
Dataset Name
No response
Integration Version
1.9.0
Agent Version
8.19.6
OS Version and Architecture
Linux
User Goal
The user is using the Elastic Integration for AWS Firehose to ingest logs from AWS WAF. They have encountered an issue where the default ingest pipeline, logs-aws.waf-, drops important fields from the logs and does not provide an option to retain the original event data. This behavior leads to data loss and requires manual intervention to modify the pipeline after every upgrade.
Existing Features
The user has implemented a workaround by modifying the default ingest pipeline to preserve the dropped fields. However, this workaround needs to be reapplied after every integration update, which is not efficient.
What did you see?
Fields are being dropped, causing data loss.
Anything else?
The user requests that the WAF Firehose integration should include an option to preserve the original event data. This feature would allow users to retain all fields from the source logs, even if they are not explicitly processed in the ingest pipeline. This would prevent data loss and eliminate the need for manual pipeline modifications.