[akamai.siem]: Pagination logic causes excessive API requests #16186
Description
Integration Name
Akamai [akamai]
Dataset Name
akamai.siem
Integration Version
3.0.0
Agent Version
No response
Agent Output Type
elasticsearch
Elasticsearch Version
No response
OS Version and Architecture
No response
Software/API Version
Akamai SIEM API v1
Error Message
No response
Event Original
No response
What did you do?
Configured Akamai SIEM integration to collect security events from the API.
What did you see?
The integration uses incorrect logic to determine when to continue paginating through API responses. The CEL program at packages/akamai/data_stream/siem/agent/stream/cel.yml.hbs:112-117 checks whether the last event has a non-empty offset field to set want_more:
"want_more": (lines.size() > 0) ?
lines[lines.size() - 1].decode_json().as(lastEvent,
has(lastEvent.offset) && lastEvent.offset != ""
)
:
false,
This causes an issue where pagination may continue unnecessarily when no more data is available, causing wasted API calls.
What did you expect to see?
According to the Akamai SIEM API documentation, the response includes a metadata object on the last line with a limit field that "appears if the size limit was reached during data fetch." This is the correct indicator for whether more data is available.
The pagination logic should check for the presence of the limit field instead:
"want_more": (lines.size() > 0) ?
lines[lines.size() - 1].decode_json().as(lastEvent,
has(lastEvent.limit)
)
:
false,
Anything else?
The offset field is present on every response metadata line regardless of whether more data is available, making it an unreliable indicator for pagination. Only the limit field's presence definitively signals that the API has more data to fetch.
https://techdocs.akamai.com/siem-integration/reference/get-configid