Feature Request: Include Document Identifiers in CloudWatch Log Stream Names for Better Observability

[mmiakashs]

Currently, when troubleshooting document processing failures in the GenAI IDP Accelerator, it's very difficult to identify which CloudWatch logs correspond to a specific document. CloudWatch log groups are organized only by Lambda function names, requiring users to search through multiple log groups and use complex queries to find logs for a specific document.

Current Behavior

• Log groups are named like: /aws/lambda/{StackName}-{FunctionName}-{RandomSuffix}
• Document IDs only appear within log messages, not in log group/stream names
• Users must manually search across multiple Lambda function logs

[rstrahan]

Hi @mmiakashs - Sadly Lambda functions cannot control their own CloudWatch log streams. Claude clarifies:

• Lambda Runtime Control: The log stream name is determined by the Lambda runtime, not your function code
• Standard Format: Lambda creates log streams with the pattern: YYYY/MM/DD/[$LATEST]<request-id>
• No Override Mechanism: The Python logging handler is pre-configured by Lambda and cannot be easily overridden to use custom log stream names

Each function does log the document ID in it's incoming event, which allows you to locate the right log stream with a search.

We could, potentially, create a custom log handler that also logs to its own log streams, but that add code complexity and cost as a tradeoff.

A better solution might be to enable AWS X-Ray integration for tracing a document across the whole pipeline..

The Agentic Troubleshooting agent we are working on will also help, maybe in conjunction with XRay tracing.. I'll ask Mansoor to investigate this..