Docs idea: logging 16 RAG failure modes with Langfuse (WFGY ProblemMap) #2565
Description
Hi, and thanks for Langfuse – it has become a very natural choice for tracing and debugging LLM apps.
I maintain an MIT-licensed open-source project called WFGY (~1.5k stars). Part of it is the WFGY ProblemMap, a set of 16 RAG / LLM pipeline failure modes used as a checklist when diagnosing systems (retriever issues, vector store problems, routing, hallucinations, evaluation gaps, etc.):
- ProblemMap overview:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md
The ProblemMap is referenced by external projects such as Harvard MIMS Lab’s ToolUniverse and the QCRI Multimodal RAG Survey.
Proposal
Add a docs page or cookbook entry in langfuse-docs that shows:
-
How to instrument a RAG pipeline so that each of the 16 failure modes leaves a clear trace in Langfuse
(e.g. tags, observations, spans, metrics). -
A compact “pattern → query” mapping, for example:
- Problem No. 3 – chunking / context fragmentation → how to log chunk stats and visualise them in Langfuse.
- Problem No. 5 – retriever mis-prioritisation → how to inspect retrieved documents vs. gold labels.
- Problem No. 11 – evaluation blind spots → how to log evaluation results and spot skew.
-
A small end-to-end example (Python) that sends structured traces to Langfuse and demonstrates how to use the UI to locate a few of the failure modes.
This would be a docs-only contribution, focused on examples and best practices.
Why this is a good fit for Langfuse
Langfuse already positions itself as the “observability layer” for LLM applications. The 16-problem map provides a widely reusable vocabulary for what we are observing when debugging RAG systems.
A guide that says “here is how to see all 16 failure modes in Langfuse” could:
- make it easier for teams to justify adopting Langfuse as their main tracing tool,
- reduce guesswork when instrumenting RAG apps,
- and link Langfuse into an emerging shared language for RAG debugging.
If this sounds useful, I would be happy to help prepare an initial cookbook PR.