Add LLM-Friendly Documentation with llms.txt Generation

[sundarshankar89]

Summary

Integrated the @signalwire/docusaurus-plugin-llms-txt plugin (v2.0.0-alpha.7) to automatically generate AI-accessible documentation during the build process. This enables AI assistants and LLM-powered tools to easily consume DQX documentation through standardized llms.txt format with hierarchical organization.

---

Key Changes

Added @signalwire/docusaurus-plugin-llms-txt@2.0.0-alpha.7 with comprehensive v2.0 configuration:

Section Structure:

1. Home           → DQX homepage
2. User Guide     → /docs/guide/** (all feature guides)
3. Reference      → /docs/reference/** (API, CLI, profiler, benchmarks)
4. Development    → /docs/dev/** (contributing, authoring)
5. Demos          → /docs/demos (example implementations)

Access Endpoints

Once deployed to GitHub Pages:

• https://databrickslabs.github.io/dqx/llms.txt - Main index
• https://databrickslabs.github.io/dqx/llms-full.txt - Complete export
• https://databrickslabs.github.io/dqx/docs/**/*.md - Individual pages

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[github-actions]

✅ 602/602 passed, 36 skipped, 6h18m2s total

_{Running from acceptance #4030}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.81%. Comparing base (1b0c113) to head (9df255f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1029      +/-   ##
==========================================
+ Coverage   91.54%   91.81%   +0.26%     
==========================================
  Files          98       98              
  Lines        8945     8945              
==========================================
+ Hits         8189     8213      +24     
+ Misses        756      732      -24     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mwojtyczka]

Good idea to add llms.txt support — it makes DQX docs easily consumable by AI tooling. A couple of configuration concerns and one dependency issue worth addressing.

yarn.lock — duplicate package resolutions: The lockfile now has two separate entries that resolve to different versions of the same semver range:

• remark-gfm@^4 → 4.0.1 (new) and remark-gfm@^4.0.0 → 4.0.0 (existing)
• unist-util-visit@^5 → 5.1.0 (new) and unist-util-visit@^5.0.0 → 5.0.0 (existing)

Both ranges (^4 and ^4.0.0) are semantically identical, so yarn should have resolved them to the same version. Running yarn dedupe remark-gfm and yarn dedupe unist-util-visit after adding the dependency would collapse these to a single version and reduce bundle size.

[sundarshankar89]

Good idea to add llms.txt support — it makes DQX docs easily consumable by AI tooling. A couple of configuration concerns and one dependency issue worth addressing.

yarn.lock — duplicate package resolutions: The lockfile now has two separate entries that resolve to different versions of the same semver range:

• remark-gfm@^4 → 4.0.1 (new) and remark-gfm@^4.0.0 → 4.0.0 (existing)
• unist-util-visit@^5 → 5.1.0 (new) and unist-util-visit@^5.0.0 → 5.0.0 (existing)

Both ranges (^4 and ^4.0.0) are semantically identical, so yarn should have resolved them to the same version. Running yarn dedupe remark-gfm and yarn dedupe unist-util-visit after adding the dependency would collapse these to a single version and reduce bundle size.

addressed these review comments, we are usign alpha release now since that has the required capabities for generated nested md for every routes.

[sundarshankar89]

@mwojtyczka is there a permission issue which is causing the CI to fail?

[mwojtyczka]

LGTM