Add graceful degradation for large SQL queries by kkm-horikawa · Pull Request #2291 · django-commons/django-debug-toolbar

kkm-horikawa · 2026-01-09T13:37:13Z

Description

Add graceful degradation for SQL queries that exceed sqlparse's token limits, preventing crashes in the SQL panel.

Problem:
When SQL queries contain thousands of parameters (e.g., UUIDs in IN clauses), the SQL panel crashes with SQLParseError (sqlparse >= 0.5.5).

Solution:
Catch SQLParseError and retry formatting with grouping disabled. This approach:

Handles the crash automatically without requiring configuration
Eliminates the need for a new setting like SQL_PRETTIFY_MAX_LENGTH

Out of scope: The freezing behavior with older sqlparse versions (< 0.5.5) is a pre-existing issue and not addressed in this PR.

Before (debug-toolbar 5.2 + sqlparse 0.5.3) - page freezes (out of scope):

Before (debug-toolbar 6.1 + sqlparse 0.5.5) - SQLParseError:

After (this PR):

Reproduction project: https://github.com/kkm-horikawa/debug-toolbar-perf-issue

Fixes #2293

Checklist:

I have added the relevant tests for this change.
I have added an item to the Pending section of docs/changes.rst.

tim-schilling · 2026-01-13T21:42:33Z

Let's split the error handling of SQLParseError out of this PR, into its own.

kkm-horikawa · 2026-01-14T05:28:14Z

Thanks for the feedback! I've removed the SQLParseError handling from this PR.

This PR now focuses solely on the SQL_PRETTIFY_MAX_LENGTH setting (proactive length check).

I'll create a separate PR for SQLParseError handling after this one is merged. Does this approach work for you?

- Add SQL_PRETTIFY_MAX_LENGTH setting (default 50000) to skip formatting for long queries - Catch SQLParseError from sqlparse >= 0.5.5 when MAX_GROUPING_TOKENS is exceeded - Display informative message with SQL preview when formatting is skipped - Add tests for the new behavior This prevents the SQL panel from freezing or crashing when queries contain large IN clauses with thousands of parameters (e.g., UUIDs). Fixes django-commons#1402

The preview was being escaped twice - once when assigned to the preview variable and again when inserted into the HTML string.

for more information, see https://pre-commit.ci

Split SQLParseError exception handling into a separate PR as requested. This PR now focuses solely on the SQL_PRETTIFY_MAX_LENGTH setting.

tim-schilling

Thank you for suggesting this. However, I think we have too many utility functions for rendering the SQL at this point. A change will need to review more of the whole process and identify why it's appropriate compared to other options. For example, I think there may be something achievable by modifying parse_sql and/or using SQLParse's FilterStack API. Let me know what you think please.

tim-schilling · 2026-01-14T19:30:07Z

debug_toolbar/panels/sql/utils.py

+    max_length = dt_settings.get_config()["SQL_PRETTIFY_MAX_LENGTH"]
+    if max_length and len(sql) > max_length:


I feel like this should be integrated into the sqlparse logic in get_filter_stack(). Have you looked into that at all to see if we can do something with FilterStack?

Done. I've integrated the length check directly into parse_sql instead.

tim-schilling · 2026-01-14T19:31:18Z

tests/panels/test_sql.py

+        """
+        Test that SQL formatting is skipped for queries exceeding the max length threshold.
+        """
+        from debug_toolbar.panels.sql.utils import reformat_sql


Please avoid importing things within functions.

Fixed. Moved the import to the top of the file.

tim-schilling · 2026-01-14T19:33:19Z

debug_toolbar/panels/sql/utils.py

+    if max_length and len(sql) > max_length:
+        skipped = _format_skipped_sql(
+            sql,
+            f"query length {len(sql):,} exceeds threshold {max_length:,}",


We may want to convert this all to a template and use render_to_string() so that it's easier to use the intcomma template filter. This would better handle internationalization of using commas versus periods.

I've removed the message display for now to keep the change minimal. If a notice is desired, I can add it using a template as you suggested - see my comment above for details.

tim-schilling · 2026-01-14T19:36:44Z

debug_toolbar/settings.py

        "django.utils.functional",
    ),
    "PRETTIFY_SQL": True,
+    "SQL_PRETTIFY_MAX_LENGTH": 50000,  # Skip formatting for SQL longer than this


Suggested change

"SQL_PRETTIFY_MAX_LENGTH": 50000, # Skip formatting for SQL longer than this

"PRETTIFY_SQL_MAX_LENGTH": 10000, # Skip formatting for SQL longer than this

Let's rename this to better match the other setting. Let's also use the default SQLParse max value from https://sqlparse.readthedocs.io/en/latest/api.html#security-and-performance-considerations.

kkm-horikawa · 2026-01-15T03:00:33Z

Thank you for the feedback. You're right that adding a separate utility function is unnecessary.

I've refactored to integrate the length check directly into parse_sql. I've also renamed the setting to PRETTIFY_SQL_MAX_LENGTH with a default of 10000 to match sqlparse's MAX_GROUPING_TOKENS.

The current implementation doesn't show a message when formatting is skipped - it just returns the escaped SQL. I made this choice to avoid adding new files or functions for this specific case, keeping the change minimal. However, if you'd prefer to have a notice displayed to users, I can add that using a template with render_to_string() and intcomma as you suggested. What do you think?

tim-schilling · 2026-01-19T22:47:21Z

debug_toolbar/panels/sql/utils.py

+    max_length = dt_settings.get_config()["PRETTIFY_SQL_MAX_LENGTH"]
+    if max_length and len(sql) > max_length:


I'm not sure this is right. This would limit sql queries to a string length of 10,000 characters. I was under the impression we were trying to improve performance when the query has more than 10,000 tokens. I believe that's not the same thing as a query with a string length of 10,000 characters.

And if we're removing the formatting in this case, it would make sense to avoid calling stack.enable_grouping() in that case.

I believe catching the error explicitly does what you're after:

@lru_cache(maxsize=128) def parse_sql(sql, *, simplify=False): stack = get_filter_stack(simplify=simplify) try: return "".join(stack.run(sql)) except SQLParseError: # The query either exceeds the number of tokens or depth of tokens. # Recreate the FilterStack and explicitly disable the grouping to avoid # those errors. stack = get_filter_stack(simplify=simplify) stack._grouping = False return "".join(stack.run(sql))

Thank you for the review!

Your suggested approach of catching SQLParseError is much better. It handles the issue only when it actually occurs,
and eliminates the need for a new configuration setting.

I'll update the implementation in this direction. Thank you for your patience and continued guidance through multiple
reviews!

Done. Thank you for the review!

I've updated the implementation to catch SQLParseError as you suggested. This approach also eliminates the need for
the PRETTIFY_SQL_MAX_LENGTH setting.

This PR focuses on handling the SQLParseError crash introduced in sqlparse >= 0.5.5. The freezing behavior with older
versions (< 0.5.5) is pre-existing and considered out of scope for this PR.

I've also updated the PR description to reflect these changes.

Thank you for your patience and continued guidance through multiple reviews!

for more information, see https://pre-commit.ci

tim-schilling · 2026-01-27T01:40:49Z

tests/panels/test_sql.py

+        from unittest.mock import patch
+
+        from sqlparse.exceptions import SQLParseError
+
+        from debug_toolbar.panels.sql.utils import parse_sql


@kkm-horikawa please review the LLM output more thoroughly in the future. For some reason, LLMs love to generate inline import statements. We try to avoid that practice in this project.

github-actions · 2026-01-27T01:45:39Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
debug_toolbar/panels/sql
utils.py
Project Total

_{This report was generated by python-coverage-comment-action}

tim-schilling · 2026-01-27T01:54:17Z

tests/panels/test_sql.py

+        call_count = 0
+        original_run = None
+
+        def mock_run(sql):


Give the side_effect attribute a look and then the call_count attribute. We can reuse some standard library tools to avoid rebuilding the wheel 😁

If you get stuck, let me know. I purposefully didn't write out the code, but I'm happy to include that if you're just wanting this PR to wrap up quickly.

tim-schilling

I'd like to see the test improved a bit more before we merge this. Thank you for continuing to iterate on this!

tim-schilling · 2026-02-22T18:40:28Z

@kkm-horikawa can you merge your changes in from your other PR when you have time please?

kkm-horikawa mentioned this pull request Jan 12, 2026

SQL panel freezes or crashes when queries contain large IN clauses (e.g., thousands of UUIDs) #2293

Closed

This comment was marked as outdated.

Sign in to view

kkm-horikawa force-pushed the fix/graceful-degradation-large-sql branch from 140dd68 to 9ac9ddb Compare January 14, 2026 05:24

kkm-horikawa and others added 5 commits January 14, 2026 14:32

Fix double HTML escaping in skipped SQL preview

f03c1ee

The preview was being escaped twice - once when assigned to the preview variable and again when inserted into the HTML string.

Add changelog entry for SQL_PRETTIFY_MAX_LENGTH

93066e0

[pre-commit.ci] auto fixes from pre-commit.com hooks

628930f

for more information, see https://pre-commit.ci

Remove SQLParseError handling per review feedback

0809510

Split SQLParseError exception handling into a separate PR as requested. This PR now focuses solely on the SQL_PRETTIFY_MAX_LENGTH setting.

kkm-horikawa force-pushed the fix/graceful-degradation-large-sql branch from 9ac9ddb to 0809510 Compare January 14, 2026 05:33

tim-schilling requested changes Jan 14, 2026

View reviewed changes

kkm-horikawa force-pushed the fix/graceful-degradation-large-sql branch from ddb1b7e to 7da3aa9 Compare January 15, 2026 02:59

tim-schilling reviewed Jan 19, 2026

View reviewed changes

Handle SQLParseError gracefully by disabling grouping

ae42a1e

kkm-horikawa force-pushed the fix/graceful-degradation-large-sql branch from 248e656 to ae42a1e Compare January 20, 2026 05:39

kkm-horikawa requested a review from tim-schilling January 20, 2026 05:49

fix: resolve merge conflicts

e16db82

kkm-horikawa force-pushed the fix/graceful-degradation-large-sql branch from 41e9083 to e16db82 Compare January 23, 2026 05:39

[pre-commit.ci] auto fixes from pre-commit.com hooks

9241507

for more information, see https://pre-commit.ci

tim-schilling reviewed Jan 27, 2026

View reviewed changes

tim-schilling requested changes Jan 27, 2026

View reviewed changes

kkm-horikawa mentioned this pull request Jan 29, 2026

refactor: improve test using side_effect and move imports to top kkm-horikawa/django-debug-toolbar#1

Open

tim-schilling mentioned this pull request Mar 9, 2026

Fix #2292: Gracefully handle SQLParseError in SQL formatting #2322

Closed

3 tasks

		max_length = dt_settings.get_config()["SQL_PRETTIFY_MAX_LENGTH"]
		if max_length and len(sql) > max_length:

	"SQL_PRETTIFY_MAX_LENGTH": 50000, # Skip formatting for SQL longer than this
	"PRETTIFY_SQL_MAX_LENGTH": 10000, # Skip formatting for SQL longer than this

		max_length = dt_settings.get_config()["PRETTIFY_SQL_MAX_LENGTH"]
		if max_length and len(sql) > max_length:

Conversation

kkm-horikawa commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist:

Uh oh!

This comment was marked as outdated.

tim-schilling commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kkm-horikawa commented Jan 14, 2026

Uh oh!

tim-schilling left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kkm-horikawa commented Jan 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kkm-horikawa Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 27, 2026

Coverage report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tim-schilling left a comment

Choose a reason for hiding this comment

Uh oh!

tim-schilling commented Feb 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kkm-horikawa commented Jan 9, 2026 •

edited

Loading

tim-schilling commented Jan 13, 2026 •

edited

Loading

kkm-horikawa Jan 20, 2026 •

edited

Loading