Feature Request: Option to highlight text in request stack

[erikcw]

I've been using httpdbg to find 3rd party API calls in my test suite so that I can mock them. I find myself scrolling through the request "stack" tab quite often and it would be really useful to highlight the lines in the trace from my application code vs library code.

Perhaps the best way to approach this flexibly is to add a setting where users can add an arbitrary regex pattern and a color code. So I could have all my test files highlighted in one color, and all my application code highlighted in another. Maybe a certain library could be highlighted using a 3rd color.

Thanks again for the great tool!

[cle-b]

Have you tried configuring a custom initiator? By default, the initiator of the requests is the HTTP library itself, but we can define any package as the initiator of the requests. It could be the third-party library you want to mock. You can find the documentation here. I think that may be helpful in your case. If necessary, I can update the filtering mechanism to allow filtering the requests by initiator.

Are you using httpdbg with pytest or with another test runner? I ask this question because, with pytest, the requests are grouped by test and not by initiator, unlike in all other cases. This view may not be useful in your case for easily finding the requests sent from a third-party library.

[erikcw]

I have tried messing with initiators a little but haven't gotten them to work. I'm testing a Django application using the standard unittest module.

For instance, I'm trying to mock out calls to Stripe, so I've used: pyhttpdbg -i stripe --script manage.py test --noinput --timing -- but that doesn't seem to change the output at all so I'm clearly doing something wrong.

[cle-b]

I have never worked with Django or Stripe, but I tried to create a small Django app that uses the Stripe API client, and it seems that the initiator configuration works in my case.

pyhttpdbg --script  manage.py test
.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
  httpdbg - HTTP(S) requests available at http://localhost:4909/
.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.203s

OK

pyhttpdbg -i stripe --script  manage.py test
.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
  httpdbg - HTTP(S) requests available at http://localhost:4909/
.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
Found 1 test(s).
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.245s

OK

I don't know why it doesn't work in your case. Yesterday, I fixed an issue related to the initiators. Maybe you can try the latest version of httpdbg, although I don't think it will fix your problem since it wasn't affecting the custom initiators. Otherwise, I don't have any other ideas, sorry. If you provide me with more details, like the version of Python you're using, your OS, the versions of Django and Stripe, etc., I can take a closer look.

[erikcw]

So when I try to use an initiator, I start getting strange errors about invalid headers.

DATABASE_URL=sqlite://:memory: pyhttpdbg -i stripe --script manage.py test --noinput --timing example.tests.TestExample

.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
  httpdbg - HTTP(S) requests available at http://localhost:4909/
.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
WARNING [celery - setup_periodic_tasks()] Mock Screenshots are enabled in this environment
INFO [djstripe_monkeypatch -                 init()] Initializing djstripe monkey patch
INFO [services - load_rate_limit_overrides()] Skipping loading rate limit overrides for this environment.
Found 2 test(s).
Creating test database for alias 'default'...
Traceback (most recent call last):
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/http_client.py", line 323, in _request_internal
    result = self._thread_local.session.request(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/httpdbg/hooks/requests.py", line 21, in hook
    ret = method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/requests/sessions.py", line 575, in request
    prep = self.prepare_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/requests/sessions.py", line 484, in prepare_request
    p.prepare(
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/requests/models.py", line 368, in prepare
    self.prepare_headers(headers)
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/requests/models.py", line 490, in prepare_headers
    check_header_validity(header)
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/requests/utils.py", line 1042, in check_header_validity
    _validate_header_part(header, value, 1)
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/requests/utils.py", line 1051, in _validate_header_part
    raise InvalidHeader(
requests.exceptions.InvalidHeader: Header part (<module 'stripe.api_version' from '/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/api_version.py'>) from ('Stripe-Version', <module 'stripe.api_version' from '/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/api_version.py'>) must be of type str or bytes, not <class 'module'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/httpdbg/mode_script.py", line 18, in run_script
    spec.loader.exec_module(module)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/erik/Dropbox/home/git/my-backend/manage.py", line 22, in <module>
    main()
  File "/Users/erik/Dropbox/home/git/my-backend/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/commands/test.py", line 24, in run_from_argv
    super().run_from_argv(argv)
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/commands/test.py", line 68, in handle
    failures = test_runner.run_tests(test_labels)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/test/runner.py", line 1054, in run_tests
    old_config = self.setup_databases(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/test/runner.py", line 950, in setup_databases
    return _setup_databases(
           ^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/test/utils.py", line 221, in setup_databases
    connection.creation.create_test_db(
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/db/backends/base/creation.py", line 78, in create_test_db
    call_command(
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/commands/migrate.py", line 383, in handle
    emit_post_migrate_signal(
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/core/management/sql.py", line 52, in emit_post_migrate_signal
    models.signals.post_migrate.send(
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
    return [
           ^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/Dropbox/home/git/my-backend/My/payments/apps.py", line 10, in populate_stripe_models
    tests_utils.setup_test_payments()
  File "/Users/erik/Dropbox/home/git/my-backend/My/payments/tests/utils.py", line 259, in setup_test_payments
    _setup_test_stripe()
  File "/Users/erik/Dropbox/home/git/my-backend/My/payments/tests/utils.py", line 306, in _setup_test_stripe
    _cache_stripe_account_fixtures(cassette)
  File "/Users/erik/Dropbox/home/git/my-backend/My/payments/tests/utils.py", line 272, in _cache_stripe_account_fixtures
    accounts = stripe.Account.list()
               ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/api_resources/abstract/listable_api_resource.py", line 15, in list
    return cls._static_request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/api_resources/abstract/api_resource.py", line 139, in _static_request
    response, api_key = requestor.request(method_, url_, params, headers)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/httpdbg/hooks/generic.py", line 42, in hook
    ret = method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/api_requestor.py", line 119, in request
    rbody, rcode, rheaders, my_api_key = self.request_raw(
                                         ^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/httpdbg/hooks/generic.py", line 42, in hook
    ret = method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/api_requestor.py", line 366, in request_raw
    rcontent, rcode, rheaders = self._client.request_with_retries(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/httpdbg/hooks/generic.py", line 42, in hook
    ret = method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/http_client.py", line 115, in request_with_retries
    return self._request_with_retries_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/httpdbg/hooks/generic.py", line 42, in hook
    ret = method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/http_client.py", line 170, in _request_with_retries_internal
    raise connection_error
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/http_client.py", line 142, in _request_with_retries_internal
    response = self.request(method, url, headers, post_data)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/httpdbg/hooks/generic.py", line 42, in hook
    ret = method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/http_client.py", line 296, in request
    return self._request_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/httpdbg/hooks/generic.py", line 42, in hook
    ret = method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/http_client.py", line 353, in _request_internal
    self._handle_request_error(e)
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/httpdbg/hooks/generic.py", line 42, in hook
    ret = method(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/http_client.py", line 405, in _handle_request_error
    raise error.APIConnectionError(msg, should_retry=should_retry)
stripe.error.APIConnectionError: Unexpected error communicating with Stripe.  If this problem persists,
let us know at support@stripe.com.

(Network error: InvalidHeader: Header part (<module 'stripe.api_version' from '/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/api_version.py'>) from ('Stripe-Version', <module 'stripe.api_version' from '/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/api_version.py'>) must be of type str or bytes, not <class 'module'>)
.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
  httpdbg - HTTP(S) requests available at http://localhost:4909/
.... - - .--. -.. -... --. .... - - .--. -.. -... --. .... - - .--. -.. -... --.
Waiting until all the requests have been loaded in the web interface.
Press Ctrl+C to quit.

It looks like the initiator is doing the equivalent of import stripe.api_version. If I remove the initiator completely or switch to something like -i requests everything works just fine.

IPython 8.26.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import stripe

In [2]: stripe.api_version

In [3]: import stripe.api_version

In [4]: stripe.api_version
Out[4]: <module 'stripe.api_version' from '/Users/erik/.pyenv/versions/my-backend/lib/python3.11/site-packages/stripe/api_version.py'>

Here is the function making the stripe call:

def _cache_stripe_account_fixtures(cassette):
    """
    Populate VCR cache with stripe accounts.
    """
    old_key = stripe.api_key
    stripe.api_key = settings.STRIPE_TEST_SECRET_KEY
    accounts = stripe.Account.list()
    for a in accounts:
        a.refresh()
    stripe.api_key = old_key

[cle-b]

Thank you for the bug description! It's much clearer now.

When a custom initiator is configured, httpdbg imports the package and all its subpackages to locate the functions to hook. In the case of the stripe library, the issue arises because there is both a variable named api_version defined in stripe/__init__.py and a module called stripe/api_version.py. During the initiator’s initialization, we import the stripe.api_version module, and later, when the stripe library attempts to use the variable stripe.api_version, it fails because stripe.api_version is no longer a string as expected but a module.

I'm trying to find a generic solution but this is not obvious :-)

[cle-b]

The custom initiator feature has been fixed in httpdbg 1.2.2 by #192.