Documentation: build failure with Sphinx 7.2.2

[jayaddison]

The documentation for pip-tools fails to build using the currently-specified version 7.2.2 of Sphinx.

This appears to be due to an issue that has been resolved from version 7.2.3 of Sphinx: sphinx-doc/sphinx#11625

Environment Versions

1. OS: Debian Linux (forky / testing)
2. Python version: Python 3.13.7
3. pip version: pip 25.1
4. pip-tools version: pip-compile, version 7.5.1.dev28+dirty

Steps to replicate

1. Set the working directory to the docs dir.
2. pip install -r requirements.txt
3. pip install -e .. # ensure pip-tools itself is installed as a package
4. sphinx-build -b html . _build

Expected result

A complete successful HTML documentation build, with output in the _build directory.

Actual result

The build fails with the following traceback:

Traceback (most recent call last):
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/cmd/build.py", line 298, in build_main
    app.build(args.force_all, args.filenames)
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/application.py", line 355, in build
    self.builder.build_update()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 293, in build_update
    self.build(to_build,
    ~~~~~~~~~~^^^^^^^^^^
               summary=__('targets for %d source files that are out of date') %
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               len(to_build))
               ^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 313, in build
    updated_docnames = set(self.read())
                           ~~~~~~~~~^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 420, in read
    self._read_serial(docnames)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 441, in _read_serial
    self.read_doc(docname)
    ~~~~~~~~~~~~~^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/builders/__init__.py", line 498, in read_doc
    publisher.publish()
    ~~~~~~~~~~~~~~~~~^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/core.py", line 234, in publish
    self.document = self.reader.read(self.source, self.parser,
                    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
                                     self.settings)
                                     ^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/io.py", line 105, in read
    self.parse()
    ~~~~~~~~~~^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/readers/__init__.py", line 76, in parse
    self.parser.parse(self.input, document)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/parsers.py", line 81, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/parsers/rst/states.py", line 169, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
                                 input_source=document['source'])
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/statemachine.py", line 233, in run
    context, next_state, result = self.check_line(
                                  ~~~~~~~~~~~~~~~^
        context, state, transitions)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/statemachine.py", line 445, in check_line
    return method(match, context, next_state)
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/parsers/rst/states.py", line 2357, in explicit_markup
    self.explicit_list(blank_finish)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/parsers/rst/states.py", line 2382, in explicit_list
    newline_offset, blank_finish = self.nested_list_parse(
                                   ~~~~~~~~~~~~~~~~~~~~~~^
          self.state_machine.input_lines[offset:],
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
          blank_finish=blank_finish,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
          match_titles=self.state_machine.match_titles)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/parsers/rst/states.py", line 316, in nested_list_parse
    state_machine.run(block, input_offset, memo=self.memo,
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                      node=node, match_titles=match_titles)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/statemachine.py", line 233, in run
    context, next_state, result = self.check_line(
                                  ~~~~~~~~~~~~~~~^
        context, state, transitions)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/statemachine.py", line 445, in check_line
    return method(match, context, next_state)
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/parsers/rst/states.py", line 2660, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
                             ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/parsers/rst/states.py", line 2367, in explicit_construct
    return method(self, expmatch)
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/parsers/rst/states.py", line 2104, in directive
    return self.run_directive(
           ~~~~~~~~~~~~~~~~~~^
        directive_class, match, type_name, option_presets)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/docutils/parsers/rst/states.py", line 2154, in run_directive
    result = directive_instance.run()
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/ext/autodoc/directive.py", line 139, in run
    documenter.generate(more_content=self.content)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/ext/autodoc/__init__.py", line 1886, in generate
    return super().generate(more_content=more_content,
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
                            check_module=check_module,
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
                            all_members=all_members)
                            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/ext/autodoc/__init__.py", line 869, in generate
    if not self.parse_name():
           ~~~~~~~~~~~~~~~^^
  File "<base-dir>/pip-tools/docs/.venv/lib/python3.13/site-packages/sphinx/ext/autodoc/__init__.py", line 388, in parse_name
    assert matched is not None
           ^^^^^^^^^^^^^^^^^^^
AssertionError

Edit: correction: update hyperlink to point to the relevant Sphinx issue.

[sirosen]

@jayaddison, I don't think you're currently listed with attribution in our changelog. You're in the commit history, of course, but maybe this would be a good one to get your name in there?

I'd welcome a PR to add a contrib fragment for this (2240) or any of the other items you've touched to give you attribution.

@webknjaz, do you think we should go ahead and add such a one? Or maybe a $sha.contrib.md which simply says "Fixes and improvements for documentation and doc builds"?

[jayaddison]

Thank you, @sirosen :)

I didn't feel a need to add one since I feel that each of the contributions I've made is small -- that said, I'd be happy with an attribution along the lines of the documentation fixups you mention.

[webknjaz]

@sirosen agreed. It's good to have such change notes.

---

The documentation for pip-tools fails to build using the currently-specified version 7.2.2 of Sphinx.

This appears to be due to an issue that has been resolved from version 7.2.3 of Sphinx: sphinx-doc/sphinx#11523

I don't understand why Sphinx breaks for you as RTD was using the pins already and it was green (same for GHA). @jayaddison were you seeing it breaking anywhere outside your machine?

Here's one of the builds of your PR that uses Sphinx v7.2.2 and is green: https://app.readthedocs.org/projects/pip-tools/builds/29190509/#282141394--1. It fact, if you look into the builds list, you won't see any recent failures.

FWIW, the correct integrated method for running it is tox run -e build-docs -q. Perhaps you have something additional messing up your environment. I recommend you use the same invocations as our CI does to reduce divergence.

[webknjaz]

One notable difference is that we have python -m sphinx (so a runpy invocation) in tox.ini which mutates sys.path. So it's possible that this influences the docs build. I don't believe that bumping the version has anything to do with it.

[sirosen]

Sorry guys, I was over-eager in merging #2217 !
I think it's a pretty harmless mistake, but still a mistake.

I saw that it was bumping versions, which I considered "not quite right, but not a big deal" and didn't want to block the PR on it¹. I didn't realize at the time that it linked to this issue. If I'd read the chain more carefully, I would probably have held that PR to get clarity on this.

I'm not sure if any follow-up here is necessary, although if @jayaddison finds the error recurs we can reopen to discuss.
The tox run -e build-docs invocation is working fine for me, and is now in use in RTD and seems to be running okay. I intend to leave this closed as a mix of "CNR" + "reporter considers it resolved".

Footnotes

1. This is, IMO, the major mistake. Overlooking something for the sake of expediency, which can often cost more work later. Sorry! 😅 ↩

[jayaddison]

@webknjaz

I don't understand why Sphinx breaks for you as RTD was using the pins already and it was green (same for GHA). @jayaddison were you seeing it breaking anywhere outside your machine?

No, I did not encounter breakage elsewhere; and building locally with tox -e build-docs does work fine for me.

Reverting to sphinx v7.2.2, regardless of the sphinxcontrib-serializinghtml upgrade -- allows me to replicate the problem when using sphinx-build directly.

Self-quoting:

The documentation for pip-tools fails to build using the currently-specified version 7.2.2 of Sphinx.

This appears to be due to an issue that has been resolved from version 7.2.3 of Sphinx: sphinx-doc/sphinx#11523

In addition to the package naming issue (pip install sphinx-build instead of pip install sphinx) previously noted it does not help that I linked to the wrong issue in Sphinx here; that should have been: sphinx-doc/sphinx#11625

[jayaddison]

Sorry guys, I was over-eager in merging #2217 ! I think it's a pretty harmless mistake, but still a mistake.

@sirosen I could have been more careful to check for alternative ways to build the documentation; I tend to try to like to build using a minimal command-line and dependency base, and sometimes that can cause me to follow an approach that varies from continuous integration. I'd like to take some time to try to determine why exactly this error occurred (that is, whether it is the sys.path issue @webknjaz mentions or something else).

After merge I re-checked the requirements.txt and confirmed that in fact it is an autogenerated file (I suppose I should have considered that reasonably likely within pip-tools itself) - so again a mistake on my behalf there.

[jayaddison]

The directory that the virtualenv is created in appears to be a factor; if created within the docs dir (as I reported in this issue), then the problem is replicable -- if created within the top-level pip-tools.git clone directory, then the problem does not occur. So it does seem likely to be load-path related.

[webknjaz]

Oh, interesting... I suppose Sphinx was looking into the virtualenv dir recursively in your case while discovering the source files. I can imagine how that would break things.

I don't think I've ever seen venvs being a part of the docs dir, it's rather uncommon.

Anyway, I consider the matter resolved. Just wanted to make sure the PR changes weren't misguided because of the error in judgment: that justification might be misleading when someone performs git paleontology in the future..

---

P.S. I saw the linked issue being wrong/broken but forgot to mention that bit of confusion 😂

[jayaddison]

Thanks - even so, it does seem strange to me that the location of the virtualenv makes any difference at all; I feel that provided that all relevant Python packages (e.g. from the project, its dependencies, and the documentation) are available, then the documentation should be buildable. I'll try to figure out a bit more about when/how the regression occurred in Sphinx, and to work backwards from that.

[jayaddison]

I think that the cause of problem is most neatly summarised by the discussion at: sphinx-doc/sphinx#2066

(it certainly wasn't/isn't clear from the error messaging though -- and at some level I feel that the assertion that caused it, subsequently removed, may be valid and worthwhile)