0.4.0 causes Sphinx to be imported every time Python starts

[jsiirola]

Description

Beginning in version 0.4.0, having sphinx-jinja2-compat installed causes all of Sphinx to be imported every time Python is started. This has a noticeable impact on the Python startup time.

Steps to Reproduce

1. pip install sphinx-jinja2-compat==0.4.0
2. time python -c 'import sys; print(len(sys.modules)); print(sorted(sys.modules))'

Actual result:

% time python -c 'import sys; print("Number of loaded modules: ", len(sys.modules)); print(sorted(sys.modules))'
Number of loaded modules: 370
['__future__', '__main__', '_abc', '_ast', '_bisect', '_blake2', '_bz2', '_codecs', '_collections', '_collections_abc', '_compat_pickle', '_compression', '_contextvars', '_csv', '_ctypes', '_datetime', '_decimal', '_distutils_hack', '_frozen_importlib', '_frozen_importlib_external', '_functools', '_hashlib', '_heapq', '_imp', '_io', '_json', '_locale', '_lzma', '_opcode', '_operator', '_pickle', '_queue', '_random', '_sha512', '_signal', '_sitebuiltins', '_socket', '_sre', '_stat', '_string', '_struct', '_sysconfigdata__linux_x86_64-linux-gnu', '_thread', '_typing', '_uuid', '_warnings', '_weakref', '_weakrefset', '_zoneinfo', 'abc', 'argparse', 'array', 'ast', 'atexit', 'babel', 'babel.core', 'babel.dates', 'babel.localedata', 'babel.localtime', 'babel.localtime._fallback', 'babel.localtime._helpers', 'babel.localtime._unix', 'babel.messages', 'babel.messages.catalog', 'babel.messages.mofile', 'babel.messages.plurals', 'babel.messages.pofile', 'babel.plural', 'babel.util', 'base64', 'binascii', 'bisect', 'builtins', 'bz2', 'calendar', 'codecs', 'collections', 'collections.abc', 'colorama', 'colorama.ansi', 'colorama.ansitowin32', 'colorama.initialise', 'colorama.win32', 'colorama.winterm', 'configparser', 'contextlib', 'contextvars', 'copy', 'copyreg', 'csv', 'ctypes', 'ctypes._endian', 'dataclasses', 'datetime', 'decimal', 'difflib', 'dis', 'docutils', 'docutils.core', 'docutils.frontend', 'docutils.io', 'docutils.languages', 'docutils.nodes', 'docutils.parsers', 'docutils.parsers.rst', 'docutils.parsers.rst.directives', 'docutils.parsers.rst.languages', 'docutils.parsers.rst.languages.en', 'docutils.parsers.rst.roles', 'docutils.parsers.rst.states', 'docutils.parsers.rst.tableparser', 'docutils.readers', 'docutils.readers.doctree', 'docutils.readers.standalone', 'docutils.statemachine', 'docutils.transforms', 'docutils.transforms.frontmatter', 'docutils.transforms.misc', 'docutils.transforms.parts', 'docutils.transforms.references', 'docutils.transforms.universal', 'docutils.utils', 'docutils.utils.code_analyzer', 'docutils.utils.punctuation_chars', 'docutils.utils.roman', 'docutils.utils.smartquotes', 'docutils.utils.urischemes', 'docutils.writers', 'email', 'email._encoded_words', 'email._parseaddr', 'email._policybase', 'email.base64mime', 'email.charset', 'email.encoders', 'email.errors', 'email.header', 'email.iterators', 'email.message', 'email.quoprimime', 'email.utils', 'encodings', 'encodings.aliases', 'encodings.utf_8', 'enum', 'errno', 'filecmp', 'fnmatch', 'functools', 'genericpath', 'gettext', 'hashlib', 'heapq', 'importlib', 'importlib._abc', 'importlib._bootstrap', 'importlib._bootstrap_external', 'importlib.abc', 'importlib.machinery', 'importlib.metadata', 'importlib.metadata._adapters', 'importlib.metadata._collections', 'importlib.metadata._functools', 'importlib.metadata._itertools', 'importlib.metadata._meta', 'importlib.metadata._text', 'importlib.resources', 'importlib.resources._adapters', 'importlib.resources._common', 'importlib.resources._legacy', 'importlib.resources.abc', 'importlib.util', 'inspect', 'io', 'ipaddress', 'itertools', 'jinja2', 'jinja2._identifier', 'jinja2.async_utils', 'jinja2.bccache', 'jinja2.compiler', 'jinja2.defaults', 'jinja2.environment', 'jinja2.exceptions', 'jinja2.filters', 'jinja2.idtracking', 'jinja2.lexer', 'jinja2.loaders', 'jinja2.nodes', 'jinja2.optimizer', 'jinja2.parser', 'jinja2.runtime', 'jinja2.tests', 'jinja2.utils', 'jinja2.visitor', 'json', 'json.decoder', 'json.encoder', 'json.scanner', 'keyword', 'linecache', 'locale', 'logging', 'logging.handlers', 'lzma', 'markupsafe', 'markupsafe._speedups', 'marshal', 'math', 'ntpath', 'numbers', 'opcode', 'operator', 'optparse', 'os', 'os.path', 'packaging', 'packaging._structures', 'packaging.version', 'pathlib', 'pickle', 'platform', 'posix', 'posixpath', 'pprint', 'pygments', 'pygments.filter', 'pygments.filters', 'pygments.formatter', 'pygments.formatters', 'pygments.formatters._mapping', 'pygments.formatters.html', 'pygments.formatters.latex', 'pygments.lexer', 'pygments.lexers', 'pygments.lexers._css_builtins', 'pygments.lexers._lilypond_builtins', 'pygments.lexers._mapping', 'pygments.lexers._scheme_builtins', 'pygments.lexers.c_cpp', 'pygments.lexers.css', 'pygments.lexers.data', 'pygments.lexers.html', 'pygments.lexers.javascript', 'pygments.lexers.jvm', 'pygments.lexers.lilypond', 'pygments.lexers.lisp', 'pygments.lexers.markup', 'pygments.lexers.python', 'pygments.lexers.ruby', 'pygments.lexers.shell', 'pygments.lexers.special', 'pygments.modeline', 'pygments.plugin', 'pygments.regexopt', 'pygments.style', 'pygments.styles', 'pygments.styles._mapping', 'pygments.styles.friendly', 'pygments.token', 'pygments.unistring', 'pygments.util', 'pytz', 'pytz.exceptions', 'pytz.lazy', 'pytz.tzfile', 'pytz.tzinfo', 'queue', 'quopri', 'random', 're', 're._casefix', 're._compiler', 're._constants', 're._parser', 'reprlib', 'select', 'selectors', 'shutil', 'site', 'socket', 'sphinx', 'sphinx-prompt', 'sphinx.addnodes', 'sphinx.application', 'sphinx.config', 'sphinx.deprecation', 'sphinx.directives', 'sphinx.domains', 'sphinx.domains._domains_container', 'sphinx.domains._index', 'sphinx.domains.std', 'sphinx.environment', 'sphinx.environment.adapters', 'sphinx.environment.adapters.toctree', 'sphinx.errors', 'sphinx.events', 'sphinx.extension', 'sphinx.highlighting', 'sphinx.io', 'sphinx.locale', 'sphinx.parsers', 'sphinx.project', 'sphinx.pycode', 'sphinx.pycode.ast', 'sphinx.pycode.parser', 'sphinx.pygments_styles', 'sphinx.registry', 'sphinx.roles', 'sphinx.transforms', 'sphinx.transforms.i18n', 'sphinx.transforms.references', 'sphinx.util', 'sphinx.util._files', 'sphinx.util._importer', 'sphinx.util._lines', 'sphinx.util._pathlib', 'sphinx.util._serialise', 'sphinx.util._timestamps', 'sphinx.util._uri', 'sphinx.util.build_phase', 'sphinx.util.console', 'sphinx.util.display', 'sphinx.util.docfields', 'sphinx.util.docutils', 'sphinx.util.i18n', 'sphinx.util.index_entries', 'sphinx.util.inspect', 'sphinx.util.logging', 'sphinx.util.matching', 'sphinx.util.nodes', 'sphinx.util.osutil', 'sphinx.util.parsing', 'sphinx.util.rst', 'sphinx.util.tags', 'sphinx.util.texescape', 'sphinx.util.typing', 'sphinx.versioning', 'sphinx_jinja2_compat', 'sphinx_jinja2_compat._installers', 'sphinx_prompt', 'stat', 'string', 'struct', 'sys', 'sysconfig', 'tempfile', 'textwrap', 'threading', 'time', 'token', 'tokenize', 'traceback', 'types', 'typing', 'typing.io', 'typing.re', 'unicodedata', 'urllib', 'urllib.parse', 'usercustomize', 'uuid', 'warnings', 'weakref', 'zipfile', 'zipimport', 'zlib', 'zoneinfo', 'zoneinfo._common', 'zoneinfo._tzpath']

real	0m0.693s
user	0m0.554s
sys	0m0.116s

Expected result:

The following was with sphinx-jinja2-compat==0.3.0:

% time python -c 'import sys; print("Number of loaded modules: ", len(sys.modules)); print(sorted(sys.modules))'
Number of loaded modules: 136
['__future__', '__main__', '_abc', '_ast', '_bisect', '_blake2', '_bz2', '_codecs', '_collections', '_collections_abc', '_compat_pickle', '_compression', '_distutils_hack', '_frozen_importlib', '_frozen_importlib_external', '_functools', '_hashlib', '_imp', '_io', '_json', '_lzma', '_opcode', '_operator', '_pickle', '_random', '_sha512', '_signal', '_sitebuiltins', '_sre', '_stat', '_string', '_struct', '_thread', '_typing', '_warnings', '_weakref', '_weakrefset', 'abc', 'ast', 'bisect', 'builtins', 'bz2', 'codecs', 'collections', 'collections.abc', 'contextlib', 'copyreg', 'dis', 'encodings', 'encodings.aliases', 'encodings.utf_8', 'enum', 'errno', 'fnmatch', 'functools', 'genericpath', 'hashlib', 'importlib', 'importlib._abc', 'importlib._bootstrap', 'importlib._bootstrap_external', 'importlib.machinery', 'importlib.util', 'inspect', 'io', 'ipaddress', 'itertools', 'jinja2', 'jinja2._identifier', 'jinja2.async_utils', 'jinja2.bccache', 'jinja2.compiler', 'jinja2.defaults', 'jinja2.environment', 'jinja2.exceptions', 'jinja2.filters', 'jinja2.idtracking', 'jinja2.lexer', 'jinja2.loaders', 'jinja2.nodes', 'jinja2.optimizer', 'jinja2.parser', 'jinja2.runtime', 'jinja2.tests', 'jinja2.utils', 'jinja2.visitor', 'json', 'json.decoder', 'json.encoder', 'json.scanner', 'keyword', 'linecache', 'lzma', 'markupsafe', 'markupsafe._speedups', 'marshal', 'math', 'numbers', 'opcode', 'operator', 'os', 'os.path', 'pickle', 'posix', 'posixpath', 'random', 're', 're._casefix', 're._compiler', 're._constants', 're._parser', 'reprlib', 'shutil', 'site', 'sphinx_jinja2_compat', 'sphinx_jinja2_compat._installers', 'stat', 'string', 'struct', 'sys', 'tempfile', 'threading', 'time', 'token', 'tokenize', 'types', 'typing', 'typing.io', 'typing.re', 'urllib', 'urllib.parse', 'usercustomize', 'warnings', 'weakref', 'zipimport', 'zlib']

real	0m0.144s
user	0m0.109s
sys	0m0.032s

The preferred solution is to not unconditionally import anything as part of the python startup. Uninstalling (only) sphinx-jinja2-compat further improves the Python startup time:

% time python -c 'import sys; print("Number of loaded modules: ", len(sys.modules)); print(sorted(sys.modules))'
Number of loaded modules:  35
['__main__', '_abc', '_codecs', '_collections_abc', '_distutils_hack', '_frozen_importlib', '_frozen_importlib_external', '_imp', '_io', '_signal', '_sitebuiltins', '_stat', '_thread', '_warnings', '_weakref', 'abc', 'builtins', 'codecs', 'encodings', 'encodings.aliases', 'encodings.utf_8', 'genericpath', 'io', 'marshal', 'os', 'os.path', 'posix', 'posixpath', 'site', 'sphinx_jinja2_compat', 'stat', 'sys', 'time', 'usercustomize', 'zipimport']

real	0m0.019s
user	0m0.014s
sys	0m0.005s

Reproduces how often:

Deterministic (every time).

Version

• Operating System: Linux
• Python: 3.11
• sphinx-jinja2-compat: 0.4.0

Installation source

PyPI

Other Additional Information:

[domdfcoding]

My expectation (but maybe I was wrong) was that sphinx-jinja2-compat would only be installed in environments for building Sphinx documentation, so this wouldn't be an issue as sphinx will be imported sooner or later. But I take it your setup is different?

There is an escape hatch. If you set the NO_SPHINX_JINJA2_COMPAT environment variable (to anything) sphinx-jinja2-compat is imported but doesn't do anything beyond import sys, typing and types.

[austinyu]

I think I have a similar problem. When sphinx-jinja2-compat is upgraded to 0.4, I see these messages in my terminal whenever I use the virtual environment that has the package.

Error processing line 1 of ...\.venv\Lib\site-packages\_sphinx_jinja2_compat.pth:

  Traceback (most recent call last):
    File "<frozen site>", line 206, in addpackage
    File "<string>", line 1, in <module>
    File "...\.venv\Lib\site-packages\sphinx_jinja2_compat\__init__.py", line 81, in <module>
      __import__("_distutils_hack").add_shim()
      ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  ModuleNotFoundError: No module named '_distutils_hack'

Remainder of file ignored

[domdfcoding]

@austinyu I think your issue is a different problem but with ultimately the same cause - a new patch for sphinx-prompt. Try 0.4.1 and it should fix your problem.

[jsiirola]

FWIW, I tend to set up a single common environment using Environment Modules that I use for everything [development, build, test, use]. With a large number of users and development / testing machines, it tends to be easier to set up a single "known good" environment and make it available everywhere instead of having a pile of virtual environments scattered around.

I saw the environment variable, but have to admit that rather than getting all the users to add it to their default environment (or putting it in the main environment module setup), the simpler option for our use case may be to remove sphinx-jinja2-compat/sphinx-toolbox from the default environment module and let users who need it pip install --user it when necessary.

[Tatsh]

I think I have a similar problem. When sphinx-jinja2-compat is upgraded to 0.4, I see these messages in my terminal whenever I use the virtual environment that has the package.

Error processing line 1 of ...\.venv\Lib\site-packages\_sphinx_jinja2_compat.pth:

  Traceback (most recent call last):
    File "<frozen site>", line 206, in addpackage
    File "<string>", line 1, in <module>
    File "...\.venv\Lib\site-packages\sphinx_jinja2_compat\__init__.py", line 81, in <module>
      __import__("_distutils_hack").add_shim()
      ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  ModuleNotFoundError: No module named '_distutils_hack'

Remainder of file ignored

Also seeing this and I am using 0.4.1.

[Tatsh]

It looks like you should except AttributeError here based on the error we are getting.