gh-140550: PEP 793 reference documentation (GH-141197)

* gh-140550: PEP 793 reference documentation

Since the PEP calls for soft-deprecation of the existing initialization
function, this reorganizes the relevant docs to put the new way of
doing things first, and de-emphasize the old.

Some bits, like the tutorial, are left out of this patch. (See the
issue for a list.)

Author: encukou

Doc/c-api/extension-modules.rst

@@ -8,7 +8,8 @@ Defining extension modules
 A C extension for CPython is a shared library (for example, a ``.so`` file
 on Linux, ``.pyd`` DLL on Windows), which is loadable into the Python process
 (for example, it is compiled with compatible compiler settings), and which
-exports an :ref:`initialization function <extension-export-hook>`.
+exports an :dfn:`export hook` function (or an
+old-style :ref:`initialization function <extension-pyinit>`).
 
 To be importable by default (that is, by
 :py:class:`importlib.machinery.ExtensionFileLoader`),
@@ -23,25 +24,127 @@ and must be named after the module name plus an extension listed in
    One suitable tool is Setuptools, whose documentation can be found at
    https://setuptools.pypa.io/en/latest/setuptools.html.
 
-Normally, the initialization function returns a module definition initialized
-using :c:func:`PyModuleDef_Init`.
-This allows splitting the creation process into several phases:
+.. _extension-export-hook:
+
+Extension export hook
+.....................
+
+.. versionadded:: next
+
+   Support for the :samp:`PyModExport_{<name>}` export hook was added in Python
+   3.15. The older way of defining modules is still available: consult either
+   the :ref:`extension-pyinit` section or earlier versions of this
+   documentation if you plan to support earlier Python versions.
+
+The export hook must be an exported function with the following signature:
+
+.. c:function:: PyModuleDef_Slot *PyModExport_modulename(void)
+
+For modules with ASCII-only names, the :ref:`export hook <extension-export-hook>`
+must be named :samp:`PyModExport_{<name>}`,
+with ``<name>`` replaced by the module's name.
+
+For non-ASCII module names, the export hook must instead be named
+:samp:`PyModExportU_{<name>}` (note the ``U``), with ``<name>`` encoded using
+Python's *punycode* encoding with hyphens replaced by underscores. In Python:
+
+.. code-block:: python
+
+    def hook_name(name):
+        try:
+            suffix = b'_' + name.encode('ascii')
+        except UnicodeEncodeError:
+            suffix = b'U_' + name.encode('punycode').replace(b'-', b'_')
+        return b'PyModExport' + suffix
+
+The export hook returns an array of :c:type:`PyModuleDef_Slot` entries,
+terminated by an entry with a slot ID of ``0``.
+These slots describe how the module should be created and initialized.
+
+This array must remain valid and constant until interpreter shutdown.
+Typically, it should use ``static`` storage.
+Prefer using the :c:macro:`Py_mod_create` and :c:macro:`Py_mod_exec` slots
+for any dynamic behavior.
+
+The export hook may return ``NULL`` with an exception set to signal failure.
+
+It is recommended to define the export hook function using a helper macro:
+
+.. c:macro:: PyMODEXPORT_FUNC
+
+   Declare an extension module export hook.
+   This macro:
+
+   * specifies the :c:expr:`PyModuleDef_Slot*` return type,
+   * adds any special linkage declarations required by the platform, and
+   * for C++, declares the function as ``extern "C"``.
 
+For example, a module called ``spam`` would be defined like this::
+
+   PyABIInfo_VAR(abi_info);
+
+   static PyModuleDef_Slot spam_slots[] = {
+       {Py_mod_abi, &abi_info},
+       {Py_mod_name, "spam"},
+       {Py_mod_init, spam_init_function},
+       ...
+       {0, NULL},
+   };
+
+   PyMODEXPORT_FUNC
+   PyModExport_spam(void)
+   {
+       return spam_slots;
+   }
+
+The export hook is typically the only non-\ ``static``
+item defined in the module's C source.
+
+The hook should be kept short -- ideally, one line as above.
+If you do need to use Python C API in this function, it is recommended to call
+``PyABIInfo_Check(&abi_info, "modulename")`` first to raise an exception,
+rather than crash, in common cases of ABI mismatch.
+
+
+.. note::
+
+   It is possible to export multiple modules from a single shared library by
+   defining multiple export hooks.
+   However, importing  them requires a custom importer or suitably named
+   copies/links of the extension file, because Python's import machinery only
+   finds the function corresponding to the filename.
+   See the `Multiple modules in one library <https://peps.python.org/pep-0489/#multiple-modules-in-one-library>`__
+   section in :pep:`489` for details.
+
+
+.. _multi-phase-initialization:
+
+Multi-phase initialization
+..........................
+
+The process of creating an extension module follows several phases:
+
+- Python finds and calls the export hook to get information on how to
+  create the module.
 - Before any substantial code is executed, Python can determine which
   capabilities the module supports, and it can adjust the environment or
   refuse loading an incompatible extension.
-- By default, Python itself creates the module object -- that is, it does
-  the equivalent of :py:meth:`object.__new__` for classes.
-  It also sets initial attributes like :attr:`~module.__package__` and
-  :attr:`~module.__loader__`.
-- Afterwards, the module object is initialized using extension-specific
-  code -- the equivalent of :py:meth:`~object.__init__` on classes.
+  Slots like :c:data:`Py_mod_abi`, :c:data:`Py_mod_gil` and
+  :c:data:`Py_mod_multiple_interpreters` influence this step.
+- By default, Python itself then creates the module object -- that is, it does
+  the equivalent of calling :py:meth:`~object.__new__` when creating an object.
+  This step can be overridden using the :c:data:`Py_mod_create` slot.
+- Python sets initial module attributes like :attr:`~module.__package__` and
+  :attr:`~module.__loader__`, and inserts the module object into
+  :py:attr:`sys.modules`.
+- Afterwards, the module object is initialized in an extension-specific way
+  -- the equivalent of :py:meth:`~object.__init__` when creating an object,
+  or of executing top-level code in a Python-language module.
+  The behavior is specified using the :c:data:`Py_mod_exec` slot.
 
 This is called *multi-phase initialization* to distinguish it from the legacy
-(but still supported) *single-phase initialization* scheme,
-where the initialization function returns a fully constructed module.
-See the :ref:`single-phase-initialization section below <single-phase-initialization>`
-for details.
+(but still supported) :ref:`single-phase initialization <single-phase-initialization>`,
+where an initialization function returns a fully constructed module.
 
 .. versionchanged:: 3.5
 
@@ -53,7 +156,7 @@ Multiple module instances
 
 By default, extension modules are not singletons.
 For example, if the :py:attr:`sys.modules` entry is removed and the module
-is re-imported, a new module object is created, and typically populated with
+is re-imported, a new module object is created and, typically, populated with
 fresh method and type objects.
 The old module is subject to normal garbage collection.
 This mirrors the behavior of pure-Python modules.
@@ -83,36 +186,34 @@ A module may also be limited to the main interpreter using
 the :c:data:`Py_mod_multiple_interpreters` slot.
 
 
-.. _extension-export-hook:
+.. _extension-pyinit:
 
-Initialization function
-.......................
+``PyInit`` function
+...................
 
-The initialization function defined by an extension module has the
-following signature:
+.. deprecated:: next
+
+   This functionality is :term:`soft deprecated`.
+   It will not get new features, but there are no plans to remove it.
+
+Instead of :c:func:`PyModExport_modulename`, an extension module can define
+an older-style :dfn:`initialization function` with the signature:
 
 .. c:function:: PyObject* PyInit_modulename(void)
 
 Its name should be :samp:`PyInit_{<name>}`, with ``<name>`` replaced by the
 name of the module.
+For non-ASCII module names, use :samp:`PyInitU_{<name>}` instead, with
+``<name>`` encoded in the same way as for the
+:ref:`export hook <extension-export-hook>` (that is, using Punycode
+with underscores).
 
-For modules with ASCII-only names, the function must instead be named
-:samp:`PyInit_{<name>}`, with ``<name>`` replaced by the name of the module.
-When using :ref:`multi-phase-initialization`, non-ASCII module names
-are allowed. In this case, the initialization function name is
-:samp:`PyInitU_{<name>}`, with ``<name>`` encoded using Python's
-*punycode* encoding with hyphens replaced by underscores. In Python:
+If a module exports both :samp:`PyInit_{<name>}` and
+:samp:`PyModExport_{<name>}`, the :samp:`PyInit_{<name>}` function
+is ignored.
 
-.. code-block:: python
-
-    def initfunc_name(name):
-        try:
-            suffix = b'_' + name.encode('ascii')
-        except UnicodeEncodeError:
-            suffix = b'U_' + name.encode('punycode').replace(b'-', b'_')
-        return b'PyInit' + suffix
-
-It is recommended to define the initialization function using a helper macro:
+Like with :c:macro:`PyMODEXPORT_FUNC`, it is recommended to define the
+initialization function using a helper macro:
 
 .. c:macro:: PyMODINIT_FUNC
 
@@ -123,51 +224,24 @@ It is recommended to define the initialization function using a helper macro:
    * adds any special linkage declarations required by the platform, and
    * for C++, declares the function as ``extern "C"``.
 
-For example, a module called ``spam`` would be defined like this::
-
-   static struct PyModuleDef spam_module = {
-       .m_base = PyModuleDef_HEAD_INIT,
-       .m_name = "spam",
-       ...
-   };
-
-   PyMODINIT_FUNC
-   PyInit_spam(void)
-   {
-       return PyModuleDef_Init(&spam_module);
-   }
-
-It is possible to export multiple modules from a single shared library by
-defining multiple initialization functions. However, importing them requires
-using symbolic links or a custom importer, because by default only the
-function corresponding to the filename is found.
-See the `Multiple modules in one library <https://peps.python.org/pep-0489/#multiple-modules-in-one-library>`__
-section in :pep:`489` for details.
-
-The initialization function is typically the only non-\ ``static``
-item defined in the module's C source.
 
+Normally, the initialization function (``PyInit_modulename``) returns
+a :c:type:`PyModuleDef` instance with non-``NULL``
+:c:member:`~PyModuleDef.m_slots`. This allows Python to use
+:ref:`multi-phase initialization <multi-phase-initialization>`.
 
-.. _multi-phase-initialization:
-
-Multi-phase initialization
-..........................
-
-Normally, the :ref:`initialization function <extension-export-hook>`
-(``PyInit_modulename``) returns a :c:type:`PyModuleDef` instance with
-non-``NULL`` :c:member:`~PyModuleDef.m_slots`.
 Before it is returned, the ``PyModuleDef`` instance must be initialized
 using the following function:
 
-
 .. c:function:: PyObject* PyModuleDef_Init(PyModuleDef *def)
 
    Ensure a module definition is a properly initialized Python object that
    correctly reports its type and a reference count.
 
    Return *def* cast to ``PyObject*``, or ``NULL`` if an error occurred.
 
-   Calling this function is required for :ref:`multi-phase-initialization`.
+   Calling this function is required before returning a :c:type:`PyModuleDef`
+   from a module initialization function.
    It should not be used in other contexts.
 
    Note that Python assumes that ``PyModuleDef`` structures are statically
@@ -178,18 +252,37 @@ using the following function:
    .. versionadded:: 3.5
 
 
+For example, a module called ``spam`` would be defined like this::
+
+   static struct PyModuleDef spam_module = {
+       .m_base = PyModuleDef_HEAD_INIT,
+       .m_name = "spam",
+       ...
+   };
+
+   PyMODINIT_FUNC
+   PyInit_spam(void)
+   {
+       return PyModuleDef_Init(&spam_module);
+   }
+
+
 .. _single-phase-initialization:
 
 Legacy single-phase initialization
-..................................
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-.. attention::
-   Single-phase initialization is a legacy mechanism to initialize extension
+.. deprecated:: next
+
+   Single-phase initialization is :term:`soft deprecated`.
+   It is a legacy mechanism to initialize extension
    modules, with known drawbacks and design flaws. Extension module authors
    are encouraged to use multi-phase initialization instead.
 
-In single-phase initialization, the
-:ref:`initialization function <extension-export-hook>` (``PyInit_modulename``)
+   However, there are no plans to remove support for it.
+
+In single-phase initialization, the old-style
+:ref:`initializaton function <extension-pyinit>` (``PyInit_modulename``)
 should create, populate and return a module object.
 This is typically done using :c:func:`PyModule_Create` and functions like
 :c:func:`PyModule_AddObjectRef`.
@@ -242,6 +335,8 @@ in the following ways:
 * Single-phase modules support module lookup functions like
   :c:func:`PyState_FindModule`.
 
+* The module's :c:member:`PyModuleDef.m_slots` must be NULL.
+
 .. [#testsinglephase] ``_testsinglephase`` is an internal module used
    in CPython's self-test suite; your installation may or may not
    include it.

Doc/c-api/import.rst

@@ -353,7 +353,7 @@ Importing Modules
    the same as for :c:func:`PyImport_AppendInittab`.
 
    On success, create and return a module object.
-   This module will not be initialized; call :c:func:`!PyModule_Exec`
+   This module will not be initialized; call :c:func:`PyModule_Exec`
    to initialize it.
    (Custom importers should do this in their
    :py:meth:`~importlib.abc.Loader.exec_module` method.)

Doc/c-api/init.rst

@@ -1717,7 +1717,8 @@ function. You can create and destroy them using the following functions:
       Only C-level static and global variables are shared between these
       module objects.
 
-   *  For modules using single-phase initialization,
+   *  For modules using legacy
+      :ref:`single-phase initialization <single-phase-initialization>`,
       e.g. :c:func:`PyModule_Create`, the first time a particular extension
       is imported, it is initialized normally, and a (shallow) copy of its
       module's dictionary is squirreled away.