docs: document the language-aware parsing of source files for traceability

Also, a few minor updates along the way.

Author: stanislaw

docs/_assets/StrictDoc_Architecture.drawio.png

@@ -13,7 +13,7 @@ Summary of StrictDoc features:
 - The documentation files are stored as human-readable text files.
 - A simple domain-specific language DSL is used for writing the documents. The
   text format for encoding this language is called SDoc (strict-doc).
-- StrictDoc reads ``*.sdoc`` files and builds an in-memory representation of a
+- StrictDoc reads \*.sdoc files and builds an in-memory representation of a
   document tree.
 - From this in-memory representation, StrictDoc can generate the documentation
   into a number of formats including HTML, RST, ReqIF, PDF, JSON, Excel.
@@ -28,9 +28,9 @@ Summary of StrictDoc features:
 - Requirements to source files traceability (experimental). See
   :ref:`Traceability between requirements and source code <SECTION-TRACEABILITY-REQS-TO-SOURCE-CODE>`.
 - Custom grammar and custom fields support. The StrictDoc's grammar can be
-  extended to support arbitrary special fields, such as ``PRIORITY``, ``OWNER``,
+  extended to support arbitrary special fields, such as "PRIORITY", "OWNER",
   or even more specialized fields, such as
-  ``Automotive Safety Integrity Level (ASIL)`` or ``Verification method``.
+  "Automotive Safety Integrity Level (ASIL)" or "Verification method".
   See :ref:`Custom grammars <SECTION-CUSTOM-GRAMMARS>`.
 - Good performance of the `textX <https://github.com/textX/textX>`_
   parser and parallelized incremental generation of documents: generation of
@@ -1722,10 +1722,142 @@ Traceability between requirements and source code
 
 **Note:** This feature is experimental, the documentation is incomplete.
 
-StrictDoc allows connecting requirements to source code files. Two types of
-links are supported:
+StrictDoc allows connecting requirements to source code files in two ways:
 
-1\) A basic link where a requirement links to a whole file.
+1. Linking source files to requirements by adding special markers in the source code without modifying the requirements.
+2. Linking requirements to source files by adding relations in the requirements without altering the source code.
+
+The advantage of the first approach is that requirements remain agnostic to the source code and implementation details. Special markers are added solely to the source files, creating traceability back to the requirements.
+
+The benefit of the second approach is that traceability to source files is established without any modification to the source code. This can be useful in projects where adding markers to the source code is undesirable or not feasible.
+
+Both options can be used independently or in combination, depending on the project setup and allocation of software components.
+
+To activate the traceability to source files, configure the project config with a dedicated feature:
+
+.. code-block:: yaml
+
+    [project]
+
+    features = [
+      "REQUIREMENT_TO_SOURCE_TRACEABILITY"
+    ]
+
+By default, StrictDoc looks for source files in a directory from which the ``strictdoc`` command is run. This can be changed by using the ``source_root_path`` project-level option.
+
+See :ref:`Project-level options <SDOC_UG_OPTIONS_PROJECT_LEVEL>` for more details about the project-level options.
+
+The
+`strictdoc-examples <https://github.com/strictdoc-project/strictdoc-examples>`_
+repository contains executable examples including the example of
+requirements-to-source-code traceability.
+
+.. _SECTION-UG-Language-aware-parsing-of-source-code:
+
+Language-aware parsing of source code
+-------------------------------------
+
+For parsing source code and calculating traceability to requirements, StrictDoc uses a general parser that is agnostic of specific programming languages and their constructs, such as classes or functions. However, for languages with these constructs, establishing traceability to them can simplify the tracing process.
+
+As an experimental option, StrictDoc supports parsing source files of selected programming languages (currently Python and C) to recognize language syntax, primarily enabling traceability of functions (in Python, C, and others) and classes (in Python, C++, and others) to requirements.
+
+To activate language-aware traceability, configure the project with the following features:
+
+.. code:: toml
+
+    [project]
+
+    features = [
+      "REQUIREMENT_TO_SOURCE_TRACEABILITY",
+      "SOURCE_FILE_LANGUAGE_PARSERS"
+    ]
+
+Currently, only Python and C parsers are implemented. Upcoming implementations include parsers for Rust, C++, Bash, and more.
+
+Linking source code to requirements
+-----------------------------------
+
+To connect a source file to a requirement, a dedicated ``@relation`` marker must be added to the source file. Several marker types are supported, depending on the programming language. For example, the ``scope=class`` option is available for Python files but not for C files, as C does not support classes.
+
+.. note::
+
+    For language-specific parsing of source code, e.g., Python and C, make sure to enable the corresponding option, see :ref:`Language-aware parsing of source code <SECTION-UG-Language-aware-parsing-of-source-code>`.
+
+**1\) Linking a file to a requirement**
+
+The marker must be added to the top comment of a file. Currently supported only in Python.
+
+.. code:: python
+
+    """
+    This file implements ...
+
+    @relation(REQ-1, scope=file)
+    """
+
+**2\) Linking a class to a requirement (Python only)**
+
+.. code:: python
+
+    class Foo:
+        """
+        This class implements ...
+
+        @relation(REQ-1, scope=file)
+        """
+
+**3\) Linking a function to a requirement (Python and C only)**
+
+.. code:: python
+
+    class Foo:
+        def bar(self):
+            """
+            This function implements ...
+
+            @relation(REQ-1, scope=function)
+            """
+
+or
+
+.. code:: c
+
+    /**
+     * Some text.
+     *
+     * @relation(REQ-1, scope=function)
+     */
+    void hello_world(void) {
+        print("Hello, World\n");
+    }
+
+**4\) Linking a range to a requirement**
+
+.. code:: python
+
+    def foo():
+        # @relation(REQ-1, scope=range_start)
+        print("Hello, World!")
+        # @relation(REQ-1, scope=range_end)
+
+**5\) Linking a single line to a requirement**
+
+.. code:: python
+
+    def foo():
+        # @relation(REQ-1, scope=line)
+        print("Hello, World!")
+
+Linking requirements to source code
+-----------------------------------
+
+The linking of requirements to source files is arranged with a special RELATION type ``File``.
+
+.. note::
+
+    For language-specific parsing of source code, e.g., Python and C, make sure to enable the corresponding option, see :ref:`Language-aware parsing of source code <SECTION-UG-Language-aware-parsing-of-source-code>`.
+
+**1\) Linking a requirement to a whole source file**
 
 .. code-block:: text
 
@@ -1737,53 +1869,51 @@ links are supported:
     TITLE: File reference
     STATEMENT: This requirement references the file.
 
-2\) A range-based link where a requirement links to a file and
-additionally in the file, there is a reverse link that connects a source range
-back to the requirement:
-
-The requirement declaration contains a reference of the type ``File``:
+**2\) Linking a requirement to range in a source file**
 
 .. code-block:: text
 
     [REQUIREMENT]
     UID: REQ-002
+    TITLE: Range file reference
+    STATEMENT: This requirement references the file.py file.
     RELATIONS:
     - TYPE: File
       VALUE: file.py
-    TITLE: Range file reference
-    STATEMENT: This requirement references the file.py file.
-    COMMENT: >>>
-    If the file.py contains a source range that is connected back to this
-    requirement (REQ-002), the link becomes a link to the source range.
-    <<<
-
-The source file:
-
-.. code-block:: py
+      LINE_RANGE: 2, 4
 
-    # @sdoc[REQ-002]
-    def hello_world():
-        print("hello world")
-    # @sdoc[/REQ-002]
+**3\) Linking a requirement to a function in a source file**
 
-To activate the traceability to source files, configure the project config with a dedicated feature:
+.. code-block:: text
 
-.. code-block:: yaml
+    [REQUIREMENT]
+    UID: REQ-002
+    TITLE: Function reference
+    STATEMENT: This requirement references a function in a file.
+    RELATIONS:
+    - TYPE: File
+      VALUE: file.py
+      FUNCTION: hello_world
+    - TYPE: File
+      VALUE: file.c
+      FUNCTION: Foo.hello_world_2
 
-    [project]
+.. note::
 
-    features = [
-      "REQUIREMENT_TO_SOURCE_TRACEABILITY"
-    ]
+    For linking to functions in classes, a class name has to be added in a format: ``<class name>.<function name>``. This is currently only supported for Python source files.
 
-By default, StrictDoc looks for source files in a directory from which the ``strictdoc`` command is run. This can be changed by using the ``source_root_path`` project-level option.
+**4\) Linking a requirement to a class in a source file (Python only)**
 
-See :ref:`Project-level options <SDOC_UG_OPTIONS_PROJECT_LEVEL>` for more details about the project-level options.
+.. code-block:: text
 
-The
-`strictdoc-examples <https://github.com/strictdoc-project/strictdoc-examples>`_
-repository contains executable examples including the example of
-requirements-to-source-code traceability.
+    [REQUIREMENT]
+    UID: REQ-002
+    TITLE: Class reference
+    STATEMENT: This requirement references a class in a file.
+    RELATIONS:
+    - TYPE: File
+      VALUE: file.py
+      CLASS: Foo
 
 ReqIF support
 =============

docs/_assets/StrictDoc_Workspace-Architecture.drawio.png

@@ -296,8 +296,6 @@ FRET has an impressive list of
 
 FRET's user interface is built with Electron.
 
-The detailed comparison is coming.
-
 How long has the StrictDoc project been around?
 ===============================================
 
@@ -330,6 +328,8 @@ The custom RST parser was replaced with a TextX-based DSL. Since then, StrictDoc
 
 The FastAPI/Turbo/Stimulus-based Web interface prototype was created to complement the text-based interface with a graphical user interface (GUI). When the Web-based GUI is stable, StrictDoc may become useable by non-programmers too.
 
+See also: :ref:`Project milestones <SECTION-DP-Project-milestones>`.
+
 Which StrictDoc statistics are available?
 =========================================
 

docs/_assets/StrictDoc_Workspace-Roadmap.drawio.png

@@ -29,6 +29,8 @@ An important feature of StrictDoc is its focus on open data, ensuring ease of da
 
 StrictDoc shall be compatible with other software and engineering tools. This includes at least the compatibility with the Python ecosystem, the model-based systems engineering tools, such as Capella, and the formats providing Software Bill of Materials, such as SPDX.
 
+.. _SECTION-DP-Project-milestones:
+
 Project milestones
 ==================
 
@@ -84,7 +86,7 @@ As an open-source project, StrictDoc is developed without strict deadlines, howe
    * - 2024-Q3
      - HTML2PDF improvements. ReqIF roundtrip for RELATION/ROLE. Consistent automatic escaping of Jinja templates. Passthrough->export command migration.
    * - 2024-Q4
-     - TBD
+     - Connecting requirements to functions (C, Python) and classes (Python) in source code. Support the linking in both directions independently: from requirements to source using RELATION/File and from source to requirements using @relation markers.
 
 The roadmap diagram
 -------------------

docs/sphinx/source/_assets/StrictDoc_Workspace-Architecture.drawio.png

@@ -3,6 +3,17 @@ $$$$$$$$$$$$$
 
 This document maintains a record of all changes to StrictDoc since November 2023. It serves as a user-friendly version of the changelog, complementing the automatically generated, commit-by-commit changelog available here: `StrictDoc Changelog <https://github.com/strictdoc-project/strictdoc/blob/main/CHANGELOG.md>`_.
 
+Unreleased 0.1.0 (2024-11-01)
+=============================
+
+This backward-compatible release introduces several new features for tracing requirements to source files:
+
+- StrictDoc now integrates with `Tree-sitter <https://tree-sitter.github.io/tree-sitter/>`_, enabling it to parse multiple programming languages. Using AST information, it achieves more precise tracing of requirements to source code.
+- Language-specific parsers for Python and C have been added, allowing functions (in C and Python) or classes (in Python) to be linked to requirements.
+- Both forward linking of requirements to source files and backward linking of source files to requirements are supported. These features can be used independently or together within the same project.
+
+With this release, we are also transitioning to a more `semantic versioning <https://semver.org>`_-oriented release scheme. From now on, the MAJOR.MINOR.PATCH version components will be maintained according to the recommendations of the semantic versioning specification.
+
 0.0.60 (2024-10-26)
 ===================
 

docs/sphinx/source/_assets/StrictDoc_Workspace-Roadmap.drawio.png

@@ -22,6 +22,7 @@ following steps:
    single contribution.
 4. If the contribution is not trivial, read through the complete development
    guide.
+5. If the contribution is not trivial, please update the Release Notes with a high-level summary of your contribution. Refer to the release notes from previous releases for guidance on what information is desirable.
 
 How can I help?
 ===============