Add documentation for Tree-sitter-based language modules

Updated the language module documentation to clarify core components and added sections for ANTLR and Tree-sitter parsing technologies. Included detailed examples for setting up language modules with Tree-sitter, emphasizing its implementation specifics.

Author: holtvogt

docs/4.-Adding-New-Languages.md

@@ -63,7 +63,7 @@ If a hard-coded (as opposed to dynamically generated) parser library is availabl
 
 # Language Module Structure
 
-A language module consists of these parts:
+A language module consists of these core components:
 
 | Component/Class                         | Superclass                | Function                                         | How to get there                                            | 
 |-----------------------------------------|---------------------------|--------------------------------------------------|-------------------------------------------------------------|
@@ -73,8 +73,14 @@ A language module consists of these parts:
 | TokenType class                         | `de.jplag.TokenType`      | contains the language-specific token types       | **implement new**                                           |
 |                                         |                           |                                                  |
 | Lexer and Parser                        | -                         | transform code into AST                          | depends on technology                                       |
-| ParserAdapter class                     | `de.jplag.AbstractParser` | sets up Parser and calls Traverser               | depends on technology                                       |
-| Traverser/<br>TraverserListener classes | -                         | creates tokens traversing the AST                | depends on technology                                       |
+| ParserAdapter class                     | varies by technology      | sets up Parser and calls Traverser               | depends on technology                                       |
+| Traverser/<br>TraverserListener classes | varies by technology      | creates tokens traversing the AST                | depends on technology                                       |
+
+## Parser Technology Options
+
+JPlag supports two main parsing technologies for implementing language modules:
+
+### ANTLR-based Language Modules
 
 For example, if ANTLR is used, the setup is as follows:
 
@@ -95,6 +101,28 @@ As the table shows, much of a language module can be reused, especially when usi
   - It should still be rather easy to implement the ParserAdapter from the library documentation.
   - Instead of using a listener pattern, the library may require you to do the token extraction in a _Visitor subclass_. In that case, there is only one method call per element, called e.g. `traverseClassDeclaration`. The advantage of this version is that the traversal of the subtrees can be controlled freely. See the Scala language module for an example.
 
+
+### Tree-sitter-based Language Modules
+
+Tree-sitter provides an alternative parsing approach with native performance and community-maintained grammars. For Tree-sitter-based modules, the setup is as follows:
+
+| Tree-sitter specific parts/files | Superclass/Interface | Function | How to get there |
+|-----------------------------------|---------------------|----------|------------------|
+| Native grammar libraries | - | Platform-specific parsing libraries | Built from Tree-sitter grammar repositories |
+| Tree-sitter Language class | `de.jplag.treesitter.TreeSitterLanguage` | Loads native grammar | **implement new** |
+| Parser class | `de.jplag.treesitter.AbstractTreeSitterParser` | Sets up parser and calls visitor | copy with small adjustments |
+| TokenCollector class | `de.jplag.treesitter.TreeSitterVisitor` | Extracts tokens from AST using visitor pattern | **implement new** |
+
+As with ANTLR modules, much can be reused. The parts to implement specifically for each Tree-sitter language module are:
+ - the Tree-sitter Language class (grammar loader)
+ - the TokenCollector (visitor implementation), and
+ - the TokenTypes.
+
+**Note** about Tree-sitter advantages:
+  - Native parsing performance with robust error recovery
+  - Community-maintained grammars that stay current with language evolution
+  - Cross-platform native library distribution
+
 ## Setting up a new language module with ANTLR
 
 JPlag provides a small framework to make it easier to implement language modules with ANTLR
@@ -218,6 +246,127 @@ Additional features for rules:
 2. Semantics - Can be passed by using withSemantics after the map call (see CPP language module for examples)
 3. Delegate  - To have more precise control over the token position and length a delegated visitor can be used (see Go language module for examples)
 
+## Setting up a new language module with Tree-sitter
+
+JPlag provides a comprehensive framework for implementing Tree-sitter-based language modules with built-in handler infrastructure and native library management.
+
+### Create the Tree-sitter Language class
+
+Create a class that extends `TreeSitterLanguage` to load your language's native grammar:
+
+```java
+public class TreeSitterYourLanguage extends TreeSitterLanguage {
+    private static final TreeSitterYourLanguage INSTANCE = new TreeSitterYourLanguage();
+    private static final String SYMBOL_NAME = "tree_sitter_your_language";
+
+    private TreeSitterYourLanguage() {
+        // Private constructor for singleton pattern
+    }
+
+    public static MemorySegment language() {
+        return INSTANCE.call();
+    }
+
+    @Override
+    protected NativeLibraryType libraryType() {
+        return NativeLibraryType.TREE_SITTER_YOUR_LANGUAGE; // Add to enum first
+    }
+
+    @Override
+    protected String symbolName() {
+        return SYMBOL_NAME;
+    }
+}
+```
+
+### Create the Parser class
+
+Implement a parser that extends `AbstractTreeSitterParser`:
+
+```java
+public class YourLanguageParser extends AbstractTreeSitterParser {
+    @Override
+    protected MemorySegment getLanguageMemorySegment() {
+        return TreeSitterYourLanguage.language();
+    }
+
+    @Override
+    protected List<Token> extractTokens(File file, Node rootNode) {
+        YourLanguageTokenCollector collector = new YourLanguageTokenCollector(file);
+        collector.traverse(rootNode);
+        List<Token> tokens = collector.getTokens();
+        tokens.add(Token.fileEnd(file));
+        return tokens;
+    }
+}
+```
+
+### Implement the TokenCollector
+
+The TokenCollector extends `TreeSitterVisitor` and uses a handler-based approach:
+
+```java
+public class YourLanguageTokenCollector extends TreeSitterVisitor {
+    private final File file;
+
+    public YourLanguageTokenCollector(File file) {
+        this.file = file;
+    }
+
+    @Override
+    protected void initializeHandlers() {
+        // Map Tree-sitter node types to token creation handlers
+        enterHandlers.put("class_definition", node -> addToken(YourLanguageTokenType.CLASS_BEGIN, node));
+        enterHandlers.put("function_definition", node -> addToken(YourLanguageTokenType.METHOD_BEGIN, node));
+        
+        // Add exit handlers for balanced constructs
+        exitHandlers.put("class_definition", node -> addToken(YourLanguageTokenType.CLASS_END, node));
+        exitHandlers.put("function_definition", node -> addToken(YourLanguageTokenType.METHOD_END, node));
+    }
+
+    private void addToken(YourLanguageTokenType tokenType, Node node) {
+        int line = node.getStartPoint().row() + 1;
+        int column = node.getStartPoint().column() + 1;
+        int length = tokenType.getLength();
+        
+        tokens.add(new Token(tokenType, file, line, column, line, column + length, length));
+    }
+}
+```
+
+### Implement the token type enum
+
+This is similar to ANTLR modules, but includes length information for performance:
+
+```java
+public enum YourLanguageTokenType implements TokenType {
+    CLASS_BEGIN("CLASS{", 5), // class keyword
+    CLASS_END("}CLASS", 5), // end of class
+    METHOD_BEGIN("METHOD{", 3), // def/function keyword
+    METHOD_END("}METHOD", 3), // end of method
+    // Add more tokens as needed
+
+    private final String description;
+    private final int length;
+
+    YourLanguageTokenType(String description, int length) {
+        this.description = description;
+        this.length = length;
+    }
+
+    @Override
+    public String getDescription() {
+        return description;
+    }
+
+    public int getLength() {
+        return length;
+    }
+}
+```
+
+The handler-based approach eliminates boilerplate code and provides a clean, declarative way to map Tree-sitter node types to JPlag tokens. The abstract base class handles traversal and token collection automatically.
+
 ## Basic procedure outline
 
 ```mermaid