Add Tree-Sitter-based Language Module for Python #2548

holtvogt · 2025-08-07T13:55:14Z

JPlag's current Python support relies on ANTLR grammars that struggle with modern Python syntax (3.10+ match statements, 3.11+ exception groups, 3.12+ type aliases) and require maintaining custom grammars that lag behind language evolution.

This PR introduces Tree-sitter as JPlag's new parsing foundation, starting with Python. Tree-sitter provides native parsing performance, community-maintained grammars that stay current with language specs, and cross-platform native library distribution.

Technical Notes

Requires Java JDK 22 for Java's Foreign Function and Memory (FFM) API and jextract to generate Java bindings from Tree-sitter C headers
Depends on Zig build support in Tree-sitter repositories for Windows library compilation

Architecture Changes

New `language-tree-sitter-utils` module

Uses Java's FFM API bindings for Tree-sitter native libraries
Introduces an abstract base class for future Tree-sitter language implementations
Enables cross-platform native library loading

First Tree-sitter Python language module implementation

PythonParser: Orchestrates parsing and delegates to token collector
PythonTokenCollector: AST visitor that maps Tree-sitter nodes to JPlag tokens
TreeSitterPython: Singleton grammar loader using FFM API
Handler-based token extraction with support for nested constructs

Native library build system

GitHub Actions workflow for cross-platform compilation (Linux/macOS/Windows)
Local native library compilation via mvn -Pbuild-native-libraries generate-resources
Supports versioning of native Tree-sitter libraries to enable language modules to run on different Tree-sitter releases

Testing

# Build native libraries locally
mvn -Pbuild-native-libraries generate-resources

# Test new Python module
mvn clean test-compile -pl :python -am
mvn test -pl :python

# Test CLI integration
mvn -Pwith-report-viewer clean package assembly:single
java -jar cli/target/jplag-*-jar-with-dependencies.jar -l python [files]

scripts/README.md

Convert to abstract class with handler maps and template method pattern. Reduces boilerplate and enforces consistent visitor implementations.

The "Adapter" suffix was misleading as these classes are direct parsers, not adapters between incompatible interfaces. Updated all references and documentation to reflect the cleaner, more accurate naming.

Removed redundant token list initialization from PythonTokenCollector and added a centralized token list in TreeSitterVisitor. Introduced a method to retrieve collected tokens.

Updated the language module documentation to clarify core components and added sections for ANTLR and Tree-sitter parsing technologies. Included detailed examples for setting up language modules with Tree-sitter, emphasizing its implementation specifics.

holtvogt · 2025-09-04T07:05:58Z

languages/python/src/main/java/de/jplag/python/PythonTokenCollector.java

+    private boolean isEndToken(PythonTokenType tokenType) {
+        return switch (tokenType) {
+            case CLASS_END, METHOD_END, WHILE_END, FOR_END, TRY_END, EXCEPT_END, FINALLY_END, IF_END, DECORATOR_END, WITH_END, MATCH_END -> true;
+            default -> false;
+        };
+    }


Similar to getLength(), should we move this into PythonTokenType?

IMO, yes, to provide a more object-oriented implementation.

Copilot

Pull Request Overview

This PR adds comprehensive Tree-sitter-based Python language support to JPlag, providing modern Python syntax support (3.10+ match statements, 3.11+ exception groups, 3.12+ type aliases) through native parsing performance and community-maintained grammars.

Implements new language-tree-sitter-utils module with abstract base classes and native library management
Adds Tree-sitter Python language module with token extraction for all Python constructs
Updates Java compiler target from JDK 21 to JDK 22 for Foreign Function and Memory API support

Reviewed Changes

Copilot reviewed 52 out of 53 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`scripts/build-native-libraries.sh`	Cross-platform native library build script for Tree-sitter grammars
`pom.xml`	Updates Java version to 22 and adds Tree-sitter dependencies and build profile
`language-tree-sitter-utils/`	New module with abstract base classes for Tree-sitter language implementations
`languages/python/`	Complete Tree-sitter Python implementation with parser, token collector, and tests
CI workflows	Updates Java version and adds native library build steps

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-15T06:56:47Z

scripts/build-native-libraries.sh

+        "windows")
+            # Zig puts output in zig-out/lib/ directory and names it $library_name.dll
+            if [ -f "zig-out/bin/$library_name.dll" ]; then
+                # Hence we need to rename it to $LIBRARY_NAME.dll


The comment on line 175 doesn't match the code logic. The comment says "Hence we need to rename it" but the variable names suggest the opposite - $library_name (lowercase with hyphens) is being renamed to $LIBRARY_NAME (with lib prefix). Update the comment to accurately describe the renaming from the build output name to the expected library name format.

Suggested change

# Hence we need to rename it to $LIBRARY_NAME.dll

# Rename the build output ($library_name.dll) to the expected format ($LIBRARY_NAME.dll)

Copilot · 2025-09-15T06:56:48Z

report-viewer/src/utils/CodeHighlighter.ts

+import { ParserLanguage, type Language } from '@/model/Language'
 import hljs from 'highlight.js'
-import scheme from 'highlight.js/lib/languages/scheme'
 import llvm from 'highlight.js/lib/languages/llvm'
+import scheme from 'highlight.js/lib/languages/scheme'
 import typescript from 'highlight.js/lib/languages/typescript'


[nitpick] The imports are not consistently ordered. Consider organizing imports alphabetically within each group (external libraries first, then local imports) for better maintainability. Move the hljs import before the specific language imports.

Copilot · 2025-09-15T06:56:48Z

report-viewer/src/model/Language.ts

@@ -43,4 +44,4 @@ function getLanguageParser(language: string): Language {
  return 'unknown language'
 }

-export { ParserLanguage, type Language, getLanguageParser }
+export { ParserLanguage, getLanguageParser, type Language }


[nitpick] The export order change appears inconsistent with the import order change in CodeHighlighter.ts. Consider maintaining a consistent ordering pattern across the codebase - either alphabetical or by type (functions before types).

Suggested change

export { ParserLanguage, getLanguageParser, type Language }

export { getLanguageParser, ParserLanguage, type Language }

Copilot · 2025-09-15T06:56:48Z

languages/java/src/test/java/de/jplag/java/JavaLanguageTest.java

+        // StringTemplate was introduced in Java 21, but removed in Java 22
+        // collector.testFile("StringConcat.java", "StringTemplate.java").testSourceCoverage().testTokenSequence(J_CLASS_BEGIN,
+        // J_METHOD_BEGIN, J_VARDEF,
+        // J_VARDEF, J_VARDEF, J_APPLY, J_METHOD_END, J_CLASS_END);


Instead of commenting out the test, consider using @Disabled annotation with a reason or conditional test execution based on Java version. This provides better test documentation and maintains the test structure for potential future re-enablement.

Copilot · 2025-09-15T06:56:48Z

pom.xml

+            <!-- Tree-sitter -->
            <dependency>
-                <groupId>net.sourceforge.argparse4j</groupId>
-                <artifactId>argparse4j</artifactId>
-                <version>0.9.0</version>
+                <groupId>io.github.tree-sitter</groupId>
+                <artifactId>jtreesitter</artifactId>
+                <version>${java-tree-sitter.version}</version>
            </dependency>

            <!-- CoreNLP -->


[nitpick] The dependency management section has inconsistent organization. The Tree-sitter dependency is placed between ANTLR and CoreNLP dependencies, breaking the logical grouping. Consider placing it in a separate Tree-sitter section or moving it to maintain better logical organization of related dependencies.

holtvogt and others added 30 commits May 21, 2025 22:46

Add language-tree-sitter-utils module

8ada11c

Use original tree-sitter dependency

7fb1356

Use JTreeSitter as internal tree-sitter parser

aa1f8f2

Add TreeSitterLanguage abstract class for tree-sitter integration

e76c6e7

Add OS utility class for operating system and architecture detection

c1d3fc7

Add classes for native library loading support in tree-sitter

04315bf

Add service loader configuration for LibraryLookup

12cb33c

Add tree-sitter utils to converage report

82aae4a

Remove test utils and build options for now

5d82e52

Sort dependencies

5ad72c4

Bump to JDK 22

cd68528

Add Tree-sitter-based Python module

a966457

Add colon to method end

2621eb1

Fix Javadoc reference

607afdb

Use template method to provide language specific grammar

928b262

Add Python language class

7516f39

Add Pytho parser adapter

edae33a

Add whitespace for code formatting in TreeSitterPython

5a82810

Add new token types for Python 3.12

65d8fa7

Add binary files for Java and Python TreeSitter language support

1e90d6a

Add Python dependency to CLI module

34094fe

Update Python identifier to return python

d85e2a7

Remove OS magic numbers

ddbda29

Remove symbol name magic number

c38be13

Add libraries for Mac OS

9168ac6

Add visitor interface for Tree-sitter

fef7292

Add utility class for traversing Tree-sitter syntax trees

3cff92b

Implement token extraction in PythonParserAdapter

680224b

Use a constant for symbol name

565345f

Add PythonTokenCollector class for token extraction

6a97879

holtvogt added 11 commits August 30, 2025 20:42

Reorder Windows subjobs

7b7c32e

Use newer Zig GitHub action

d4801e9

Do not use new GitHub Action

8e35ae8

Use 14.1

d047ee9

13.0 seems to work

6912f5f

Fix dict comprehension handler name

2eb2940

Add exception group handler

0d4dd8c

Fix comprehensions test

2f5a8b6

Fix exceptions group test

addc109

Exclude virtual environment files

4ac4198

Fix tokenization for elif and else

3aefeb2

holtvogt commented Aug 30, 2025

View reviewed changes

scripts/README.md Outdated Show resolved Hide resolved

holtvogt added 10 commits September 2, 2025 19:31

Update README to reflect versioned output structure for built libraries

dd4a8dc

Refactor token mappings for regression testing

7fad071

Convert TreeSitterVisitor to abstract class with built-in handlers

7cf95a7

Convert to abstract class with handler maps and template method pattern. Reduces boilerplate and enforces consistent visitor implementations.

Remove static traversal utility

5008d1b

Rename parser classes to remove misleading "Adapter" suffix

ffb09f5

The "Adapter" suffix was misleading as these classes are direct parsers, not adapters between incompatible interfaces. Updated all references and documentation to reflect the cleaner, more accurate naming.

Apply spotless for AbstractTreeSitterParser constructor

66e5989

Map exception group tokens to exceptions

fa7ce74

Refactor token collection in TreeSitterVisitor and PythonTokenCollector

77c2bfd

Removed redundant token list initialization from PythonTokenCollector and added a centralized token list in TreeSitterVisitor. Introduced a method to retrieve collected tokens.

Remove example code from documentation

30c8aab

holtvogt marked this pull request as ready for review September 2, 2025 21:03

holtvogt commented Sep 4, 2025

View reviewed changes

tsaglam added enhancement Issue/PR that involves features, improvements and other changes major Major issue/feature/contribution/change language PR / Issue deals (partly) with new and/or existing languages for JPlag labels Sep 4, 2025

robinmaisch requested a review from a team September 4, 2025 13:52

tsaglam requested a review from Copilot September 15, 2025 06:55

Copilot AI reviewed Sep 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Tree-Sitter-based Language Module for Python #2548

Add Tree-Sitter-based Language Module for Python #2548

Uh oh!

holtvogt commented Aug 7, 2025 •

edited

Loading

Uh oh!

Uh oh!

holtvogt Sep 4, 2025

Uh oh!

robinmaisch Sep 8, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 15, 2025

Uh oh!

Copilot AI Sep 15, 2025

Uh oh!

Copilot AI Sep 15, 2025

Uh oh!

Copilot AI Sep 15, 2025

Uh oh!

Copilot AI Sep 15, 2025

Uh oh!

Uh oh!

	# Hence we need to rename it to $LIBRARY_NAME.dll
	# Rename the build output ($library_name.dll) to the expected format ($LIBRARY_NAME.dll)

	export { ParserLanguage, getLanguageParser, type Language }
	export { getLanguageParser, ParserLanguage, type Language }

Add Tree-Sitter-based Language Module for Python #2548

Are you sure you want to change the base?

Add Tree-Sitter-based Language Module for Python #2548

Uh oh!

Conversation

holtvogt commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Technical Notes

Architecture Changes

New language-tree-sitter-utils module

First Tree-sitter Python language module implementation

Native library build system

Testing

Uh oh!

Uh oh!

holtvogt Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

robinmaisch Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

holtvogt commented Aug 7, 2025 •

edited

Loading

New `language-tree-sitter-utils` module