Skip to content

feat(toxgen): Generate TESTPATH for integrated test suites #4863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 25, 2025
Merged

feat(toxgen): Generate TESTPATH for integrated test suites #4863

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
544a9ff
feat(toxgen): Generate TESTPATHs for integrated test suites
sentrivana Sep 25, 2025
535e291
.
sentrivana Sep 25, 2025
59f2893
Merge branch 'master' into ivana/toxgen/testpath
sentrivana Sep 25, 2025
e1a547c
.
sentrivana Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions scripts/populate_tox/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,5 +207,4 @@ picked (this doesn't count towards `num_versions`).
"Defining constraints" section for the format.
3. Add the integration to one of the groups in the `GROUPS` dictionary in
`scripts/split_tox_gh_actions/split_tox_gh_actions.py`.
4. Add the `TESTPATH` for the test suite in `tox.jinja`'s `setenv` section.
5. Run `scripts/generate-test-files.sh` and commit the changes.
4. Run `scripts/generate-test-files.sh` and commit the changes.
18 changes: 17 additions & 1 deletion scripts/populate_tox/populate_tox.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -513,7 +513,11 @@ def _render_dependencies(integration: str, releases: list[Version]) -> list[str]
def write_tox_file(packages: dict) -> None:
template = ENV.get_template("tox.jinja")

context = {"groups": {}}
context = {
"groups": {},
"testpaths": [],
}

for group, integrations in packages.items():
context["groups"][group] = []
for integration in integrations:
Expand All @@ -528,6 +532,14 @@ def write_tox_file(packages: dict) -> None:
),
}
)
context["testpaths"].append(
(
integration["name"],
f"tests/integrations/{integration['integration_name']}",
)
)

context["testpaths"].sort()

rendered = template.render(context)

Expand Down Expand Up @@ -759,6 +771,10 @@ def main(fail_on_changes: bool = False) -> None:
"package": package,
"extra": extra,
"releases": test_releases,
"integration_name": TEST_SUITE_CONFIG[integration].get(
"integration_name"
)
or integration,
}
)

Expand Down
2 changes: 1 addition & 1 deletion scripts/populate_tox/releases.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
{"info": {"classifiers": ["Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Programming Language :: Python", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7"], "name": "boto3", "requires_python": "", "version": "1.12.49", "yanked": false}}
{"info": {"classifiers": ["Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9"], "name": "boto3", "requires_python": ">= 3.6", "version": "1.20.54", "yanked": false}}
{"info": {"classifiers": ["Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9"], "name": "boto3", "requires_python": ">= 3.7", "version": "1.28.85", "yanked": false}}
{"info": {"classifiers": ["Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", "Programming Language :: Python :: 3.9"], "name": "boto3", "requires_python": ">=3.9", "version": "1.40.37", "yanked": false}}
{"info": {"classifiers": ["Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", "Programming Language :: Python :: 3.9"], "name": "boto3", "requires_python": ">=3.9", "version": "1.40.38", "yanked": false}}
{"info": {"classifiers": ["Development Status :: 4 - Beta", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 2.5", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.2", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Topic :: Internet :: WWW/HTTP :: Dynamic Content :: CGI Tools/Libraries", "Topic :: Internet :: WWW/HTTP :: HTTP Servers", "Topic :: Internet :: WWW/HTTP :: WSGI", "Topic :: Internet :: WWW/HTTP :: WSGI :: Application", "Topic :: Internet :: WWW/HTTP :: WSGI :: Middleware", "Topic :: Internet :: WWW/HTTP :: WSGI :: Server", "Topic :: Software Development :: Libraries :: Application Frameworks"], "name": "bottle", "requires_python": "", "version": "0.12.25", "yanked": false}}
{"info": {"classifiers": ["Development Status :: 4 - Beta", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Topic :: Internet :: WWW/HTTP :: Dynamic Content :: CGI Tools/Libraries", "Topic :: Internet :: WWW/HTTP :: HTTP Servers", "Topic :: Internet :: WWW/HTTP :: WSGI", "Topic :: Internet :: WWW/HTTP :: WSGI :: Application", "Topic :: Internet :: WWW/HTTP :: WSGI :: Middleware", "Topic :: Internet :: WWW/HTTP :: WSGI :: Server", "Topic :: Software Development :: Libraries :: Application Frameworks"], "name": "bottle", "requires_python": null, "version": "0.13.4", "yanked": false}}
{"info": {"classifiers": ["Development Status :: 5 - Production/Stable", "License :: OSI Approved :: BSD License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Topic :: Software Development :: Object Brokering", "Topic :: System :: Distributed Computing"], "name": "celery", "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*", "version": "4.4.7", "yanked": false}}
Expand Down
57 changes: 4 additions & 53 deletions scripts/populate_tox/tox.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,67 +138,18 @@ setenv =

common: TESTPATH=tests
gevent: TESTPATH=tests
aiohttp: TESTPATH=tests/integrations/aiohttp
anthropic: TESTPATH=tests/integrations/anthropic
ariadne: TESTPATH=tests/integrations/ariadne
arq: TESTPATH=tests/integrations/arq
asgi: TESTPATH=tests/integrations/asgi
asyncpg: TESTPATH=tests/integrations/asyncpg
aws_lambda: TESTPATH=tests/integrations/aws_lambda
beam: TESTPATH=tests/integrations/beam
boto3: TESTPATH=tests/integrations/boto3
bottle: TESTPATH=tests/integrations/bottle
celery: TESTPATH=tests/integrations/celery
chalice: TESTPATH=tests/integrations/chalice
clickhouse_driver: TESTPATH=tests/integrations/clickhouse_driver
cohere: TESTPATH=tests/integrations/cohere
cloud_resource_context: TESTPATH=tests/integrations/cloud_resource_context
django: TESTPATH=tests/integrations/django
dramatiq: TESTPATH=tests/integrations/dramatiq
falcon: TESTPATH=tests/integrations/falcon
fastapi: TESTPATH=tests/integrations/fastapi
flask: TESTPATH=tests/integrations/flask
gcp: TESTPATH=tests/integrations/gcp
gql: TESTPATH=tests/integrations/gql
graphene: TESTPATH=tests/integrations/graphene
grpc: TESTPATH=tests/integrations/grpc
httpx: TESTPATH=tests/integrations/httpx
huey: TESTPATH=tests/integrations/huey
huggingface_hub: TESTPATH=tests/integrations/huggingface_hub
langchain-base: TESTPATH=tests/integrations/langchain
langchain-notiktoken: TESTPATH=tests/integrations/langchain
langgraph: TESTPATH=tests/integrations/langgraph
launchdarkly: TESTPATH=tests/integrations/launchdarkly
litestar: TESTPATH=tests/integrations/litestar
loguru: TESTPATH=tests/integrations/loguru
openai-base: TESTPATH=tests/integrations/openai
openai-notiktoken: TESTPATH=tests/integrations/openai
openai_agents: TESTPATH=tests/integrations/openai_agents
openfeature: TESTPATH=tests/integrations/openfeature
opentelemetry: TESTPATH=tests/integrations/opentelemetry
potel: TESTPATH=tests/integrations/opentelemetry
pure_eval: TESTPATH=tests/integrations/pure_eval
pymongo: TESTPATH=tests/integrations/pymongo
pyramid: TESTPATH=tests/integrations/pyramid
quart: TESTPATH=tests/integrations/quart
ray: TESTPATH=tests/integrations/ray
redis: TESTPATH=tests/integrations/redis
redis_py_cluster_legacy: TESTPATH=tests/integrations/redis_py_cluster_legacy
requests: TESTPATH=tests/integrations/requests
rq: TESTPATH=tests/integrations/rq
sanic: TESTPATH=tests/integrations/sanic
spark: TESTPATH=tests/integrations/spark
sqlalchemy: TESTPATH=tests/integrations/sqlalchemy
starlette: TESTPATH=tests/integrations/starlette
starlite: TESTPATH=tests/integrations/starlite
statsig: TESTPATH=tests/integrations/statsig
strawberry: TESTPATH=tests/integrations/strawberry
tornado: TESTPATH=tests/integrations/tornado
trytond: TESTPATH=tests/integrations/trytond
typer: TESTPATH=tests/integrations/typer
unleash: TESTPATH=tests/integrations/unleash
socket: TESTPATH=tests/integrations/socket

{% for integration, testpath in testpaths %}
{{ integration }}: TESTPATH={{ testpath }}
{% endfor %}

passenv =
SENTRY_PYTHON_TEST_POSTGRES_HOST
SENTRY_PYTHON_TEST_POSTGRES_USER
Expand Down
23 changes: 12 additions & 11 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ envlist =
{py3.6,py3.7}-boto3-v1.12.49
{py3.6,py3.9,py3.10}-boto3-v1.20.54
{py3.7,py3.11,py3.12}-boto3-v1.28.85
{py3.9,py3.12,py3.13}-boto3-v1.40.37
{py3.9,py3.12,py3.13}-boto3-v1.40.38

{py3.6,py3.7,py3.8}-chalice-v1.16.0
{py3.9,py3.12,py3.13}-chalice-v1.32.0
Expand Down Expand Up @@ -396,7 +396,7 @@ deps =
boto3-v1.12.49: boto3==1.12.49
boto3-v1.20.54: boto3==1.20.54
boto3-v1.28.85: boto3==1.28.85
boto3-v1.40.37: boto3==1.40.37
boto3-v1.40.38: boto3==1.40.38
{py3.7,py3.8}-boto3: urllib3<2.0.0

chalice-v1.16.0: chalice==1.16.0
Expand Down Expand Up @@ -715,27 +715,31 @@ setenv =

common: TESTPATH=tests
gevent: TESTPATH=tests
asgi: TESTPATH=tests/integrations/asgi
aws_lambda: TESTPATH=tests/integrations/aws_lambda
cloud_resource_context: TESTPATH=tests/integrations/cloud_resource_context
gcp: TESTPATH=tests/integrations/gcp
opentelemetry: TESTPATH=tests/integrations/opentelemetry
potel: TESTPATH=tests/integrations/opentelemetry
socket: TESTPATH=tests/integrations/socket
Comment on lines +719 to +725
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

I am wondering if we can auto-generate the TESTPATH for all integrations? The ones I have highlighted are still explicitly specified in tox.ninja.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those are hardcoded since those are tests suites that are not managed by toxgen. But agree that it looks a bit strange. Added at least a small comment e1a547c to clarify the different treatment for future reference.


aiohttp: TESTPATH=tests/integrations/aiohttp
anthropic: TESTPATH=tests/integrations/anthropic
ariadne: TESTPATH=tests/integrations/ariadne
arq: TESTPATH=tests/integrations/arq
asgi: TESTPATH=tests/integrations/asgi
asyncpg: TESTPATH=tests/integrations/asyncpg
aws_lambda: TESTPATH=tests/integrations/aws_lambda
beam: TESTPATH=tests/integrations/beam
boto3: TESTPATH=tests/integrations/boto3
bottle: TESTPATH=tests/integrations/bottle
celery: TESTPATH=tests/integrations/celery
chalice: TESTPATH=tests/integrations/chalice
clickhouse_driver: TESTPATH=tests/integrations/clickhouse_driver
cohere: TESTPATH=tests/integrations/cohere
cloud_resource_context: TESTPATH=tests/integrations/cloud_resource_context
django: TESTPATH=tests/integrations/django
dramatiq: TESTPATH=tests/integrations/dramatiq
falcon: TESTPATH=tests/integrations/falcon
fastapi: TESTPATH=tests/integrations/fastapi
fastapi: TESTPATH=tests/integrations/fastapi
flask: TESTPATH=tests/integrations/flask
gcp: TESTPATH=tests/integrations/gcp
gql: TESTPATH=tests/integrations/gql
graphene: TESTPATH=tests/integrations/graphene
grpc: TESTPATH=tests/integrations/grpc
Expand All @@ -744,16 +748,14 @@ setenv =
huggingface_hub: TESTPATH=tests/integrations/huggingface_hub
langchain-base: TESTPATH=tests/integrations/langchain
langchain-notiktoken: TESTPATH=tests/integrations/langchain
langgraph: TESTPATH=tests/integrations/langgraph
langgraph: TESTPATH=tests/integrations/langgraph
launchdarkly: TESTPATH=tests/integrations/launchdarkly
litestar: TESTPATH=tests/integrations/litestar
loguru: TESTPATH=tests/integrations/loguru
openai-base: TESTPATH=tests/integrations/openai
openai-notiktoken: TESTPATH=tests/integrations/openai
openai_agents: TESTPATH=tests/integrations/openai_agents
openfeature: TESTPATH=tests/integrations/openfeature
opentelemetry: TESTPATH=tests/integrations/opentelemetry
potel: TESTPATH=tests/integrations/opentelemetry
pure_eval: TESTPATH=tests/integrations/pure_eval
pymongo: TESTPATH=tests/integrations/pymongo
pyramid: TESTPATH=tests/integrations/pyramid
Expand All @@ -774,7 +776,6 @@ setenv =
trytond: TESTPATH=tests/integrations/trytond
typer: TESTPATH=tests/integrations/typer
unleash: TESTPATH=tests/integrations/unleash
socket: TESTPATH=tests/integrations/socket

passenv =
SENTRY_PYTHON_TEST_POSTGRES_HOST
Expand Down