-
Notifications
You must be signed in to change notification settings - Fork 30
feat: Add metadata models package with dynamic schema download #807
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 5 commits
ff01ea0
63930c6
62902f6
3de3af0
933d478
c89faab
6980060
a56208b
0f48425
07d7014
c63223a
da4371f
5373480
7e4e3f4
015a60e
fe4b9cc
66d4eeb
23837eb
c686574
ba912fe
3c2a4f8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,16 @@ | ||
| # Copyright (c) 2025 Airbyte, Inc., all rights reserved. | ||
| """Models used for standard tests.""" | ||
|
|
||
| from airbyte_cdk.test.models.connector_metadata import ( | ||
| ConnectorMetadataDefinitionV0, | ||
| ConnectorTestSuiteOptions, | ||
| ) | ||
| from airbyte_cdk.test.models.outcome import ExpectedOutcome | ||
| from airbyte_cdk.test.models.scenario import ConnectorTestScenario | ||
|
|
||
| __all__ = [ | ||
| "ConnectorMetadataDefinitionV0", | ||
| "ConnectorTestScenario", | ||
| "ConnectorTestSuiteOptions", | ||
| "ExpectedOutcome", | ||
| ] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| # Airbyte Connector Metadata Models | ||
|
|
||
| This package contains Pydantic models for validating Airbyte connector `metadata.yaml` files. | ||
|
|
||
| ## Overview | ||
|
|
||
| The models are automatically generated from JSON Schema YAML files maintained in the [airbytehq/airbyte](https://github.com/airbytehq/airbyte) repository at: | ||
| ``` | ||
| airbyte-ci/connectors/metadata_service/lib/metadata_service/models/src/ | ||
| ``` | ||
|
||
|
|
||
| During the CDK build process (`poetry run poe build`), these schemas are downloaded from GitHub and used to generate Pydantic models via `datamodel-code-generator`. All models are generated into a single Python file for simplicity and easier imports. | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Validating a metadata.yaml file | ||
|
|
||
| ```python | ||
| from pathlib import Path | ||
| import yaml | ||
| from airbyte_cdk.test.models import ConnectorMetadataDefinitionV0 | ||
|
|
||
| # Load metadata.yaml | ||
| metadata_path = Path("path/to/metadata.yaml") | ||
| metadata_dict = yaml.safe_load(metadata_path.read_text()) | ||
|
|
||
| # Validate using Pydantic | ||
| try: | ||
| metadata = ConnectorMetadataDefinitionV0(**metadata_dict) | ||
| print("✓ Metadata is valid!") | ||
| except Exception as e: | ||
| print(f"✗ Validation failed: {e}") | ||
| ``` | ||
|
|
||
| ### Accessing metadata fields | ||
|
|
||
| ```python | ||
| from airbyte_cdk.test.models import ConnectorMetadataDefinitionV0 | ||
|
|
||
| metadata = ConnectorMetadataDefinitionV0(**metadata_dict) | ||
|
|
||
| # Access fields with full type safety | ||
| print(f"Connector: {metadata.data.name}") | ||
| print(f"Docker repository: {metadata.data.dockerRepository}") | ||
| print(f"Docker image tag: {metadata.data.dockerImageTag}") | ||
| print(f"Support level: {metadata.data.supportLevel}") | ||
| ``` | ||
|
|
||
| ### Accessing other models | ||
|
|
||
| All generated models are available in the `generated.models` module: | ||
|
|
||
| ```python | ||
| from airbyte_cdk.test.models.connector_metadata.generated.models import ( | ||
| ConnectorBreakingChanges, | ||
| ConnectorReleases, | ||
| ReleaseStage, | ||
| SupportLevel, | ||
| ) | ||
| ``` | ||
|
|
||
| ### Available models | ||
|
|
||
| The main model is `ConnectorMetadataDefinitionV0`, which includes nested models for: | ||
|
|
||
| - `ConnectorType` - Source or destination | ||
| - `ConnectorSubtype` - API, database, file, etc. | ||
| - `SupportLevel` - Community, certified, etc. | ||
| - `ReleaseStage` - Alpha, beta, generally_available | ||
| - `ConnectorBreakingChanges` - Breaking change definitions | ||
| - `ConnectorReleases` - Release information | ||
| - `AllowedHosts` - Network access configuration | ||
| - And many more... | ||
|
|
||
| ## Regenerating Models | ||
|
|
||
| Models are regenerated automatically when you run: | ||
|
|
||
| ```bash | ||
| poetry run poe build | ||
| ``` | ||
|
|
||
| This command: | ||
| 1. Downloads the latest schema YAML files from the airbyte repository | ||
| 2. Generates all Pydantic models into a single file using `datamodel-code-generator` | ||
| 3. Generates a consolidated JSON schema file for external validation tools | ||
| 4. Outputs to `airbyte_cdk/test/models/connector_metadata/generated/`: | ||
| - `models.py` - All Pydantic models in a single file | ||
| - `metadata_schema.json` - Consolidated JSON schema | ||
|
|
||
| ## Schema Source | ||
|
|
||
| The authoritative schemas are maintained in the [airbyte monorepo](https://github.com/airbytehq/airbyte/tree/master/airbyte-ci/connectors/metadata_service/lib/metadata_service/models/src). | ||
|
|
||
| Any changes to metadata validation should be made there, and will be automatically picked up by the CDK build process. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| from .generated.models import ConnectorMetadataDefinitionV0, ConnectorTestSuiteOptions | ||
aaronsteers marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| __all__ = [ | ||
| "ConnectorMetadataDefinitionV0", | ||
| "ConnectorTestSuiteOptions", | ||
| ] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
|
|
Uh oh!
There was an error while loading. Please reload this page.