Skip to content
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
ff01ea0
feat: Add metadata models package with dynamic schema download
devin-ai-integration[bot] Oct 21, 2025
63930c6
refactor: Move metadata models to airbyte_cdk.test.models.connector_m…
devin-ai-integration[bot] Oct 21, 2025
62902f6
refactor: Move models to generated subdirectory and add convenience i…
devin-ai-integration[bot] Oct 21, 2025
3de3af0
refactor: Generate metadata models as single file with JSON schema ou…
devin-ai-integration[bot] Oct 21, 2025
933d478
style: Apply ruff formatting to build script
devin-ai-integration[bot] Oct 21, 2025
c89faab
docs: Move metadata models documentation to CONTRIBUTING.md
devin-ai-integration[bot] Oct 27, 2025
6980060
Merge branch 'main' into devin/1760999875-add-metadata-models
aaronsteers Oct 27, 2025
a56208b
chore: revert unrelated format changes on other generated file
aaronsteers Oct 27, 2025
0f48425
Delete airbyte_cdk/test/models/connector_metadata/README.md
aaronsteers Oct 27, 2025
07d7014
docs: clean up docstring (merged content from `README.md`)
aaronsteers Oct 27, 2025
c63223a
feat: Replace HTTP downloads with sparse git clone for metadata schemas
devin-ai-integration[bot] Oct 27, 2025
da4371f
Revert accidental formatting of generated file declarative_component_…
devin-ai-integration[bot] Oct 27, 2025
5373480
Add exclusions for auto-generated files in ruff and pre-commit configs
devin-ai-integration[bot] Oct 27, 2025
7e4e3f4
Fix JSON schema consolidation to properly resolve references
devin-ai-integration[bot] Oct 27, 2025
015a60e
Remove $schema and $id from definitions to fix IDE validation
devin-ai-integration[bot] Oct 27, 2025
fe4b9cc
Refactor: Extract metadata generation into separate script
devin-ai-integration[bot] Oct 27, 2025
66d4eeb
Move metadata generation to poe tasks instead of shell script
devin-ai-integration[bot] Oct 27, 2025
23837eb
Replace Dagger with uvx in metadata generation script
devin-ai-integration[bot] Oct 27, 2025
c686574
Simplify metadata generation: generate Python from JSON schema
devin-ai-integration[bot] Oct 27, 2025
ba912fe
Fix schema consolidation per CodeRabbit feedback
devin-ai-integration[bot] Oct 27, 2025
3c2a4f8
Add type annotations and fix formatting
devin-ai-integration[bot] Oct 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 86 additions & 58 deletions airbyte_cdk/sources/declarative/models/declarative_component_schema.py

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions airbyte_cdk/test/models/__init__.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,16 @@
# Copyright (c) 2025 Airbyte, Inc., all rights reserved.
"""Models used for standard tests."""

from airbyte_cdk.test.models.connector_metadata import (
ConnectorMetadataDefinitionV0,
ConnectorTestSuiteOptions,
)
from airbyte_cdk.test.models.outcome import ExpectedOutcome
from airbyte_cdk.test.models.scenario import ConnectorTestScenario

__all__ = [
"ConnectorMetadataDefinitionV0",
"ConnectorTestScenario",
"ConnectorTestSuiteOptions",
"ExpectedOutcome",
]
95 changes: 95 additions & 0 deletions airbyte_cdk/test/models/connector_metadata/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Airbyte Connector Metadata Models

This package contains Pydantic models for validating Airbyte connector `metadata.yaml` files.

## Overview

The models are automatically generated from JSON Schema YAML files maintained in the [airbytehq/airbyte](https://github.com/airbytehq/airbyte) repository at:
```
airbyte-ci/connectors/metadata_service/lib/metadata_service/models/src/
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Add language specifier to fenced code block, wdyt?

The fenced code block showing the repository path should have a language specifier for proper syntax highlighting and markdown lint compliance.

Apply this diff:

-```
+```text
 airbyte-ci/connectors/metadata_service/lib/metadata_service/models/src/

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 markdownlint-cli2 (0.18.1)</summary>

8-8: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

</details>

</details>

<details>
<summary>🤖 Prompt for AI Agents</summary>

In airbyte_cdk/metadata_models/README.md around lines 8 to 10, the fenced code
block lacks a language specifier; update the opening fence from totext
so the block becomes a plain-text fenced code block (i.e., replace the existing
fence with ```text and keep the content and closing fence unchanged).


</details>

<!-- This is an auto-generated comment by CodeRabbit -->


During the CDK build process (`poetry run poe build`), these schemas are downloaded from GitHub and used to generate Pydantic models via `datamodel-code-generator`. All models are generated into a single Python file for simplicity and easier imports.

## Usage

### Validating a metadata.yaml file

```python
from pathlib import Path
import yaml
from airbyte_cdk.test.models import ConnectorMetadataDefinitionV0

# Load metadata.yaml
metadata_path = Path("path/to/metadata.yaml")
metadata_dict = yaml.safe_load(metadata_path.read_text())

# Validate using Pydantic
try:
metadata = ConnectorMetadataDefinitionV0(**metadata_dict)
print("✓ Metadata is valid!")
except Exception as e:
print(f"✗ Validation failed: {e}")
```

### Accessing metadata fields

```python
from airbyte_cdk.test.models import ConnectorMetadataDefinitionV0

metadata = ConnectorMetadataDefinitionV0(**metadata_dict)

# Access fields with full type safety
print(f"Connector: {metadata.data.name}")
print(f"Docker repository: {metadata.data.dockerRepository}")
print(f"Docker image tag: {metadata.data.dockerImageTag}")
print(f"Support level: {metadata.data.supportLevel}")
```

### Accessing other models

All generated models are available in the `generated.models` module:

```python
from airbyte_cdk.test.models.connector_metadata.generated.models import (
ConnectorBreakingChanges,
ConnectorReleases,
ReleaseStage,
SupportLevel,
)
```

### Available models

The main model is `ConnectorMetadataDefinitionV0`, which includes nested models for:

- `ConnectorType` - Source or destination
- `ConnectorSubtype` - API, database, file, etc.
- `SupportLevel` - Community, certified, etc.
- `ReleaseStage` - Alpha, beta, generally_available
- `ConnectorBreakingChanges` - Breaking change definitions
- `ConnectorReleases` - Release information
- `AllowedHosts` - Network access configuration
- And many more...

## Regenerating Models

Models are regenerated automatically when you run:

```bash
poetry run poe build
```

This command:
1. Downloads the latest schema YAML files from the airbyte repository
2. Generates all Pydantic models into a single file using `datamodel-code-generator`
3. Generates a consolidated JSON schema file for external validation tools
4. Outputs to `airbyte_cdk/test/models/connector_metadata/generated/`:
- `models.py` - All Pydantic models in a single file
- `metadata_schema.json` - Consolidated JSON schema

## Schema Source

The authoritative schemas are maintained in the [airbyte monorepo](https://github.com/airbytehq/airbyte/tree/master/airbyte-ci/connectors/metadata_service/lib/metadata_service/models/src).

Any changes to metadata validation should be made there, and will be automatically picked up by the CDK build process.
6 changes: 6 additions & 0 deletions airbyte_cdk/test/models/connector_metadata/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
from .generated.models import ConnectorMetadataDefinitionV0, ConnectorTestSuiteOptions

__all__ = [
"ConnectorMetadataDefinitionV0",
"ConnectorTestSuiteOptions",
]
Empty file.
1 change: 1 addition & 0 deletions airbyte_cdk/test/models/connector_metadata/py.typed
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@

Loading
Loading