Skip to content

Commit d313bea

Browse files
samrosesamrose
committed
feat: trigger supabase/tests with label before merge
1 parent af4a2ba commit d313bea

File tree

2 files changed

+515
-0
lines changed

2 files changed

+515
-0
lines changed

‎.github/workflows/PR_AMI_TESTING_SETUP.md‎

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,279 @@
1+
# PR AMI Integration Testing Setup
2+
3+
This document describes the setup for running integration tests from the private `supabase/tests` repository against AMIs built from PRs in the `supabase/postgres` repository.
4+
5+
## Overview
6+
7+
The integration test workflow allows you to:
8+
1. Build a test AMI from a PR branch
9+
2. Trigger integration tests in the private `supabase/tests` repo with that AMI
10+
3. Report test results back to the PR as a required status check
11+
4. Gate PR merges on successful integration tests
12+
13+
## Architecture
14+
15+
```
16+
┌─────────────────────────────────────────┐
17+
│ supabase/postgres (PR to develop) │
18+
│ │
19+
│ 1. Label PR with "test-ami" │
20+
│ 2. Build test AMI │
21+
│ 3. Create pending status check │
22+
│ 4. Dispatch event to tests repo ────────┼────┐
23+
└─────────────────────────────────────────┘ │
24+
│
25+
▼
26+
┌───────────────────────────┐
27+
│ supabase/tests (private) │
28+
│ │
29+
│ 5. Run integration tests │
30+
│ 6. Report results back │
31+
│ 7. Clean up AMI │
32+
└───────────────────────────┘
33+
│
34+
▼
35+
┌─────────────────────────────────────────┐
36+
│ GitHub PR Status Checks │
37+
│ │
38+
│ ✅ ami-integration-tests/pg15 │
39+
│ ✅ ami-integration-tests/pg16 │
40+
│ ✅ ami-integration-tests/pg17 │
41+
└─────────────────────────────────────────┘
42+
```
43+
44+
## Required Secrets
45+
46+
### In `supabase/postgres` repository:
47+
48+
1. **`TESTS_REPO_DISPATCH_PAT`** (required)
49+
- A GitHub Personal Access Token with `repo` scope
50+
- Used to trigger workflows in the private `supabase/tests` repo via `repository_dispatch`
51+
- Token should be created by a user with write access to `supabase/tests`
52+
- Add via: Settings → Secrets and variables → Actions → New repository secret
53+
54+
2. **`DEV_AWS_ROLE`** (already exists)
55+
- AWS IAM role for building and managing test AMIs
56+
- Used for AWS authentication via OIDC
57+
58+
### In `supabase/tests` repository:
59+
60+
1. **`POSTGRES_REPO_STATUS_PAT`** (required)
61+
- A GitHub Personal Access Token with `repo` scope
62+
- Used to update commit status checks on `supabase/postgres` PRs
63+
- Token should be created by a user with write access to `supabase/postgres`
64+
- Add via: Settings → Secrets and variables → Actions → New repository secret
65+
66+
2. **Other secrets** (already exist)
67+
- `AWS_DEV_ROLE` - AWS IAM role
68+
- `INFRA_INTEGRATION_STAGING_V0_KEY` - Platform API key
69+
- `INFRA_INTEGRATION_STAGING_V1_KEY` - Platform API key
70+
- `PLATFORM_THROTTLE_SKIP_TOKEN_STAGING` - API throttle bypass
71+
72+
## Workflow Files
73+
74+
### 1. `supabase/postgres/.github/workflows/pr-ami-test-trigger.yml`
75+
76+
**Purpose:** Build test AMI and trigger integration tests
77+
78+
**Triggers:**
79+
- Pull request labeled with `test-ami`
80+
- Pull request synchronized (new commits) when label is present
81+
82+
**What it does:**
83+
1. Checks for `test-ami` label
84+
2. Builds AMI for each PostgreSQL version (15, 16, 17)
85+
3. Creates pending status check on PR
86+
4. Dispatches event to `supabase/tests` with AMI details
87+
5. Comments on PR with AMI information
88+
6. Cleans up resources on completion/failure
89+
90+
### 2. `supabase/tests/.github/workflows/postgres_pr_ami_test.yml`
91+
92+
**Purpose:** Run integration tests and report results
93+
94+
**Triggers:**
95+
- `repository_dispatch` event type: `postgres-ami-pr-test`
96+
97+
**What it does:**
98+
1. Receives AMI details from dispatch payload
99+
2. Creates Supabase project with specified Postgres version
100+
3. Runs integration test suite:
101+
- Platform integration tests
102+
- Client library tests
103+
- Platform logs tests
104+
4. Updates commit status on original PR
105+
5. Comments on PR with test results
106+
6. Cleans up test project and AMI
107+
108+
## Setup Instructions
109+
110+
### Step 1: Create GitHub Personal Access Tokens
111+
112+
#### Token 1: For `supabase/postgres` (TESTS_REPO_DISPATCH_PAT)
113+
1. Go to GitHub → Settings → Developer settings → Personal access tokens → Fine-grained tokens
114+
2. Generate new token with:
115+
- **Repository access:** Only select repositories → `supabase/tests`
116+
- **Permissions:**
117+
- Repository permissions → Actions: Read and write
118+
- Repository permissions → Contents: Read-only
119+
3. Copy token and add to `supabase/postgres` secrets as `TESTS_REPO_DISPATCH_PAT`
120+
121+
#### Token 2: For `supabase/tests` (POSTGRES_REPO_STATUS_PAT)
122+
1. Generate new token with:
123+
- **Repository access:** Only select repositories → `supabase/postgres`
124+
- **Permissions:**
125+
- Repository permissions → Commit statuses: Read and write
126+
- Repository permissions → Issues: Read and write (for PR comments)
127+
- Repository permissions → Pull requests: Read-only
128+
2. Copy token and add to `supabase/tests` secrets as `POSTGRES_REPO_STATUS_PAT`
129+
130+
### Step 2: Deploy Workflow Files
131+
132+
1. **In `supabase/postgres` repository:**
133+
```bash
134+
# Add and commit the workflow file
135+
git add .github/workflows/pr-ami-test-trigger.yml
136+
git commit -m "feat: add PR AMI integration test trigger workflow"
137+
git push
138+
```
139+
140+
2. **In `supabase/tests` repository:**
141+
```bash
142+
# Add and commit the workflow file
143+
git add .github/workflows/postgres_pr_ami_test.yml
144+
git commit -m "feat: add postgres PR AMI test receiver workflow"
145+
git push
146+
```
147+
148+
### Step 3: Configure Branch Protection Rules
149+
150+
To require integration tests before merging to `develop`:
151+
152+
1. Go to `supabase/postgres` → Settings → Branches
153+
2. Edit branch protection rule for `develop`
154+
3. Enable "Require status checks to pass before merging"
155+
4. Add required status checks:
156+
- `ami-integration-tests/pg15`
157+
- `ami-integration-tests/pg16`
158+
- `ami-integration-tests/pg17`
159+
5. Save changes
160+
161+
## Usage
162+
163+
### Running Integration Tests on a PR
164+
165+
1. **Open or update a PR to `develop` branch**
166+
167+
2. **Add the `test-ami` label:**
168+
- Go to the PR page
169+
- Click "Labels" on the right sidebar
170+
- Add the `test-ami` label
171+
172+
3. **Wait for AMI build:**
173+
- The workflow will start automatically
174+
- Check the "Actions" tab for progress
175+
- AMI build takes ~45-60 minutes per PostgreSQL version
176+
177+
4. **Monitor test execution:**
178+
- Once AMI is built, tests will run in the private `supabase/tests` repo
179+
- PR status checks will update automatically
180+
- You'll see comments on the PR with progress and results
181+
182+
5. **Review results:**
183+
- ✅ Green check: Tests passed, PR can be merged
184+
- ❌ Red X: Tests failed, review the test logs
185+
- Click the "Details" link in status checks to view test output
186+
187+
### Manual Trigger (Optional)
188+
189+
You can also trigger tests manually via workflow_dispatch in the GitHub Actions UI.
190+
191+
## Troubleshooting
192+
193+
### Status check never completes
194+
- Check if the workflow ran in `supabase/tests`
195+
- Verify the `POSTGRES_REPO_STATUS_PAT` secret is valid
196+
- Check Actions logs for authentication errors
197+
198+
### AMI build fails
199+
- Check AWS credentials and role permissions
200+
- Review the packer build logs in the workflow
201+
- Verify all dependencies are available in the nix flake
202+
203+
### Tests fail unexpectedly
204+
- Check if the staging environment is healthy
205+
- Review test logs in the `supabase/tests` workflow
206+
- Verify all required secrets are configured
207+
208+
### Permission errors
209+
- Verify PAT tokens have correct scopes
210+
- Check repository access permissions for the tokens
211+
- Ensure tokens haven't expired
212+
213+
## Customization
214+
215+
### Running different test suites
216+
217+
Edit `supabase/tests/.github/workflows/postgres_pr_ami_test.yml` to:
218+
- Add/remove test commands
219+
- Change test environment (staging vs prod)
220+
- Adjust timeout values
221+
- Modify resource cleanup behavior
222+
223+
### Changing trigger conditions
224+
225+
Edit `supabase/postgres/.github/workflows/pr-ami-test-trigger.yml` to:
226+
- Use different label names
227+
- Trigger on different events
228+
- Build for specific PostgreSQL versions only
229+
- Use different AWS regions
230+
231+
### Test result notifications
232+
233+
The workflow already:
234+
- Updates PR status checks
235+
- Adds PR comments with results
236+
- Links to detailed test logs
237+
238+
You can extend this to:
239+
- Send Slack notifications
240+
- Create GitHub issues for failures
241+
- Update external dashboards
242+
243+
## Cost Considerations
244+
245+
- Each test run builds temporary AMIs and creates test projects
246+
- AMIs are automatically cleaned up after tests
247+
- Test projects are deleted after test completion
248+
- Estimated cost: ~$1-2 per full test run (all PG versions)
249+
250+
## Maintenance
251+
252+
### Keeping workflows updated
253+
254+
- Review workflow files when updating PostgreSQL versions
255+
- Update test suites as new features are added
256+
- Monitor for deprecated GitHub Actions
257+
- Rotate PAT tokens before expiration (typically 90 days)
258+
259+
### Monitoring
260+
261+
- Set up alerts for workflow failures
262+
- Track test execution times
263+
- Monitor resource usage and costs
264+
- Review test coverage periodically
265+
266+
## Security
267+
268+
- PAT tokens are stored as encrypted secrets
269+
- Workflows use OIDC for AWS authentication (no long-lived credentials)
270+
- Test AMIs are isolated to development environment
271+
- Automatic cleanup prevents resource leaks
272+
273+
## Support
274+
275+
For issues or questions:
276+
- Check workflow logs in the Actions tab
277+
- Review this documentation
278+
- Contact the infrastructure team
279+
- Open an issue in the relevant repository

0 commit comments

Comments
 (0)