Skip to content

Commit 5b4664b

Browse files
serverless-momserverless-mom
committed
new guide
1 parent 01a4882 commit 5b4664b

File tree

4 files changed

+353
-0
lines changed

4 files changed

+353
-0
lines changed

‎site/assets/guides/images/suites-groups-01.png‎

293 KB
Loading

‎site/assets/guides/images/suites-groups-02.png‎

484 KB
Loading

‎site/assets/guides/images/suites-groups-03.png‎

138 KB
Loading

‎site/content/guides/check-suites-and-groups.md‎

Lines changed: 353 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,353 @@
1+
---
2+
title: Organize Monitoring with Playwright Check Suites and Groups
3+
displayTitle: Organize Monitoring with Playwright Check Suites and Groups
4+
description: >-
5+
By combining Playwright Check Suites with Checkly Groups, you can create a robust monitoring system that scales with your development team.
6+
author: Nočnica Mellifera
7+
avatar: 'images/avatars/nica-mellifera.png'
8+
tags:
9+
- FAQ
10+
---
11+
12+
For mature operations teams, organizing your monitoring is as important as its initial design and configuration. Without clear grouping and consistent configuration management, your team will quickly suffer from alert fatigue and confusion during incidents. This guide will show you how to use Checkly’s newest check suite functionality along with groups to manage all your checks and get focused, actionable alerts. Let’s take a look at our goal for this guide:
13+
14+
![The Checkly Web UI with multiple groups and Check Suites](/guides/images/suites-groups-01.png)
15+
*A well-organized Checkly Web UI with Check Suites and groups*
16+
17+
After following this guide, your Checkly web UI should contain:
18+
19+
- Individual tests collected into Playwright Check Suites to represent functional portions of your site or service
20+
- Groups to let you configure multiple checks simultaneously, controlling your alerting so that everyone gets the notifications they need
21+
- A clear and readable web interface that shows you your services' status at a glance
22+
23+
## Understanding Check Groups vs. Playwright Check Suites
24+
25+
Before diving into implementation, it's important to understand the distinction between Checkly's two organizational features:
26+
27+
### Playwright Check Suites
28+
[Playwright Check Suites](https://www.checklyhq.com/docs/playwright-checks/) are **test execution units** that group multiple Playwright tests together to run as a single monitor. They focus on **what you're testing**:
29+
30+
- **Purpose**: Bundle related tests that validate a specific feature or user journey
31+
- **Scope**: Technical grouping based on test dependencies and shared context
32+
- **Example**: All tests for your checkout flow (search product → add to cart → complete purchase)
33+
- **Result**: Single pass/fail status for the entire suite
34+
- **When to use**: When tests are interdependent or represent a cohesive user workflow
35+
36+
### Check Groups
37+
[Check Groups](https://www.checklyhq.com/docs/groups/) are **management containers** that organize multiple checks (including Check Suites) for shared configuration and alerting. They focus on **how you manage** your monitoring:
38+
39+
- **Purpose**: Organize checks by team, priority, or operational requirements
40+
- **Scope**: Operational grouping based on alert routing and configuration needs
41+
- **Example**: "Critical Production Monitors" containing checkout, authentication, and payment suites
42+
- **Result**: Shared alerting rules, locations, and notification channels for all contained checks
43+
- **When to use**: When checks need the same alert escalation, notification recipients, or monitoring frequency
44+
45+
### The Relationship
46+
47+
Think of it this way:
48+
- **Check Suites** = "These tests belong together functionally"
49+
- **Check Groups** = "These monitors should be managed together operationally"
50+
51+
A single Check Group can contain multiple Check Suites, and each Check Suite inherits the group's configuration (alerting, scheduling, locations, etc.).
52+
53+
## Creating Playwright Check Suites
54+
55+
Checkly’s Playwright Check Suites are a new feature that lets you use your existing Playwright tests as live, scheduled monitoring checks. No rewrites required. Your team can define what to monitor using code, selecting the right tags and projects, and run your checks globally, using the [Checkly CLI](https://www.checklyhq.com/docs/cli).
56+
57+
Checkly’s support for Playwright native projects offers a number of advantages for experienced Playwright teams:
58+
59+
- Bring in custom dependencies to run on Checkly’s globally distributed monitoring infrastructure.
60+
- Quickly go from a Playwright project running locally, to scheduled monitoring, with no need to rewrite your project.
61+
- Group tests as needed into Check Suites, showing the status of a whole feature at once.
62+
63+
It’s this last feature that is critical for this guide.
64+
65+
### Assigning Playwright Tests to a Check Suite
66+
67+
Check suites group multiple small tests into a single result for display on your dashboard. Once we’ve imported a test file to Checkly, we’ll run these tests as a check suite regularly, and get a report back. Here’s an example of the results from a run of a check suite:
68+
69+
![The Checkly Web UI reporting on a single check run](/guides/images/suites-groups-02.png)
70+
71+
*Since this monitor will generate alerts, it makes sense to collect these tests into one Check Suite, and get an alert when any one of them fails.*
72+
73+
It makes the most sense to group multiple playwright tests into a Playwright Check Suite when you want to run the tests together, and a failure of any included test would mean that the feature wasn’t working.
74+
75+
Let’s take a look at the Playwright tests that we want to turn into the monitor above. Here’s the test file we’ve previously run locally as an end-to-end test:
76+
77+
```ts {title="catalog-e2e.spec.ts"}
78+
import { test, expect } from '@playwright/test';
79+
80+
let totalCost: number = 0;
81+
test.describe.configure({ mode: 'serial' }); //these interdependent tests need to run in sequence
82+
83+
test('search and find book', {
84+
tag: '@catalog', //we need to tag our tests to group them into suites
85+
}, async ({ page }) => {
86+
await page.goto('http://danube-web.shop/');
87+
await page.getByRole('textbox').click();
88+
await page.getByRole('textbox').fill('scott');
89+
await page.getByRole('button', { name: 'Search' }).click();
90+
await expect(page.locator('#app-content')).toContainText('$9.95');
91+
const bookCost: String = await page.locator('#app-content').textContent() || "0"
92+
totalCost += parseFloat(bookCost.match(/\$(\d+\.\d+)/)?.[1] || "0")
93+
});
94+
95+
//alternatively you can add @tag in the test name
96+
test('click book from @catalog', async ({ page }) => {
97+
await page.goto('http://danube-web.shop/');
98+
await page.getByText('Celsius').click();
99+
await expect(page.getByText('Price: $')).toBeVisible();
100+
await expect(page.locator('#app-content')).toContainText('$9.95');
101+
const bookCost: String = await page.locator('#app-content').textContent() || "0"
102+
totalCost += parseFloat(bookCost.match(/\$(\d+\.\d+)/)?.[1] || "0")
103+
});
104+
105+
test('add to cart from @catalog', async ({ page }) => {
106+
await page.goto('http://danube-web.shop/');
107+
await page.getByText('The Grand Grotsby').click();
108+
await page.getByRole('button', { name: 'Add to cart' }).click();
109+
await page.getByText('Celsius').click();
110+
await page.getByRole('button', { name: 'Add to cart' }).click();
111+
const priceString = totalCost.toString();
112+
//we've added the same two books to our cart, now check the total price is correct
113+
await expect(page.locator('#total-price')).toContainText(priceString);
114+
});
115+
```
116+
117+
*This file runs three tests in sequence: looking up the price of two books, then adding both to a cart and expecting the cart total to equal the sum of the book prices. The success of the third test relies on the first two tests.*
118+
119+
These tests work great when run from our local playwright environment, but now we are going to run them as an ongoing, scheduled monitor of our catalog service. Due to these tests’ interdependence, it doesn’t make much sense to try and run them separately. Further, the tests in this file cover a single function, our eCommerce shop’s book catalog, so it makes sense to add them as a single check suite.
120+
121+
After completing the [setup steps for Checkly and installing the Checkly CLI](https://www.checklyhq.com/docs/playwright-checks/), all we’ll need to create this first Check Suite is a configuration file:
122+
123+
```ts {title="checkly.config.ts"}
124+
import { defineConfig } from 'checkly'
125+
import { Frequency } from 'checkly/constructs'
126+
127+
export default defineConfig({
128+
/* A human friendly name for your project */
129+
projectName: 'Playwright Group Test',
130+
logicalId: 'playwrightGroupTest',
131+
checks: {
132+
playwrightConfigPath: './playwright.config.ts',
133+
playwrightChecks: [
134+
{
135+
name: 'Catalog e2e',
136+
logicalId: 'catalog-test-suite',
137+
pwProjects: ['chromium'],
138+
pwTags: '@catalog', //all tests with this tag will be included
139+
installCommand: 'npm install --dev',
140+
frequency: Frequency.EVERY_1M,
141+
testCommand: 'npx playwright test --grep@checkly --config=playwright.config.ts',
142+
activated: true, //run these tests
143+
muted: false, //send alerts on failure
144+
}
145+
]
146+
},
147+
})
148+
```
149+
150+
With this new config file saved in our project we can run `npx checkly test` to see our tests run through the Checkly global infrastructure:
151+
152+
```ts
153+
npx checkly test
154+
155+
Parsing your project... ✅
156+
Validating project resources... ✅
157+
Bundling project resources... ✅
158+
159+
playwright.config.ts
160+
161+
✔ Catalog e2e (8s)
162+
163+
1 passed, 1 total
164+
165+
Tip > Use `--record` to get test results in Checkly with full traces, videos and logs: `npx checkly test --record`
166+
```
167+
168+
Once tests are passing, you can deploy this first Check Suite to Checkly with `npx checkly deploy`.
169+
170+
> Note: by default, a playwright test only captures a trace if it’s running a retry, indicating something has failed at least once. If you want to see traces for your Check Suites even if they’ve run successfully, edit your `playwright.config.ts` file. The default value for traces is `trace: 'on-first-retry'` , edit this to be `trace: 'on'` .
171+
>
172+
173+
### When to Separate Tests Into Multiple Check Suites
174+
175+
In the previous example it made sense to group our three interdependent and closely related tests into a single check suite. Let’s look at a scenario where it makes sense to separate our tests into two check suites.
176+
177+
In this test file, our two tests are looking at quite different areas of our service.
178+
179+
```ts {title="misc-tests.spec.ts"}
180+
import { test, expect } from '@playwright/test';
181+
182+
test('can open support chat', async ({ page }) => {
183+
await page.goto('https://danube-web.shop/');
184+
await page.getByText('support').click();
185+
await expect(page).toHaveTitle(/GetHelp/);
186+
});
187+
188+
test('can check out', async ({ page }) => {
189+
await page.goto('https://danube-web.shop/cart');
190+
await page.getByText('Checkout').click();
191+
// Multiple Steps of entering checkout details
192+
await page.getByText('Place My Order').click();
193+
});
194+
```
195+
196+
Here our test is looking at two totally separate parts of the site, and we don’t want to receive alerts saying that the checkout process is failing when the problem is related to our support service.
197+
198+
![Message Alert](/guides/images/suites-groups-03.png)
199+
200+
*Checkly delivers alerts when one of your monitors fails, and we don’t want to add confusion by mixing up the features being monitored!*
201+
202+
The process to separate these checks is quite easy, we want to add tags as appropriate to the tests:
203+
204+
```ts {title="misc-tests.spec.ts"}
205+
import { test, expect } from '@playwright/test';
206+
207+
test('can open support chat', {tag: '@docsite'}, async ({ page }) => {
208+
await page.goto('https://danube-web.shop/');
209+
await page.getByText('support').click();
210+
await expect(page).toHaveTitle(/GetHelp/);
211+
});
212+
213+
test('can check out', {tag: '@estore'}, async ({ page }) => {
214+
await page.goto('https://danube-web.shop/cart');
215+
await page.getByText('Checkout').click();
216+
// Multiple Steps of entering checkout details
217+
await page.getByText('Place My Order').click();
218+
});
219+
```
220+
221+
To create these Check Suites, just add these tags to our Checkly configuration
222+
223+
```ts {title="checkly.config.ts"}
224+
import { defineConfig } from 'checkly'
225+
import { Frequency } from 'checkly/constructs'
226+
227+
export default defineConfig({
228+
/* A human friendly name for your project */
229+
projectName: 'Playwright Group Test',
230+
logicalId: 'playwrightGroupTest',
231+
checks: {
232+
playwrightConfigPath: './playwright.config.ts',
233+
playwrightChecks: [
234+
{
235+
name: 'Catalog e2e',
236+
logicalId: 'catalog-test-suite',
237+
pwProjects: ['chromium'],
238+
pwTags: '@catalog', //all tests with this tag will be included
239+
240+
},
241+
{
242+
name: 'Docs Site',
243+
logicalId: 'support-test-suite',
244+
pwProjects: ['chromium'],
245+
pwTags: '@docsite', //all tests with this tag will be included
246+
247+
},
248+
{
249+
name: 'Storefront eCommerce Tests',
250+
logicalId: 'estore-test-suite',
251+
pwProjects: ['chromium'],
252+
pwTags: '@estore', //all tests with this tag will be included
253+
254+
}
255+
]
256+
},
257+
})
258+
```
259+
260+
*The configuration presented here is minimal, we’ll add some more detail in the next section, but for a full explanation, read the [reference pages on Playwright Check Suites](https://www.checklyhq.com/docs/playwright-checks/reference/).*
261+
262+
We may want to separate out these tests into two separate tests files. If you are sharing code between these two tests, there should always be a way to separate the shared code to an [import that both test files can use](https://www.checklyhq.com/docs/cli/import/). For example a standard login function can be brought into each new test file with:
263+
264+
```ts
265+
import { loginFunction } from '../utils/login.ts'
266+
```
267+
268+
However, it’s not necessary for the purposes of Playwright Check Suites to separate tests into multiple files. In short: just because two tests are in the same file does *not* mean they have to be run in the same Check Suite.
269+
270+
While Check Suites are perfect for running multiple tests on a single feature, we’d also like to group up our checks to manage them together. For that, we have Checkly Groups.
271+
272+
## Check Groups
273+
274+
In our ideal Checkly Web UI, our check suites are organized into logical groups:
275+
276+
![The Checkly Web UI with multiple groups and Check Suites](/guides/images/suites-groups-01.png)
277+
278+
It generally makes sense to organize groups by alert priority and working group. For example, the operations team and dev teams will probably use separate groups, and performance measurement shouldn’t be grouped up with critical user paths. We can create these groups via a [Checkly construct](https://www.checklyhq.com/docs/cli/constructs-reference/#checkgroupv2):
279+
280+
```ts
281+
//{title="groups.check.ts"}
282+
import { CheckGroupV2, Frequency, AlertEscalationBuilder } from 'checkly/constructs'
283+
// import alert channel configuration from a local file
284+
import { smsChannel, emailChannel } from './alert-channels'
285+
286+
export const criticalGroup = new CheckGroupV2('check-group-critical', {
287+
name: 'Critical Monitors',
288+
activated: true,
289+
muted: false,
290+
locations: ['us-east-1', 'us-west-1', 'eu-central-1', ],
291+
// By setting an alertEscalationPolicy, these settings will override those on individual checks
292+
alertEscalationPolicy: AlertEscalationBuilder.runBasedEscalation(
293+
4, // Alert after 4 failures
294+
{ amount: 2, interval: 5 }, // Send 2 reminders, 5 minutes apart
295+
{ enabled: true, percentage: 50 } // Alert if 50% of parallel runs fail
296+
),
297+
alertChannels: [emailChannel, smsChannel]
298+
})
299+
300+
export const devGroup = new CheckGroupV2('check-group-dev', {
301+
name: 'Dev Environment Monitors',
302+
activated: true,
303+
muted: false,
304+
locations: ['us-east-1', 'eu-central-1', ],
305+
alertEscalationPolicy: AlertEscalationBuilder.runBasedEscalation(
306+
1, // Alert after 1 failure, this is way too sensitive for every day but makes sense while testing changes on Dev
307+
{ amount: 2, interval: 5 }, // Send 2 reminders, 5 minutes apart
308+
{ enabled: true, percentage: 50 } // Alert if 50% of parallel runs fail
309+
),
310+
alertChannels: [emailChannel]
311+
})
312+
```
313+
314+
Our groups are based on how we want to be notified, and shared configuration for all the included checks. For our critical monitors, we want rapid updates in the event of failure, via both email and SMS. Critical monitors also perform a few retries before generating an alert. For our Dev environment, email is fine as an alert channel, but we want to be notified the minute a failure is detected.
315+
316+
Next we can update our `checkly.config.ts` to add our Check Suites to groups. Update the items in `playwrightChecks` with a group name:
317+
318+
```ts
319+
{
320+
name: 'Catalog e2e',
321+
logicalId: 'catalog-test-suite',
322+
pwProjects: ['chromium'],
323+
pwTags: '@catalog',
324+
groupName: 'Critical Monitors',
325+
// additional config removed for brevity
326+
}
327+
```
328+
329+
Check Suites will be added to groups based on the `name` value in the group configuration. You can also add groups to groups you’ve previously created by matching the name.
330+
331+
When we’re done, you can deploy your group, and update your checks to those groups, with the `deploy` command:
332+
333+
```bash
334+
npx checkly deploy --preview #confirm what will be deployed
335+
npx checkly deploy -o # deploy and output a list of what was updated
336+
```
337+
338+
## Conclusion
339+
340+
By combining Playwright Check Suites with Checkly Groups, you've created a robust monitoring system that scales with your development team. Check Suites allow you to leverage your existing Playwright tests as live monitors without rewrites, while Groups provide the organizational structure needed to manage alerts effectively and reduce noise during incidents.
341+
342+
This approach delivers several key benefits:
343+
- **Developer-friendly monitoring** that integrates naturally with existing test workflows
344+
- **Clear organization** that prevents alert fatigue and confusion during outages
345+
- **Focused alerting** that ensures the right teams get notified about relevant failures
346+
- **Scalable architecture** that grows with your application and team structure
347+
348+
As your monitoring needs evolve, you can easily expand this foundation by adding new Check Suites for additional features and creating specialized Groups for different teams or alert priorities.
349+
350+
### Further Reading
351+
352+
- Full documentation on creating a [Playwright Suite](https://www.checklyhq.com/docs/playwright-checks/reference/).
353+
- Explore [Group Configurations](https://www.checklyhq.com/docs/groups/#group-level-configuration).

0 commit comments

Comments
 (0)