Skip to content

Commit 0d2b100

Browse files
mraiblemraible
committed
Add E2E testing improvements and document current status
1 parent 92934d3 commit 0d2b100

10 files changed

+531
-62
lines changed

‎e2e-plan.md‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
# E2E Testing Plan - ServiceNow IDP
2+
3+
## Current Status
4+
5+
### Completed Improvements
6+
1. ✅ Updated field mapping logic to correctly identify and fill ServiceNow API integration fields
7+
2. ✅ Implemented multi-instance API integration handling (for `x-cs-multi-instance: true`)
8+
3. ✅ Enhanced error detection to capture installation failure messages
9+
4. ✅ Improved test credentials using realistic formats for Basic Auth
10+
5. ✅ Added better logging for debugging installation issues
11+
12+
### Current Issue: Installation Validation Failure
13+
14+
**Problem:** App installation fails with error message "Install failed" despite:
15+
- Workflows having `provision_on_install: false` (no workflows run during installation)
16+
- Fake credentials being provided for the API integration
17+
- All form fields being correctly filled
18+
19+
**Error Message Captured:**
20+
```
21+
❌ Installation failed - error message: Install failed
22+
```
23+
24+
**Configuration Being Provided:**
25+
- Name: "Test ServiceNow Integration"
26+
- Host: "https://dev123456.service-now.com"
27+
- Username: "foundry_test_user"
28+
- Password: "Test123!Password"
29+
30+
**Test Output:**
31+
```
32+
ℹ️ Configuration fields detected, filling them
33+
ℹ️ Filled field (context: "name...", name: "name") with: Test ServiceNow Integration
34+
ℹ️ Filled field (context: "host...", name: "params.path.host") with: https://dev123456.service-now.com
35+
ℹ️ Filled field (context: "username...", name: "auth.username") with: foundry_test_user
36+
ℹ️ Filled password field (context: "password...") with test credentials
37+
👆 [10] AppCatalogPage: Click Install button
38+
ℹ️ Clicked install button
39+
ℹ️ Waiting for installation to complete...
40+
ℹ️ Installation started - "installing" toast visible
41+
❌ Installation failed - error message: Install failed
42+
```
43+
44+
### Hypothesis
45+
46+
The ServiceNow API integration has `x-cs-multi-instance: true` in the OpenAPI spec, which suggests:
47+
48+
1. **Possible Validation Issue:** The backend might be performing validation on the API integration configuration even though workflows don't run on install
49+
2. **Possible Multi-Instance Requirement:** Apps with `x-cs-multi-instance: true` might require explicit instance creation, not just form filling
50+
3. **Possible Credential Format Validation:** Basic Auth credentials might have specific format requirements
51+
52+
### Key Files Modified
53+
54+
1. **e2e/src/pages/AppCatalogPage.ts** - Main changes:
55+
- `getFieldValue()` method (lines 167-191): Context-aware field mapping for ServiceNow
56+
- `handleAppConfiguration()` method (lines 206-252): Multi-instance configuration handling
57+
- `fillConfigurationFields()` method (lines 254-273): Smart credential filling based on field type
58+
- `waitForInstallation()` method (lines 327-366): Error detection and detailed logging
59+
60+
2. **e2e.md** - Comprehensive documentation of learnings from insider-threat implementation
61+
62+
3. **Workflows** (already have `provision_on_install: false`):
63+
- `workflows/ServiceNow_to_IDP_policy_rules_synchronizer.yml`
64+
- `workflows/Initialize_or_Reset_checkpoint-LatestSysUpdatedOn.yml`
65+
66+
### Next Steps to Investigate
67+
68+
1. **Manual Testing:** Try manually installing the app through the UI to see:
69+
- What the actual error message is (more detailed than "Install failed")
70+
- If there's a button like "Add instance" or "Create configuration" that needs to be clicked
71+
- Whether the form validation shows specific field errors
72+
- Check browser console for any JavaScript errors or API responses
73+
74+
2. **Compare with Insider Threat:** Check if insider-threat's multi-instance handling includes:
75+
- Explicit "Add instance" button clicks in the E2E test code
76+
- Different credential formats or validation patterns
77+
- Additional configuration steps specific to multi-instance APIs
78+
- Look at: `/Users/mraible/dev/foundry-sample-insider-threat/e2e/src/pages/AppCatalogPage.ts`
79+
80+
3. **Check API Integration Schema:** Look for validation rules in the OpenAPI spec that might reject fake credentials
81+
- Review `api-integrations/servicenow_table_int_4.json` for validation patterns
82+
- Check if there are schema validations on the Basic Auth fields
83+
- Look for any custom `x-cs-*` properties that might enforce validation
84+
85+
4. **Debug with Playwright Tools:**
86+
- Run tests with `npx playwright test --debug` to step through installation
87+
- Use `npx playwright show-trace test-results/.../trace.zip` to see detailed execution
88+
- Check network requests during installation to see API validation responses
89+
90+
5. **Consider Alternative Approaches:** If validation can't be bypassed:
91+
- Mark API integration as optional for E2E tests (if possible in manifest)
92+
- Remove `x-cs-multi-instance: true` temporarily to see if single instance works
93+
- Focus E2E tests only on workflow/function/collection verification (skip installation)
94+
- Use integration tests that mock the Falcon API responses
95+
96+
### Questions to Answer Next Week
97+
98+
1. Does the insider-threat app's E2E test click any additional buttons like "Add instance" or "Save configuration"?
99+
2. Are there any console errors or API response messages that provide more detail than "Install failed"?
100+
3. Can we reproduce the installation failure manually to see the exact error?
101+
4. Is there a way to disable API integration validation during E2E tests?
102+
5. Should we skip installation testing and only test the deployed app's components?
103+
104+
### Working Code Examples
105+
106+
**Running the tests:**
107+
```bash
108+
cd e2e
109+
npm test # Run all tests
110+
npm run test:debug # Debug mode
111+
npm run test:ui # Interactive UI
112+
DEBUG=true npm test # Verbose logging
113+
```
114+
115+
**Check deployment status:**
116+
```bash
117+
foundry apps list-deployments
118+
foundry apps deploy --change-type=patch --change-log="E2E test deployment"
119+
```
120+
121+
**Manual installation test:**
122+
1. Navigate to https://falcon.us-2.crowdstrike.com/foundry/app-catalog
123+
2. Search for "foundry-sample-servicenow-idp"
124+
3. Click "Install now"
125+
4. Fill in the ServiceNow API integration fields with the test values above
126+
5. Observe any error messages or validation issues
127+
6. Check browser console (F12) for errors

‎e2e/playwright.config.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ export default defineConfig({
1515
expect: {
1616
timeout: process.env.CI ? 10 * 1000 : 8 * 1000,
1717
},
18-
reporter: 'html',
18+
reporter: 'list',
1919
use: {
2020
testIdAttribute: 'data-test-selector',
2121
trace: 'on-first-retry',

0 commit comments

Comments
 (0)