Add high-quality SKILL for compiler and fourslash tests by Copilot · Pull Request #2929 · microsoft/typescript-go

Copilot · 2026-02-27T17:37:18Z

Research how compiler tests and fourslash tests work in this repo
Understand the baseline system, test runners, and debugging workflows
Create a comprehensive SKILL document covering compiler tests and fourslash tests
Review the SKILL document for accuracy and completeness
Use proper skill format: SKILL.md inside a named directory with YAML frontmatter header
Integrate all review feedback from @jakebailey and @gabritto

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

…d fourslash tests Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com>

RyanCavanaugh

@copilot follow the proper skill format; this lacks the correct heder

…cture Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com>

Copilot · 2026-02-27T18:01:55Z

@copilot follow the proper skill format; this lacks the correct heder

Fixed in 7c82265 — moved to .github/skills/compiler-and-fourslash-tests/SKILL.md with YAML frontmatter (name, description).

gabritto · 2026-02-27T19:26:59Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+- Should not be manually edited (they'll be overwritten on regeneration)
+- Provide coverage for ported TypeScript behavior
+
+Hand-written tests (directly in `internal/fourslash/tests/` or in `manual/`):


Not sure if this matters because we have it elsewhere in repo instructions, but the tests in manual should correspond to a manually-fixed but generated test, so that's not exactly the right place for custom regression tests.

gabritto · 2026-02-27T19:30:19Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+2. Check baseline diffs: `diff testdata/baselines/reference/compiler/failingTest.errors.txt testdata/baselines/local/compiler/failingTest.errors.txt`
+3. If the new output is correct, accept: `npx hereby baseline-accept`
+4. If not, fix the code and re-run
+


One thing I sometimes need to do when there's an unrecovered panic is to run all tests in a package sequentially (-parallel=1) and in verbose mode, such that the last test that shows up as running before the panic is the one that caused it. May or may not be worth adding it here, I only needed it a few times.

Yep, I do this too.

jakebailey · 2026-02-27T19:28:49Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+
+#### Generating test variations
+
+Options that affect program behavior (those marked with `AffectsProgramStructure`, `AffectsEmit`, `AffectsModuleResolution`, `AffectsDiagnostics`, etc.) can specify multiple values to generate separate sub-test configurations:


Basically any option can do variation, so this detail is not helpful.

jakebailey · 2026-02-27T19:29:28Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+
+```typescript
+// @module: *
+// @target: *, !es3


This should use a better example than !es3 given I had to special case that to ignore it. We don't actually use this feature except for the one !es3 case, so it's probably not worth it anyway.

jakebailey · 2026-02-27T19:30:24Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+
+#### Symlink tests
+
+Use `// @link:` to create symlinks in the virtual filesystem:


I think this is wrong and only works for dir links due to some windows emulation constraints... which we don't have, so I personally just use @symlink.

jakebailey · 2026-02-27T19:30:46Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+
+- `// @currentDirectory: /custom/path` — Set the working directory
+- `// @noImplicitReferences` — Don't auto-include referenced files
+- `// @lib: es2020,dom` — Specify lib files


This should note that lib is not variant, so commas just add more things.

jakebailey · 2026-02-27T19:31:32Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+npx hereby test --tests "myNewTest"      # Run tests matching a pattern (passed as -run flag)
+```
+
+#### Via Go directly (faster for specific tests)


We should only suggest go test for running single tests in verbose mode for print debugging. I don't think it's a great idea to use it otherwise, except for non-compiler tests.

jakebailey · 2026-02-27T19:37:35Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+| `npx hereby test --race` | Enable Go race detector |
+| `npx hereby test --coverage` | Generate coverage profiles |
+| `npx hereby baseline-accept` | Accept local baselines as new reference |
+| `npx hereby diff` | Diff baselines with tool from `$DIFF` env var |


Also not good for agents.

jakebailey · 2026-02-27T19:37:47Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+### 3.2 Environment Variables
+
+| Variable | Purpose |
+|----------|---------|
+| `TSGO_BASELINE_TRACKING_DIR` | Directory for baseline usage tracking files |
+| `TS_TEST_PROGRAM_SINGLE_THREADED` | Set to `false` for parallel test programs |
+| `DIFF` | Path to diff tool for `npx hereby diff` |
+| `TSGO_HEREBY_RACE` | Enable race detector via env var |
+| `TSGO_HEREBY_NOEMBED` | Skip embedding bundled assets |


None of these should ever be set manually.

jakebailey · 2026-02-27T19:38:17Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+
+#### Adding a new compiler test
+1. Create `testdata/tests/cases/compiler/myTest.ts` with test code and directives
+2. Run: `go test ./internal/testrunner/ -run 'TestLocal/compiler/myTest.ts'`


Should be hereby test, as that clears out the reference directory on startup. Often the agent will add a test, then accept baselines from other previous runs.

jakebailey · 2026-02-27T19:38:35Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+
+#### Adding a new fourslash test
+1. Create `internal/fourslash/tests/myTest_test.go` with the test function
+2. Run: `go test ./internal/fourslash/tests/... -run TestMyTest`


I think this should too just do hereby test. It's really not that slow....

jakebailey · 2026-02-27T19:39:08Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+### 3.4 Key Source Files
+
+| File | Purpose |
+|------|---------|
+| `internal/testrunner/compiler_runner.go` | Compiler test runner |
+| `internal/testrunner/compiler_runner_test.go` | `TestLocal` and `TestSubmodule` entry points |
+| `internal/testrunner/test_case_parser.go` | Parses `// @` directives from test files |
+| `internal/fourslash/fourslash.go` | Fourslash test harness (all `Verify*` methods) |
+| `internal/fourslash/test_parser.go` | Parses markers (`/*name*/`, `[|range|]`) |
+| `internal/fourslash/baselineutil.go` | Fourslash baseline file generation |
+| `internal/fourslash/tests/util/` | Shared test constants (`DefaultCommitCharacters`, etc.) |
+| `internal/testutil/baseline/baseline.go` | Core baseline comparison and tracking |
+| `Herebyfile.mjs` | Build/test task definitions |


I feel like this is pointless; maybe useful for generating this file, but if an agent reads these in the process of doing anything, it's just wasting context.

RyanCavanaugh · 2026-02-27T19:41:37Z

@copilot integrate everyone's feedback and push a new commit

Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com>

Copilot · 2026-02-27T19:49:56Z

@copilot integrate everyone's feedback and push a new commit

Done in 45bec89. All feedback integrated:

Removed AffectsProgramStructure details, wildcard/exclusion syntax, env vars, baseline tracking internals, Delve debugging, hereby diff, troubleshooting table, and key source files section
Changed @link: → @symlink:, noted @lib is non-variant
Made hereby test the recommended way to run tests everywhere; go test only for print-debugging
Replaced DIFF=code npx hereby diff with git diff --diff-filter=AM --no-index
Fixed manual/ description: migrated from gen via npm run makemanual, not handwritten
Added -parallel=1 -v tip for debugging unrecovered panics

Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com>

jakebailey · 2026-02-27T19:52:14Z

.github/skills/compiler-and-fourslash-tests/SKILL.md

+
+#### Via hereby (recommended)
+
+Always use `hereby test` for running tests. It ensures a clean state by clearing stale baselines before running, so results are always trustworthy. Trust the results — if `hereby test` passes, the tests pass.


Unfortunately I think this is the opposite of what @andrewbranch tries to use. It's not clear to me whether or not this matters, though, if this skill will be pulled in when doing other testing?

I guess it talks more specifically below.

Initial plan

23be075

Copilot AI assigned Copilot and RyanCavanaugh Feb 27, 2026

Copilot started work on behalf of RyanCavanaugh February 27, 2026 17:37 View session

docs: add SKILL guide for writing, running, and debugging compiler an…

6cd97eb

…d fourslash tests Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com>

Copilot finished work on behalf of RyanCavanaugh February 27, 2026 17:51

RyanCavanaugh requested changes Feb 27, 2026

View reviewed changes

Copilot started work on behalf of RyanCavanaugh February 27, 2026 17:57 View session

fix: use proper SKILL format with YAML frontmatter and directory stru…

7c82265

…cture Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com>

Copilot finished work on behalf of RyanCavanaugh February 27, 2026 18:02

RyanCavanaugh changed the title ~~[WIP] Add high-quality SKILL for compiler and fourslash tests~~ Add high-quality SKILL for compiler and fourslash tests Feb 27, 2026

RyanCavanaugh self-requested a review February 27, 2026 19:04

gabritto reviewed Feb 27, 2026

View reviewed changes

jakebailey reviewed Feb 27, 2026

View reviewed changes

Copilot started work on behalf of RyanCavanaugh February 27, 2026 19:41 View session

Integrate all review feedback from @jakebailey and @gabritto

45bec89

Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com>

Clarify trust language for hereby test results

6498776

Co-authored-by: RyanCavanaugh <6685088+RyanCavanaugh@users.noreply.github.com>

Copilot finished work on behalf of RyanCavanaugh February 27, 2026 19:51

jakebailey reviewed Feb 27, 2026

View reviewed changes


		#### Generating test variations

		Options that affect program behavior (those marked with `AffectsProgramStructure`, `AffectsEmit`, `AffectsModuleResolution`, `AffectsDiagnostics`, etc.) can specify multiple values to generate separate sub-test configurations:


		#### Symlink tests

		Use `// @link:` to create symlinks in the virtual filesystem:


		#### Via hereby (recommended)

		Always use `hereby test` for running tests. It ensures a clean state by clearing stale baselines before running, so results are always trustworthy. Trust the results — if `hereby test` passes, the tests pass.

Conversation

Copilot AI commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RyanCavanaugh left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Feb 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RyanCavanaugh commented Feb 27, 2026

Uh oh!

Copilot AI commented Feb 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Feb 27, 2026 •

edited

Loading