[html coverage report] Add vendored package + generate HTML report #7528

ChristopherDedominici · 2025-10-08T15:12:52Z

Link to internal design doc: https://www.notion.so/nomicfoundation/Code-coverage-HTML-report-286578cdeaf580bcad22db58ce54fb47?source=copy_link

As listed in the design doc, this PR addresses point number 1 out of 3:

create the HTML report
investigate and fix the coverage bug
refactor the code to separate coverage data collection and report generation

TODOs before merge

add mention to vendored code

NOTEs for review

All the code inside the folder v-next/hardhat-vendored/src/coverage-module has been copied from vendored packages, so there’s no need for a deep review, a quick scan will be sufficient

changeset-bot · 2025-10-08T15:13:04Z

🦋 Changeset detected

Latest commit: b26e193

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2025-10-08T15:14:24Z

hardhat

Total size of the bundle: 236M
Total number of dependencies (including transitive): 48

List of dependencies (sorted by size)

229M	total
35M	@nomicfoundation/edr-linux-x64-musl
35M	@nomicfoundation/edr-linux-x64-gnu
32M	@nomicfoundation/edr-linux-arm64-musl
32M	@nomicfoundation/edr-linux-arm64-gnu
24M	@nomicfoundation/edr-win32-x64-msvc
24M	@nomicfoundation/edr-darwin-x64
20M	@nomicfoundation/edr-darwin-arm64
7.3M	@sentry/core
5.2M	zod
2.7M	micro-eth-signer
1.9M	@noble/curves
1.7M	undici
1.2M	@noble/hashes
1020K	@nomicfoundation/hardhat-utils
868K	@nomicfoundation/hardhat-vendored
864K	@streamparser/json
624K	micro-packed
592K	tsx
548K	@nomicfoundation/hardhat-errors
492K	@scure/bip39
464K	@nomicfoundation/edr
452K	fast-equals
408K	json-stream-stringify
368K	ethereum-cryptography
332K	@streamparser/json-node
320K	enquirer
320K	@nomicfoundation/hardhat-zod-utils
288K	semver
200K	ws
180K	chokidar
176K	get-tsconfig
168K	@scure/base
160K	esbuild
136K	adm-zip
96K	@scure/bip32
92K	chalk
72K	@nomicfoundation/solidity-analyzer
68K	debug
60K	readdirp
56K	rfdc
48K	ansi-colors
44K	resolve.exports
40K	resolve-pkg-maps
36K	p-map
24K	strip-ansi
24K	env-paths
24K	ansi-regex
20K	ms

ChristopherDedominici · 2025-10-13T14:31:13Z

v-next/hardhat/assets/index.js

@@ -0,0 +1,2 @@
+import libCoverage from "./lib-coverage/index.cjs";


This is a "bridge" to expose the common js code into ESM js code, so that the ts file can then import it

ChristopherDedominici · 2025-10-28T13:16:20Z

v-next/hardhat-vendored/package.json

+    "pretest": "pnpm build",
+    "pretest:only": "pnpm build",
+    "build": "tsc --build .",
+    "postbuild": "node dist/src/copy-assets.js",


This step is required to copy the assets folder, which contains the CSS and JS files used to generate the report, into the dist folder after compilation

ChristopherDedominici · 2025-10-28T13:17:08Z

v-next/hardhat-vendored/src/copy-assets.ts

@@ -0,0 +1,33 @@
+import { cp } from "node:fs/promises";


This function is invoked by the postbuild in the package.json file.

Used to copy the assets folder, which contains the CSS and JS files used to generate the report, into the dist folder after compilation

I'm open to better ideas to solve this issue

I tested this PR with verdaccio and there were no issues, so this seems to work fine and it's simple enough.

ChristopherDedominici · 2025-10-28T13:20:25Z

v-next/hardhat-vendored/tsconfig.json

@@ -0,0 +1,13 @@
+{


I need a double-check here. I copied the configuration from tsconfig.base.json, but removed all the properties that would cause errors during compilation, since this folder contains non-ESM files

ChristopherDedominici · 2025-10-28T13:22:54Z

v-next/hardhat/src/internal/builtin-plugins/coverage/coverage-manager.ts

+    report: Report,
+    htmlReportPath: string,
+  ): Promise<void> {
+    const baseDir = process.cwd();


Since there is a bug in how we generate coverage data, we currently only display executed and non-executed lines. The line counter is either 1 for executed lines or 0 for non-executed lines:

This will be improved after fixing how we collect coverage data

…overage

…to html-coverage

ChristopherDedominici · 2025-11-03T15:25:18Z

v-next/hardhat-vendored/test/index.ts

This file is intentionally empty to prevent build errors and avoid changing the default scripts in package.json

This should be a comment inside the file.

…overage

fvictorio · 2025-11-10T09:36:39Z

.changeset/gorgeous-swans-try.md

+"@nomicfoundation/hardhat": minor
+"@nomicfoundation/hardhat-vendored": patch


A couple of things:

@nomicfoundation/hardhat should be hardhat

The changeset for Hardhat should be patch. I don't think this merits a minor bump.

@nomicfoundation/hardhat-vendored is a new package, so it shouldn't be included here. Otherwise its first version will be 3.0.1, not 3.0.0. I checked locally that changeset publish will publish hardhat-vendored even if it doesn't have a changeset, because it detects that it's not published yet. (I'm not entirely sure that the CI will have the same behavior, but I don't see why it wouldn't.)

Suggested change

"@nomicfoundation/hardhat": minor

"@nomicfoundation/hardhat-vendored": patch

"hardhat": patch

fvictorio · 2025-11-10T09:55:52Z

v-next/hardhat/src/internal/builtin-plugins/coverage/coverage-manager.ts

    await writeUtf8File(lcovReportPath, lcovReport);
    log(`Saved lcov report to ${lcovReportPath}`);

+    const htmlReportPath = path.join(this.#coveragePath, "html");


Nyc, jest and vitest all dump the html inside the coverage directory, even when both the HTML and lcov reporters are used together. I personally assume that coverage/index.html will exist and was a bit surprised when it didn't.

I'm not against using a subdirectory for this (in fact, it would be my preference if we were doing this from scratch), but I think it's better to be consistent with other tools here.

Suggested change

const htmlReportPath = path.join(this.#coveragePath, "html");

const htmlReportPath = path.join(this.#coveragePath);

fvictorio · 2025-11-10T09:58:34Z

.changeset/gorgeous-swans-try.md

+"@nomicfoundation/hardhat-vendored": patch
+---
+
+Added HTML coverage reporting and a new internal `hardhat-vendored` package ([7528](https://github.com/NomicFoundation/hardhat/pull/7528)).


Changesets shouldn't include implementation details.

Suggested change

Added HTML coverage reporting and a new internal `hardhat-vendored` package ([7528](https://github.com/NomicFoundation/hardhat/pull/7528)).

Added HTML coverage reporting ([7528](https://github.com/NomicFoundation/hardhat/pull/7528)).

fvictorio · 2025-11-10T10:00:04Z

v-next/hardhat/src/internal/builtin-plugins/coverage/coverage-manager.ts


+    const htmlReportPath = path.join(this.#coveragePath, "html");
+    await this.#writeHtmlReport(report, htmlReportPath);
+    console.log(`Saved html report to ${htmlReportPath}`);


I think you meant to use a debug log here.

Suggested change

console.log(`Saved html report to ${htmlReportPath}`);

log(`Saved html report to ${htmlReportPath}`);

fvictorio · 2025-11-10T10:06:37Z

v-next/hardhat/src/internal/builtin-plugins/coverage/coverage-manager.ts

+    // Construct coverage data for each tested file,
+    // detailing whether each line was executed or not.
+    for (const [p, fileCoverageInput] of Object.entries(report)) {
+      const testedFilePath = path.isAbsolute(p) ? p : path.join(baseDir, p);


Not a blocker, but: I am surprised about this. We generate the report, so we should know if the keys are relative or absolute paths. (The Report interface names the key relativePath, so I guess they are meant to be relative).

Another concern is that, if they are relative, it's unclear whether they are relative to the cwd or the project's root (IMO it should be the latter).

Again: I wouldn't do anything in this PR, but this seems like technical debt. Not sure how/if we are capturing those.

fvictorio · 2025-11-10T10:14:33Z

v-next/hardhat/src/internal/builtin-plugins/coverage/coverage-manager.ts

+      coverageMap,
+    });
+
+    reports.create("html", undefined).execute(context);


Optional parameter with default value.

Suggested change

reports.create("html", undefined).execute(context);

reports.create("html").execute(context);

fvictorio · 2025-11-10T10:15:25Z

v-next/hardhat/src/internal/builtin-plugins/coverage/coverage-manager.ts

+          continue;
+        }
+
+        // TODO: bug line count


What does this mean?

fvictorio · 2025-11-10T10:17:49Z

v-next/hardhat-vendored/package.json

+  "name": "@nomicfoundation/hardhat-vendored",
+  "version": "3.0.0",
+  "description": "Internal dependencies used by Hardhat that have been vendored to prevent bloating the main package",
+  "homepage": "https://github.com/nomicfoundation/hardhat/tree/v-next/v-next/hardhat-vendored",


We are not using the v-next branch anymore.

Suggested change

"homepage": "https://github.com/nomicfoundation/hardhat/tree/v-next/v-next/hardhat-vendored",

"homepage": "https://github.com/nomicfoundation/hardhat/tree/main/v-next/hardhat-vendored",

fvictorio · 2025-11-10T10:19:35Z

v-next/hardhat-vendored/test/index.ts

This should be a comment inside the file.

fvictorio · 2025-11-10T10:27:14Z

v-next/hardhat-vendored/src/copy-assets.ts

@@ -0,0 +1,33 @@
+import { cp } from "node:fs/promises";


I tested this PR with verdaccio and there were no issues, so this seems to work fine and it's simple enough.

fvictorio · 2025-11-10T10:29:50Z

For the record: the way uncovered code looks is not great:

I think this is caused by us using lines instead of statements, or something along those lines. My guess is that we can work our way around this, but I wouldn't do it yet; I'd rather release this sooner.

mvp

731d409

ChristopherDedominici self-assigned this Oct 8, 2025

ChristopherDedominici added this to Hardhat Oct 8, 2025

github-project-automation bot moved this to Backlog in Hardhat Oct 8, 2025

github-actions bot added the status:triaging label Oct 8, 2025

ChristopherDedominici removed the status:triaging label Oct 8, 2025

github-actions bot assigned michalbrabec Oct 8, 2025

ChristopherDedominici moved this from Backlog to In Progress in Hardhat Oct 8, 2025

istanbul libCoverage: copy original js file in root assets folder

4088b56

ChristopherDedominici commented Oct 13, 2025

View reviewed changes

ChristopherDedominici added 18 commits October 16, 2025 17:13

libReport

e3d52e3

report

4dae722

remove dependencies from istanbul in coverage-manager

d002919

add 'node:' to imports

95f8d9a

replace import to istanbul with internal assets

f3e276f

remove unused library

c77eb8d

replace html escape package

7771d9f

replace make-dir with fs/promises

029bc26

remove dependencies from istanbul npm packages

9be67d5

base package

3a55628

move js istanbul libs in the new package

cc5cb6a

add postbuild script and re-organize coverage-manager

1856995

exclude js, cjs, css, png from the gitignore

e646399

rename to hardhat-vendored

3047483

exclude from gitignore

c8ea036

delete test file

39ce891

remove dist folder

c6f1e72

fix package name and imports

f78a224

ChristopherDedominici added 5 commits October 27, 2025 11:53

rename folder to avoid gitignore

b28430a

fix copy-assets

8c92ef9

fix exports after changing folder name

022c6ae

show colored lines in coverage report

a50b3b8

update tsconfig in new vendored package

0b89710

ChristopherDedominici commented Oct 28, 2025

View reviewed changes

minor fixes

50f3b4c

ChristopherDedominici marked this pull request as ready for review October 28, 2025 13:36

Merge branch 'main' of github.com:NomicFoundation/hardhat into html-c…

f3591e7

…overage

ChristopherDedominici moved this from In Progress to In Review in Hardhat Oct 28, 2025

Merge branch 'main' of github.com:NomicFoundation/hardhat into html-c…

d0fba8e

…overage

ChristopherDedominici added the no docs needed This PR doesn't require links to documentation label Nov 3, 2025

ChristopherDedominici added 5 commits November 3, 2025 15:39

lint:fix

3602408

Create gorgeous-swans-try.md

006f834

ignore spelling in vendored package src code

f771d34

Merge branch 'html-coverage' of github.com:NomicFoundation/hardhat in…

7724608

…to html-coverage

remove dependency from @nomicfoundation/hardhat-node-test-reporter

9dde2ee

ChristopherDedominici linked an issue Nov 3, 2025 that may be closed by this pull request

Generate HTML coverage report #6731

Open

ChristopherDedominici added 3 commits November 3, 2025 16:05

add reference to test reporter

7542cdc

Merge branch 'html-coverage' of github.com:NomicFoundation/hardhat in…

8ed50de

…to html-coverage

add hardhat-node-test-reporter in package.json

f264363

ChristopherDedominici commented Nov 3, 2025

View reviewed changes

ChristopherDedominici requested a review from fvictorio November 4, 2025 11:09

Merge branch 'main' of github.com:NomicFoundation/hardhat into html-c…

b26e193

…overage

ChristopherDedominici changed the title ~~[html coverage report]~~ [html coverage report] Add vendored package + generate HTML report Nov 6, 2025

fvictorio reviewed Nov 10, 2025

View reviewed changes

		@@ -0,0 +1,2 @@
		import libCoverage from "./lib-coverage/index.cjs";

		"@nomicfoundation/hardhat": minor
		"@nomicfoundation/hardhat-vendored": patch

	"@nomicfoundation/hardhat": minor
	"@nomicfoundation/hardhat-vendored": patch
	"hardhat": patch

	const htmlReportPath = path.join(this.#coveragePath, "html");
	const htmlReportPath = path.join(this.#coveragePath);

	Added HTML coverage reporting and a new internal `hardhat-vendored` package ([7528](https://github.com/NomicFoundation/hardhat/pull/7528)).
	Added HTML coverage reporting ([7528](https://github.com/NomicFoundation/hardhat/pull/7528)).

	console.log(`Saved html report to ${htmlReportPath}`);
	log(`Saved html report to ${htmlReportPath}`);

	reports.create("html", undefined).execute(context);
	reports.create("html").execute(context);

	"homepage": "https://github.com/nomicfoundation/hardhat/tree/v-next/v-next/hardhat-vendored",
	"homepage": "https://github.com/nomicfoundation/hardhat/tree/main/v-next/hardhat-vendored",

[html coverage report] Add vendored package + generate HTML report #7528

Are you sure you want to change the base?

[html coverage report] Add vendored package + generate HTML report #7528

Uh oh!

Conversation

ChristopherDedominici commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODOs before merge

NOTEs for review

Uh oh!

changeset-bot bot commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions bot commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

hardhat

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fvictorio commented Nov 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ChristopherDedominici commented Oct 8, 2025 •

edited

Loading

changeset-bot bot commented Oct 8, 2025 •

edited

Loading

github-actions bot commented Oct 8, 2025 •

edited

Loading