nix flake check: Skip substitutable derivations

Since `nix flake check` doesn't produce a `result` symlink, it doesn't
actually need to build/substitute derivations that are already known
to have succeeded, i.e. that are substitutable.

This can speed up CI jobs in cases where the derivations have already
been built by other jobs. For instance, a command like

  nix flake check github:NixOS/hydra/aa62c7f7db31753f0cde690f8654dd1907fc0ce2

should no longer build anything because the outputs are already in
cache.nixos.org.

Based-on: DeterminateSystems#134
Based-on: https://gerrit.lix.systems/c/lix/+/3841
Co-authored-by: Eelco Dolstra <[email protected]>

Author: 2 people

doc/manual/rl-next/faster-nix-flake-check.md

@@ -0,0 +1,9 @@
+---
+synopsis: "`nix flake check` now skips derivations that can be substituted"
+prs: [13574]
+---
+
+Previously, `nix flake check` would evaluate and build/substitute all
+derivations. Now, it will skip downloading derivations that can be substituted.
+This can drastically decrease the time invocations take in environments where
+checks may already be cached (like in CI).

src/nix/flake.cc

@@ -786,8 +786,32 @@ struct CmdFlakeCheck : FlakeCommand
         }
 
         if (build && !drvPaths.empty()) {
-            Activity act(*logger, lvlInfo, actUnknown, fmt("running %d flake checks", drvPaths.size()));
-            store->buildPaths(drvPaths);
+            // TODO: This filtering of substitutable paths is a temporary workaround until
+            // https://github.com/NixOS/nix/issues/5025 (union stores) is implemented.
+            //
+            // Once union stores are available, this code should be replaced with a proper
+            // union store configuration. Ideally, we'd use a union of multiple destination
+            // stores to preserve the current behavior where different substituters can
+            // cache different check results.
+            //
+            // For now, we skip building derivations whose outputs are already available
+            // via substitution, as `nix flake check` only needs to verify buildability,
+            // not actually produce the outputs.
+            auto missing = store->queryMissing(drvPaths);
+            // Only occurs if `drvPaths` contains a `DerivedPath::Opaque`, which should never happen
+            assert(missing.unknown.empty());
+
+            std::vector<DerivedPath> toBuild;
+            for (auto & path : missing.willBuild) {
+                toBuild.emplace_back(
+                    DerivedPath::Built{
+                        .drvPath = makeConstantStorePathRef(path),
+                        .outputs = OutputsSpec::All{},
+                    });
+            }
+
+            Activity act(*logger, lvlInfo, actUnknown, fmt("running %d flake checks", toBuild.size()));
+            store->buildPaths(toBuild);
         }
         if (hasErrors)
             throw Error("some errors were encountered during the evaluation");

tests/functional/flakes/check.sh

@@ -167,3 +167,23 @@ EOF
 if !isTestOnNixOS && $NIX_REMOTE != daemon; then
     expectStderr 100 nix flake check "$flakeDir" | grepQuiet 'builder failed with exit code 1'
 fi
+
+# Ensure non-substitutable (read: usually failed) checks are actually run
+# https://github.com/NixOS/nix/pull/13574
+cp "$config_nix" $flakeDir/
+cat > $flakeDir/flake.nix <<EOF
+{
+  outputs = { self }: with import ./config.nix; {
+    checks.${system}.expectedToFail = derivation {
+      name = "expected-to-fail";
+      inherit system;
+      builder = "not-a-real-file";
+    };
+  };
+}
+EOF
+
+# NOTE: Regex pattern is used for compatibility with older daemon versions
+# We also can't expect a specific status code. Earlier daemons return 1, but as of 2.31, we return 100
+checkRes=$(nix flake check $flakeDir 2>&1 && fail "nix flake check should have failed" || true)
+echo "$checkRes" | grepQuiet -E "builder( for .*)? failed with exit code 1"