Do you expect tests to pass when built in a Nix sandbox?

[grahamc]

Today on v1.0.0, a number of tests fail while building on 20.09:

=== CONT  TestBackend_PathTokenWriteCreate/HappyPath
    path_token_test.go:65: assertion failed: error is not nil: unable to create access token: dial tcp 127.0.0.1:43155: i/o timeout
--- FAIL: TestBackend_PathTokenWriteCreate (0.00s)
    --- PASS: TestBackend_PathTokenWriteCreate/FailedValidation (0.00s)
    --- PASS: TestBackend_PathTokenWriteCreate/FailedClient (0.00s)
    --- PASS: TestBackend_PathTokenWriteCreate/FailedOptionsParsing (0.00s)
    --- PASS: TestBackend_PathTokenWriteCreate/FailedCreate (30.00s)
    --- FAIL: TestBackend_PathTokenWriteCreate/HappyPath (30.00s)
=== CONT  TestBackend_PathTokenWriteRead/HappyPath
    path_token_test.go:65: assertion failed: error is not nil: unable to create access token: dial tcp 127.0.0.1:45323: i/o timeout
--- FAIL: TestBackend_PathTokenWriteRead (0.00s)
    --- PASS: TestBackend_PathTokenWriteRead/FailedValidation (0.00s)
    --- PASS: TestBackend_PathTokenWriteRead/FailedOptionsParsing (0.00s)
    --- PASS: TestBackend_PathTokenWriteRead/FailedClient (0.00s)
    --- FAIL: TestBackend_PathTokenWriteRead/HappyPath (30.01s)
    --- PASS: TestBackend_PathTokenWriteRead/FailedCreate (30.01s)
=== CONT  TestClient_Token/HappyPath
    client_test.go:236: assertion failed: error is not nil: unable to create access token: dial tcp 127.0.0.1:46419: i/o timeout
=== CONT  TestClient_Token/HappyPathWithTokenConstraints
    client_test.go:236: assertion failed: error is not nil: unable to create access token: dial tcp 127.0.0.1:34119: i/o timeout
=== CONT  TestClient_Token/EmptyResponse
    client_test.go:234: assertion failed: expected error to contain "unable to decode access token response", got "unable to create access token: dial tcp 127.0.0.1:35053: i/o timeout"
=== CONT  TestClient_Token/EOFResponse
    client_test.go:234: assertion failed: expected error to contain "unable to decode access token response", got "unable to create access token: dial tcp 127.0.0.1:36225: i/o timeout"
--- FAIL: TestClient_Token (0.00s)
    --- PASS: TestClient_Token/NilContext (0.00s)
    --- FAIL: TestClient_Token/HappyPath (30.00s)
    --- PASS: TestClient_Token/4xxResponse (30.00s)
    --- FAIL: TestClient_Token/EmptyResponse (30.00s)
    --- FAIL: TestClient_Token/HappyPathWithTokenConstraints (30.00s)
    --- FAIL: TestClient_Token/EOFResponse (30.00s)
FAIL
FAIL    github.com/martinbaillie/vault-plugin-secrets-github/github     60.019s
FAIL

with:

  vault-plugin-secrets-github = buildGoModule {
    name = "vault-plugin-secrets-github";
    version = "1.0.0";
    src = fetchFromGitHub {
      owner = "martinbaillie";
      repo = "vault-plugin-secrets-github";
      rev = "v1.0.0";
      sha256 = "sha256-zWU5qqyRmtj49wdx3bWp7WWFNUqXYiVJ38JjiHeSBs4=";
    };
    vendorSha256 = "sha256-fkp2JAndhwx5xaChf7VR6RfjerB4l093ijNJGS0NN28=";
    buildFlagsArray = [ "-ldflags=-s -w -X github.com/martinbaillie/vault-plugin-secrets-github/github.projectName=martinbaillie/vault-plugin-secrets-github" ];
    dontStrip = true;
  };

adding doCheck = false helps it pass, of course.

[cole-h]

I'd imagine it's because they try to network (e.g. --sandbox false would see these tests succeed), even if just with localhost. Best course of action to resolve this, however, I don't know. Does Go have a way to exclude tests from being run like e.g. Python packages in Nixpkgs do?

[martinbaillie]

Yep that's right, there's some use of Go's httptest package in the tests which would break in Nix sandbox. I must admit I've used Nix to build this before but not with sandboxing enabled so never noticed.

I guard "integration" tests (those that test full E2E integration with a real Vault) with a Go flag +integration. The use of that httptest package is arguably "integration" too depending on how tight your definition is. I'd have no qualms guarding them by the same flag or something else like +network. It's something I can maybe look at in the next week or so.