Running multiple instances in same test

[jbesraa]

The following change introduced in 0.6 or 0.7 makes it tricky for https://github.com/stratum-mining to run multiple tests in parallel or use multiple nodes in a test because it relies on unique datadir and port for each node. Whats the best way to achieve that(running multiple instances simultaneously) with latest corepc version?

args: Vec<&'a str>

Bitcoind command line arguments containing no spaces like vec!["-dbcache=300", "-regtest"] note that port, rpcport, connect, datadir, listen cannot be used because they are automatically initialized.

[0xZaddyy]

looking at this i will suggest creating a function that dynamically creates a new unique configuration for each instance of the new node. This will ensure multiple nodes run concurrently without resources conflicting.

[jbesraa]

I actually just noticed it does not run even with a single node with our current code

To reproduce:

1. git clone git@github.com:stratum-mining/stratum.git
2. cd test/integration-tests
3. cargo test --test 'sv2_mining_device' -- --nocapture    // or any other test instead of sv2_mining_device

>>Failed to start bitcoind due to JSON-RPC error: RPC error response: RpcError { code: -35, message: "Wallet \"43035\" is already loaded.", data: None }
>>Error: Cannot obtain a lock on directory ../stratum/test/integration-tests/template-provider/.bitcoin-43035/regtest. Bitcoin Core is probably already running

Will try to look into this when I have more time.

[0xZaddyy]

i want to look into this
when i run
cargo test --test 'sv2_mining_device'
test gets to pass here, is there anything i'm missing?

[tcharding]

Thanks for raising the issue. I'm struggling to understand what the exact bug report is? Is it one of these?

• node doesn't work as expected with 'standard' usage
• node doesn't work as expected with some custom usage
• Some RPC call fails unexpectedly

[jbesraa]

Apologies, my previous comment were written in a hurry.

i want to look into this when i run cargo test --test 'sv2_mining_device' test gets to pass here, is there anything i'm missing?

In order to reproduce the above error, you need to upgrade corepc to 0.7.0 as shown here

Thanks for raising the issue. I'm struggling to understand what the exact bug report is? Is it one of these?

* `node` doesn't work as expected with 'standard' usage

* `node` doesn't work as expected with some custom usage

* Some RPC call fails unexpectedly

Our config has deviations from the standard as can be seen here. This issue was opened because we are not able to upgrade from 0.5.0 to 0.7.0. We have tests that are running a single node and others running multiple nodes; neither of these scenarios is working. logs0 logs1

[jbesraa]

IIUC what is happening, corepc does not allow running a separate binary, by default it will run bitcoin version 17. Setting default-features = false is not supported anymore.

It would be nice to have a feature flag that allows crates to download their own bitcoin binary(in our case we need a forked bitcoin core that support Stratum V2), and corepc can also export flate2, minireq and tar when this feature is set.

[jbesraa]

Here is the error that is printed from corepc

corepc/node/src/lib.rs:425:37] e = JsonRpc(
    Json(
        Error("missing field `warning`", line: 1, column: 16),
    ),
)

removing the following line solved the problem

[tcharding]

Ok I see what is going on. Note that in the manifest of v0.5.0 it says

# - `cargo test --no-default-features` uses `v28_0` also.
[features]
default = ["28_0"]

You guys were using no-default-features so were implicitly testing with v28.

In v0.7.0 we changed the default to 0.17 so this is why your tests are failing. You can fix this by enabling 28_0. You also need to explicitly enable download now as well.

With all that your corepc-node dependency should be

corepc-node = { version = "0.7.0", default-features = false, features = ["download", "28_0"] }

(Equally you can leave out default-features = false because 28_0 overrides the default.)

I'm running tests locally now but I'd bet my hat that is the problem.

EDIT: My build is hanging :(

[tcharding]

FYI this change came about because of #40 and was implemented in #53.

[tcharding]

Maybe the default download endpoint is a problem?

The address used (https://github.com/Sjors/bitcoin/releases/download) returns a 404

[tcharding]

Test ran locally with suggested change. Thanks for raising the issue, sorry about the confusing changes. And thanks for using this repo!

[jbesraa]

Maybe the default download endpoint is a problem?

The address used (https://github.com/Sjors/bitcoin/releases/download) returns a 404

We construct the full url in the code like the following

                    let download_endpoint =
                        env::var("BITCOIND_DOWNLOAD_ENDPOINT").unwrap_or_else(|_| {
                            "https://github.com/Sjors/bitcoin/releases/download".to_owned()
                        });
                    let url = format!(
                        "{}/sv2-tp-{}/{}",
                        download_endpoint, VERSION_TP, download_filename
                    );

Ok I see what is going on. Note that in the manifest of v0.5.0 it says

• cargo test --no-default-features uses v28_0 also.
  [features]
  default = ["28_0"]

You guys were using no-default-features so were implicitly testing with v28.

In v0.7.0 we changed the default to 0.17 so this is why your tests are failing. You can fix this by enabling 28_0. You also need to explicitly enable download now as well.

With all that your corepc-node dependency should be

corepc-node = { version = "0.7.0", default-features = false, features = ["download", "28_0"] }

(Equally you can leave out default-features = false because 28_0 overrides the default.)

I'm running tests locally now but I'd bet my hat that is the problem.

EDIT: My build is hanging :(

Yea my build was also hanging with features = ["download", "28_0"]. But it does work fine with only "28_0". Thank you!

[tcharding]

I assume the download just take ages, I didn't investigate why.