Skip to content

Remote Network Forking Improvements #883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Oct 22, 2025
+573 −67
Merged

Remote Network Forking Improvements #883

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
be230a9
Update Flow Emulator Forking Commands
jribbink Oct 13, 2025
42df5c6
Add integraton test
jribbink Oct 14, 2025
98caa17
Add pub key deduplication shim
jribbink Oct 14, 2025
785c6ab
Fix compat shim
jribbink Oct 15, 2025
84ef55e
Optimize RPC requests
jribbink Oct 15, 2025
7c85007
Add Mainnet Test
jribbink Oct 15, 2025
f1fe23e
Add retries for remote store & optimize requests
jribbink Oct 15, 2025
7f7bd63
format
jribbink Oct 15, 2025
103df08
add license
jribbink Oct 15, 2025
c9f0678
cleanup
jribbink Oct 15, 2025
d5c5b30
rename flags & format
jribbink Oct 15, 2025
2c8b4ca
Add ConfigureServer hook
jribbink Oct 15, 2025
351f218
fix emu logs
jribbink Oct 15, 2025
19039a1
update naming
jribbink Oct 15, 2025
bca6b85
switch syntax and reduce code
jribbink Oct 15, 2025
51778f4
cleanup
jribbink Oct 16, 2025
85731ef
tidy logs
jribbink Oct 16, 2025
80ff998
don't retry not found errors as they are part of normal operation
jribbink Oct 16, 2025
02b68b1
add chainid to logs
jribbink Oct 16, 2025
35a02ca
start: remove unused parseFlowChainIDStart helper
jribbink Oct 20, 2025
f348913
remove unused fn
jribbink Oct 20, 2025
da63c1f
immutably set chain ID
jribbink Oct 20, 2025
6e4947f
remove compat shim
jribbink Oct 22, 2025
6198053
disable mainnet fork test until forte release
jribbink Oct 22, 2025
ded5af3
cleanup retries
jribbink Oct 22, 2025
6ac3b1a
simplify retries
jribbink Oct 22, 2025
ed79bba
Merge branch 'master' into jribbink/fork-rebrand
jribbink Oct 22, 2025
c662102
address feedback
jribbink Oct 22, 2025
d5164ff
Merge branch 'jribbink/fork-rebrand' of https://github.com/onflow/flo…
jribbink Oct 22, 2025
228ccb5
add port validation
jribbink Oct 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 6 additions & 8 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ values.
| `--host` | `FLOW_HOST` | ` ` | Host to listen on for emulator GRPC/REST/Admin servers (default: All Interfaces) |
| `--chain-id` | `FLOW_CHAINID` | `emulator` | Chain to simulate, if 'mainnet' or 'testnet' values are used, you will be able to run transactions against that network and a local fork will be created. Valid values are: 'emulator', 'testnet', 'mainnet' |
| `--redis-url` | `FLOW_REDIS_URL` | '' | Redis-server URL for persisting redis storage backend ( `redis://[[username:]password@]host[:port][/database]` ) |
| `--start-block-height` | `FLOW_STARTBLOCKHEIGHT` | `0` | Start block height to use when starting the network using 'testnet' or 'mainnet' as the chain-id |
| `--rpc-host` | `FLOW_RPCHOST` | '' | RPC host (access node) to query for previous state when starting the network using 'testnet' or 'mainnet' as the chain-id |
| `--fork-host` | `FLOW_FORK_HOST` | '' | gRPC access node address (`host:port`) to fork from |
| `--fork-height` | `FLOW_FORK_HEIGHT` | `0` | Block height to pin the fork (defaults to latest sealed) |
| `--legacy-upgrade` | `FLOW_LEGACYUPGRADE` | `false` | Enable upgrading of legacy contracts |
| `--computation-reporting` | `FLOW_COMPUTATIONREPORTING` | `false` | Enable computation reporting for Cadence scripts & transactions |
| `--checkpoint-dir` | `FLOW_CHECKPOINTDIR` | '' | Checkpoint directory to load the emulator state from, if starting the emulator from a checkpoint |
Expand Down Expand Up @@ -155,8 +155,7 @@ Post Data: height={block height}
```

Note: it is only possible to roll back state to a height that was previously executed by the emulator.
To roll back to a past block height when using a forked Mainnet or Testnet network, use the
`--start-block-height` flag.
To pin the starting block height when using a fork, use the `--fork-height` flag.

## Managing emulator state
It's possible to manage emulator state by using the admin API. You can at any point
Expand Down Expand Up @@ -269,15 +268,14 @@ you must specify the network name for the chain ID flag and the RPC host
to connect to.

```
flow emulator --chain-id mainnet --rpc-host access.mainnet.nodes.onflow.org:9000
flow emulator --chain-id mainnet --rpc-host access.devnet.nodes.onflow.org:9000
flow emulator --fork-host access.mainnet.nodes.onflow.org:9000
flow emulator --fork-host access.mainnet.nodes.onflow.org:9000 --fork-height 12345
```

Please note, that the actual execution on the real network may differ depending on the exact state when the transaction is
executed.

By default, the forked network will start from the latest sealed block when the emulator
is started. You can specify a different starting block height by using the `--start-block-height` flag.
By default, the forked network will start from the latest sealed block when the emulator is started. You can specify a different starting block height by using the `--fork-height` flag.

You can also store all of your changes and cached registers to a persistent db by using the `--persist` flag,
along with the other SQLite settings.
Expand Down
82 changes: 57 additions & 25 deletions cmd/emulator/start/start.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,14 +75,18 @@ type Config struct {
SqliteURL string `default:"" flag:"sqlite-url" info:"sqlite db URL for persisting sqlite storage backend "`
CoverageReportingEnabled bool `default:"false" flag:"coverage-reporting" info:"enable Cadence code coverage reporting"`
LegacyContractUpgradeEnabled bool `default:"false" flag:"legacy-upgrade" info:"enable Cadence legacy contract upgrade"`
StartBlockHeight uint64 `default:"0" flag:"start-block-height" info:"block height to start the emulator at. only valid when forking Mainnet or Testnet"`
RPCHost string `default:"" flag:"rpc-host" info:"rpc host to query when forking Mainnet or Testnet"`
ForkHost string `default:"" flag:"fork-host" info:"gRPC access node address (host:port) to fork from"`
ForkHeight uint64 `default:"0" flag:"fork-height" info:"height to pin fork; defaults to latest sealed"`
CheckpointPath string `default:"" flag:"checkpoint-dir" info:"checkpoint directory to load the emulator state from"`
StateHash string `default:"" flag:"state-hash" info:"state hash of the checkpoint to load the emulator state from"`
ComputationReportingEnabled bool `default:"false" flag:"computation-reporting" info:"enable Cadence computation reporting"`
ScheduledTransactionsEnabled bool `default:"true" flag:"scheduled-transactions" info:"enable Cadence scheduled transactions"`
SetupEVMEnabled bool `default:"true" flag:"setup-evm" info:"enable EVM setup for the emulator, this will deploy the EVM contracts"`
SetupVMBridgeEnabled bool `default:"true" flag:"setup-vm-bridge" info:"enable VM Bridge setup for the emulator, this will deploy the VM Bridge contracts"`

// Deprecated hidden aliases
StartBlockHeight uint64 `default:"0" flag:"start-block-height" info:"(deprecated) use --fork-height"`
RPCHost string `default:"" flag:"rpc-host" info:"(deprecated) use --fork-host"`
}

const EnvPrefix = "FLOW"
Expand Down Expand Up @@ -147,31 +151,23 @@ func Cmd(config StartConfig) *cobra.Command {
Exit(1, err.Error())
}

if conf.StartBlockHeight > 0 && flowChainID != flowgo.Mainnet && flowChainID != flowgo.Testnet {
Exit(1, "❗ --start-block-height is only valid when forking Mainnet or Testnet")
// Deprecation shims: map old flags to new and warn
if conf.RPCHost != "" && conf.ForkHost == "" {
logger.Warn().Msg("❗ --rpc-host is deprecated; use --fork-host")
conf.ForkHost = conf.RPCHost
}

if (flowChainID == flowgo.Mainnet || flowChainID == flowgo.Testnet) && conf.RPCHost == "" {
Exit(1, "❗ --rpc-host must be provided when forking Mainnet or Testnet")
if conf.StartBlockHeight > 0 && conf.ForkHeight == 0 {
logger.Warn().Msg("❗ --start-block-height is deprecated; use --fork-height")
conf.ForkHeight = conf.StartBlockHeight
}

serviceAddress := flowsdk.ServiceAddress(flowsdk.ChainID(flowChainID))
if conf.SimpleAddresses {
serviceAddress = flowsdk.HexToAddress("0x1")
// In non-fork mode, fork-only flags are invalid
if conf.ForkHost == "" && (conf.StartBlockHeight > 0 || conf.ForkHeight > 0) {
Exit(1, "❗ --fork-height requires --fork-host")
}

serviceFields := map[string]any{
"serviceAddress": serviceAddress.Hex(),
"servicePubKey": hex.EncodeToString(servicePublicKey.Encode()),
"serviceSigAlgo": serviceKeySigAlgo.String(),
"serviceHashAlgo": serviceKeyHashAlgo.String(),
}

if servicePrivateKey != nil {
serviceFields["servicePrivKey"] = hex.EncodeToString(servicePrivateKey.Encode())
}

logger.Info().Fields(serviceFields).Msgf("⚙️ Using service account 0x%s", serviceAddress.Hex())
// Service account logging is deferred until after server configuration to allow
// higher-level wrappers to customize fork settings via ConfigureServer.

minimumStorageReservation := fvm.DefaultMinimumStorageReservation
if conf.MinimumAccountBalance != "" {
Expand All @@ -183,6 +179,18 @@ func Cmd(config StartConfig) *cobra.Command {
storageMBPerFLOW = parseCadenceUFix64(conf.StorageMBPerFLOW, "storage-per-flow")
}

// Recompute chain ID and service address accurately for fork mode by querying the node.
forkHost := conf.ForkHost
resolvedChainID := flowChainID
forkMode := forkHost != ""
if forkMode {
parsed, err := server.DetectRemoteChainID(forkHost)
if err != nil {
Exit(1, fmt.Sprintf("failed to detect remote chain id from %s: %v", forkHost, err))
}
resolvedChainID = parsed
}

serverConf := &server.Config{
GRPCPort: conf.Port,
GRPCDebug: conf.GRPCDebug,
Expand Down Expand Up @@ -212,13 +220,13 @@ func Cmd(config StartConfig) *cobra.Command {
SkipTransactionValidation: conf.SkipTxValidation,
SimpleAddressesEnabled: conf.SimpleAddresses,
Host: conf.Host,
ChainID: flowChainID,
ChainID: resolvedChainID,
RedisURL: conf.RedisURL,
ContractRemovalEnabled: conf.ContractRemovalEnabled,
SqliteURL: conf.SqliteURL,
CoverageReportingEnabled: conf.CoverageReportingEnabled,
StartBlockHeight: conf.StartBlockHeight,
RPCHost: conf.RPCHost,
ForkHost: conf.ForkHost,
ForkHeight: conf.ForkHeight,
CheckpointPath: conf.CheckpointPath,
StateHash: conf.StateHash,
ComputationReportingEnabled: conf.ComputationReportingEnabled,
Expand All @@ -227,6 +235,24 @@ func Cmd(config StartConfig) *cobra.Command {
SetupVMBridgeEnabled: conf.SetupVMBridgeEnabled,
}

serviceAddress := flowsdk.ServiceAddress(flowsdk.ChainID(resolvedChainID))
if conf.SimpleAddresses {
serviceAddress = flowsdk.HexToAddress("0x1")
}

serviceFields := map[string]any{
"serviceAddress": serviceAddress.Hex(),
"servicePubKey": hex.EncodeToString(servicePublicKey.Encode()),
"serviceSigAlgo": serviceKeySigAlgo.String(),
"serviceHashAlgo": serviceKeyHashAlgo.String(),
}

if servicePrivateKey != nil {
serviceFields["servicePrivKey"] = hex.EncodeToString(servicePrivateKey.Encode())
}

logger.Info().Fields(serviceFields).Msgf("⚙️ Using service account 0x%s", serviceAddress.Hex())

emu := server.NewEmulatorServer(logger, serverConf)
if emu != nil {
for _, middleware := range config.RestMiddlewares {
Expand All @@ -241,6 +267,12 @@ func Cmd(config StartConfig) *cobra.Command {

initConfig(cmd)

// Hide and deprecate legacy flags while keeping them functional
_ = cmd.PersistentFlags().MarkHidden("rpc-host")
_ = cmd.PersistentFlags().MarkDeprecated("rpc-host", "use --fork-host")
_ = cmd.PersistentFlags().MarkHidden("start-block-height")
_ = cmd.PersistentFlags().MarkDeprecated("start-block-height", "use --fork-height")

return cmd
}

Expand Down
14 changes: 6 additions & 8 deletions docs/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ values.
| `--host` | `FLOW_HOST` | ` ` | Host to listen on for emulator GRPC/REST/Admin servers (default: all interfaces) |
| `--chain-id` | `FLOW_CHAINID` | `emulator` | Chain to emulate for address generation. Valid values are: 'emulator', 'testnet', 'mainnet' |
| `--redis-url` | `FLOW_REDIS_URL` | '' | Redis-server URL for persisting redis storage backend ( `redis://[[username:]password@]host[:port][/database]` ) |
| `--start-block-height` | `FLOW_STARTBLOCKHEIGHT` | `0` | Start block height to use when starting the network using 'testnet' or 'mainnet' as the chain-id |
| `--rpc-host` | `FLOW_RPCHOST` | '' | RPC host (access node) to query for previous state when starting the network using 'testnet' or 'mainnet' as the chain-id |
| `--fork-host` | `FLOW_FORK_HOST` | '' | gRPC access node address (`host:port`) to fork from |
| `--fork-height` | `FLOW_FORK_HEIGHT` | `0` | Block height to pin the fork (defaults to latest sealed) |

## Running the emulator with the Flow CLI

Expand Down Expand Up @@ -149,8 +149,7 @@ Post Data: height={block height}
```

Note: it is only possible to roll back state to a height that was previously executed by the emulator.
To roll back to a past block height when using a forked Mainnet or Testnet network, use the
`--start-block-height` flag.
To pin the starting block height when using a fork, use the `--fork-height` flag.

## Managing emulator state

Expand Down Expand Up @@ -246,14 +245,13 @@ you must specify the network name for the chain ID flag as well as the RPC host
to connect to.

```
flow emulator --chain-id mainnet --rpc-host access-008.mainnet24.nodes.onflow.org:9000
flow emulator --chain-id mainnet --rpc-host access-002.devnet49.nodes.onflow.org:9000
flow emulator --fork-host access.mainnet.nodes.onflow.org:9000
flow emulator --fork-host access.mainnet.nodes.onflow.org:9000 --fork-height 12345
```

Please note, the actual execution on the real network may differ depending on the exact state when the transaction is executed.

By default, the forked network will start from the latest sealed block when the emulator
is started. You can specify a different starting block height by using the `--start-block-height` flag.
By default, the forked network will start from the latest sealed block when the emulator is started. You can specify a different starting block height by using the `--fork-height` flag.

You can also store all of your changes and cached registers to a persistent db by using the `--persist` flag,
along with the other sqlite settings.
Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ require (
github.com/google/go-dap v0.11.0
github.com/gorilla/mux v1.8.1
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/improbable-eng/grpc-web v0.15.0
github.com/logrusorgru/aurora v2.0.3+incompatible
github.com/onflow/cadence v1.8.1
Expand Down Expand Up @@ -94,7 +95,6 @@ require (
github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/holiman/bloomfilter/v2 v2.0.3 // indirect
github.com/holiman/uint256 v1.3.2 // indirect
Expand Down
Loading
Loading