The 'beacon-chain' container stops and no longer works

[lzmrd]

Description

I'm trying to setup a private network through this repo. It works for me with this docker-compose.yml:

version: "3.9"
services:
  create-beacon-chain-genesis:
    image: "gcr.io/prysmaticlabs/prysm/cmd/prysmctl:HEAD-aec349"
    command:
      - testnet
      - generate-genesis
      - --fork=capella
      - --num-validators=64
      - --genesis-time-delay=15
      - --output-ssz=/consensus/genesis.ssz
      - --chain-config-file=/consensus/config.yml
      - --geth-genesis-json-in=/execution/genesis.json
      - --geth-genesis-json-out=/execution/genesis.json
    volumes:
      - ./consensus:/consensus
      - ./execution:/execution

  geth-remove-db:
    image: "alpine:3.19.0"
    command: rm -rf /execution/geth
    volumes:
      - ./execution:/execution

  geth-genesis:
    image: "ethereum/client-go:v1.14.3"
    command: --datadir=/execution init /execution/genesis.json
    volumes:
      - ./execution:/execution
      - ./execution/genesis.json:/execution/genesis.json
    depends_on:
      create-beacon-chain-genesis:
        condition: service_completed_successfully
      geth-remove-db:
        condition: service_completed_successfully

  beacon-chain:
    image: "gcr.io/prysmaticlabs/prysm/beacon-chain:v4.1.1"
    command:
      - --datadir=/consensus/beacondata
      - --min-sync-peers=0
      - --genesis-state=/consensus/genesis.ssz
      - --bootstrap-node=
      - --interop-eth1data-votes
      - --chain-config-file=/consensus/config.yml
      - --contract-deployment-block=0
      - --chain-id=${CHAIN_ID:-32382}
      - --rpc-host=0.0.0.0
      - --grpc-gateway-host=0.0.0.0
      - --execution-endpoint=http://geth:8551
      - --accept-terms-of-use
      - --jwt-secret=/execution/jwtsecret
      - --suggested-fee-recipient=0x123463a4b065722e99115d6c222f267d9cabb524
      - --minimum-peers-per-subnet=0
      - --enable-debug-rpc-endpoints
      - --force-clear-db
    depends_on:
      create-beacon-chain-genesis:
        condition: service_completed_successfully
    ports:
      - 4000:4000
      - 3500:3500
      - 8080:8080
      - 6060:6060
      - 9090:9090
    volumes:
      - ./consensus:/consensus
      - ./execution:/execution
      - ./execution/jwtsecret:/execution/jwtsecret

  geth:
    image: "ethereum/client-go:v1.14.3"
    command:
      - --http
      - --http.api=eth,net,web3
      - --http.addr=0.0.0.0
      - --http.corsdomain=*
      - --ws
      - --ws.api=eth,net,web3
      - --ws.addr=0.0.0.0
      - --ws.origins=*
      - --authrpc.vhosts=*
      - --authrpc.addr=0.0.0.0
      - --authrpc.jwtsecret=/execution/jwtsecret
      - --datadir=/execution
      - --allow-insecure-unlock
      - --nodiscover
      - --syncmode=full
    ports:
      - 8551:8551
      - 8545:8545
      - 8546:8546
    depends_on:
      geth-genesis:
        condition: service_completed_successfully
      beacon-chain:
        condition: service_started
    volumes:
      - ./execution:/execution
      - ./execution/jwtsecret:/execution/jwtsecret
      - ./execution/geth_password.txt:/execution/geth_password.txt

  validator:
    image: "gcr.io/prysmaticlabs/prysm/validator:v4.1.1"
    command:
      - --beacon-rpc-provider=beacon-chain:4000
      - --datadir=/consensus/validatordata
      - --accept-terms-of-use
      - --interop-num-validators=64
      - --interop-start-index=0
      - --chain-config-file=/consensus/config.yml
      - --force-clear-db
    depends_on:
      beacon-chain:
        condition: service_started
    volumes:
      - ./consensus:/consensus

Version

"com.docker.compose.version": "2.28.1",
"desktop.docker.io/wsl-distro": "Ubuntu-22.04",

Present Behaviour

The chain works, but I have a problem with "beacon-chain". If "geth" or "validator" stop, I can start them again with no problems for the chain. The issue comes when "beacon-chain" stops. I can start it again but it is no longer working.
Here are some logs. Beacon-chain:

2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/5af31293/beacon_attestation_2/ssz_snappy"
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=warning msg="numPerSlot is 0, skipping initializing topic scoring" prefix=p2p
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/5af31293/beacon_attestation_13/ssz_snappy"
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=warning msg="numPerSlot is 0, skipping initializing topic scoring" prefix=p2p
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/5af31293/beacon_attestation_39/ssz_snappy"
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=warning msg="numPerSlot is 0, skipping initializing topic scoring" prefix=p2p
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/5af31293/beacon_attestation_41/ssz_snappy"
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=warning msg="numPerSlot is 0, skipping initializing topic scoring" prefix=p2p
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/5af31293/beacon_attestation_25/ssz_snappy"
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=warning msg="numPerSlot is 0, skipping initializing topic scoring" prefix=p2p
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=info msg="Subscribed to topic" prefix=sync topic="/eth2/5af31293/beacon_attestation_5/ssz_snappy"
2024-08-01 12:14:08 time="2024-08-01 10:14:08" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=249.293532ms slot=7
2024-08-01 12:14:12 time="2024-08-01 10:14:12" level=error msg="Beacon node is not respecting the follow distance" prefix=powchain
2024-08-01 12:14:20 time="2024-08-01 10:14:20" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=212.270759ms slot=8
2024-08-01 12:14:26 time="2024-08-01 10:14:26" level=error msg="Beacon node is not respecting the follow distance" prefix=powchain
2024-08-01 12:14:32 time="2024-08-01 10:14:32" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=241.894229ms slot=9
2024-08-01 12:14:40 time="2024-08-01 10:14:40" level=error msg="Beacon node is not respecting the follow distance" prefix=powchain
2024-08-01 12:14:44 time="2024-08-01 10:14:44" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=193.18842ms slot=10
2024-08-01 12:14:44 time="2024-08-01 10:14:44" level=info msg="Peer summary" activePeers=0 inbound=0 outbound=0 prefix=p2p
2024-08-01 12:14:54 time="2024-08-01 10:14:54" level=error msg="Beacon node is not respecting the follow distance" prefix=powchain
2024-08-01 12:14:56 time="2024-08-01 10:14:56" level=info msg="Begin building block" prefix="rpc/validator" sinceSlotStartTime=212.464331ms slot=11

geth:

2024-08-01 12:13:32 INFO [08-01|10:13:32.260] Starting work on payload                 id=0x02074a132ecfd0ab
2024-08-01 12:13:32 INFO [08-01|10:13:32.260] Updated payload                          id=0x02074a132ecfd0ab number=5 hash=f7e875..84a09d txs=0 withdrawals=0 gas=0 fees=0 root=c1d76f..66d1da elapsed="45.481µs"
2024-08-01 12:13:44 INFO [08-01|10:13:44.260] Stopping work on payload                 id=0x02074a132ecfd0ab reason=timeout
2024-08-01 12:13:48 INFO [08-01|10:13:48.434] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=1m4s have=4
2024-08-01 12:13:56 INFO [08-01|10:13:56.237] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=1m12s have=4
2024-08-01 12:14:08 INFO [08-01|10:14:08.269] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=1m24s have=4
2024-08-01 12:14:20 INFO [08-01|10:14:20.236] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=1m36s have=4
2024-08-01 12:14:32 INFO [08-01|10:14:32.260] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=1m48s have=4
2024-08-01 12:14:44 INFO [08-01|10:14:44.223] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=2m    have=4
2024-08-01 12:14:56 INFO [08-01|10:14:56.237] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=2m12s have=4
2024-08-01 12:15:08 INFO [08-01|10:15:08.297] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=2m24s have=4
2024-08-01 12:15:20 INFO [08-01|10:15:20.208] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=2m36s have=4
2024-08-01 12:15:32 INFO [08-01|10:15:32.228] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=2m48s have=4
2024-08-01 12:15:44 INFO [08-01|10:15:44.291] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=3m    have=4
2024-08-01 12:15:56 INFO [08-01|10:15:56.273] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=3m12s have=4
2024-08-01 12:16:08 INFO [08-01|10:16:08.563] Ignoring beacon update to old head       number=0 hash=3c6b8d..8f4e78 age=3m24s have=4

validator:

2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=10 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=1 timeSinceSlotStart=8.150238418s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=36 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=8.152173963s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=53 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.197258545s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=13 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=8.198664086s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=30 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=0 timeSinceSlotStart=8.1987165s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=4 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=0 timeSinceSlotStart=8.197216077s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=52 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=8.200038784s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=21 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.200085016s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=57 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=8.200110836s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=7 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.200131838s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=26 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.200154891s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=27 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.200174679s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=41 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=0 timeSinceSlotStart=8.200193673s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=19 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.200242799s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=11 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=8.200627415s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=24 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.201468752s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=23 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=8.20527321s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=35 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=0 timeSinceSlotStart=8.207922973s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=4 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.217745009s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=35 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.222992142s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=4 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.231331409s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=35 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=2 timeSinceSlotStart=8.237511796s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=35 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=8.251014384s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new sync contribution and proof" aggregatorIndex=35 bitsCount=128 blockRoot=0xe2c59e6a62c8 prefix=validator slot=16 slotStartTime=2024-08-01 10:15:56 +0000 UTC subcommitteeIndex=3 timeSinceSlotStart=8.262636818s
2024-08-01 12:16:04 time="2024-08-01 10:16:04" level=info msg="Submitted new attestations" AggregatorIndices=[29] AttesterIndices=[46 20 55 63 29 18 23 15 37 44 56] BeaconBlockRoot=0xe2c59e6a62c8 CommitteeIndex=0 Slot=16 SourceEpoch=0 SourceRoot=0x000000000000 TargetEpoch=2 TargetRoot=0xe2c59e6a62c8 prefix=validator

Steps to resolve

What I'd like to do is to modify the docker-compose.yml file in order to add couple of "beacon-chain"s more. In this way if one of them goes down there are others to follow the chain. Unfortunatly I'm a newbie with Docker and I cannot set it up.
Can someone trying to help me in that, or at least, tell me if it is a possible configuration?

[0xWilliamWang]

add couple of "beacon-chain"s more
You can initialize el data on the host first, then start el-cl. eg:

init() {
    initCL
    initEL
    return
}

initCL() {
    rm -rf ./conf/consensus/genesis.ssz*
    prysmctl \
        testnet \
        generate-genesis \
        --chain-config-file=./conf/consensus/config.yml \
        --deposit-json-file=./deposit.json \
        --config-name=b2hub \
        --num-validators=64 \
        --genesis-time=0x0 \
        --genesis-time-delay=20 \
        --geth-genesis-json-in=./conf/execution/genesis.json \
        --geth-genesis-json-out=./conf/execution/genesis.json \
        --output-ssz=./conf/consensus/genesis.ssz \
        --output-json=./conf/consensus/genesis.ssz.json \
        --fork=deneb
    return
}

initEL() {
    geth version
    for item in $(seq -f 'node%g' 1 $NODE_NUM); do
        geth --datadir=./l1-tmp-data/$item/execution \
            init \
            ./conf/execution/genesis.json
        mkdir -p l1-tmp-data/$item/consensus/beacon
    done
    return
}