short lived leader election

[freddyrios]

what could cause LeaderChange to only report in one node/server that it was elected leader temporarily?

The cluster is started with ColdStart false and all the nodes in the cluster configuration are already in place before calling StartAsync.

I understand some terms to elect a leader, but what confuses me is the self reported elected leader not resulting in a LeaderChange event in the other servers. After all, it should only be possible to elect a leader with a majority vote.

Here is a sample of logged leader changes reported by the different nodes

2023-05-31 14:44:04.0426|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.25:4250 - Term 1 - Election timeout 00:00:01.5070000
2023-05-31 14:44:04.2110|WARN|leaderchange|Consensus cannot be reached - Term 1 - Election timeout 00:00:01.5070000                                       
2023-05-31 14:44:04.4681|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:01.5070000
--
2023-05-31 14:44:04.4596|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:01.5370000
--
2023-05-31 14:44:04.4570|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:02.1550000
--
2023-05-31 14:44:04.4586|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:01.5400000
--
2023-05-31 14:44:04.4852|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:01.7180000
--
2023-05-31 14:44:04.3801|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:01.7220000
--
2023-05-31 14:44:04.4855|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:01.8810000
--
2023-05-31 14:44:04.4907|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:02.2520000
--
2023-05-31 14:44:04.4988|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:02.9170000
--
2023-05-31 14:44:04.5081|WARN|leaderchange|New cluster leader is elected - Leader address 192.168.100.46:4250 - Term 2 - Election timeout 00:00:02.8680000

[sakno]

Is it reproducible on localhost?

[freddyrios]

I have not got an isolated repro yet. Looking at that next.

[freddyrios]

I was able to reproduce it with the example project. The only change to the project was adding more nodes:

        builder.Add(new IPEndPoint(IPAddress.Loopback, 3260));
        builder.Add(new IPEndPoint(IPAddress.Loopback, 3261));
        builder.Add(new IPEndPoint(IPAddress.Loopback, 3262));
        builder.Add(new IPEndPoint(IPAddress.Loopback, 3263));
        builder.Add(new IPEndPoint(IPAddress.Loopback, 3264));
        builder.Add(new IPEndPoint(IPAddress.Loopback, 3265));
        builder.Add(new IPEndPoint(IPAddress.Loopback, 3266));
        builder.Add(new IPEndPoint(IPAddress.Loopback, 3267));
        builder.Add(new IPEndPoint(IPAddress.Loopback, 3268));
        builder.Add(new IPEndPoint(IPAddress.Loopback, 3269));

I built the latest in main with the above change and added a "cluster.bat" file with this to the output folder:

START /B .\RaftNode.exe tcp 3260 node0 > node0.log
START /B .\RaftNode.exe tcp 3261 node1 > node1.log
START /B .\RaftNode.exe tcp 3262 node2 > node2.log
START /B .\RaftNode.exe tcp 3263 node3 > node3.log
START /B .\RaftNode.exe tcp 3264 node4 > node4.log
START /B .\RaftNode.exe tcp 3265 node5 > node5.log
START /B .\RaftNode.exe tcp 3266 node6 > node6.log
START /B .\RaftNode.exe tcp 3267 node7 > node7.log
START /B .\RaftNode.exe tcp 3268 node8 > node8.log
START /B .\RaftNode.exe tcp 3269 node9 > node9.log

Then ran: del -r node* && .\cluster.bat

Takes me 2-6 attempts to reproduce.

Here is what the failed leader had in its log:

New cluster leader is elected. Leader address is 127.0.0.1:3260
Term of local cluster member is 1. Election timeout 00:00:00.1590000
Consensus cannot be reached
Term of local cluster member is 1. Election timeout 00:00:00.1590000
New cluster leader is elected. Leader address is 127.0.0.1:3267
Term of local cluster member is 2. Election timeout 00:00:00.1590000
Accepting value 500
Accepting value 1000
Accepting value 1500
Accepting value 2000
...

All other nodes had (and node7 had the below + the normal save messages)

New cluster leader is elected. Leader address is 127.0.0.1:3267
Term of local cluster member is 2. Election timeout 00:00:00.1700000
Accepting value 500
Accepting value 1000
Accepting value 1500
Accepting value 2000
...

[freddyrios]

I was able to reproduce it with the 3 nodes configuration in the same amount of attempts

[freddyrios]

since there is straightforward repro now, have moved it to an issue #168