Better handling of peer scoring

[mralj]

TL;DR:

1. Peers in Gossipsub are not properly scored
2. We should revisit GossipSub scoring thresholds
3. We should merge application-level scoring with Gossipsub scoring
4. We should revisit application-level scoring

Gossipsub scoring

Gossip sub scores peers on different metrics, specs are here. One of these metrics is gossiping of (new) messages. By default JS implementation of Gossip sub expects 10 msg/s per topic, after which it starts to score peer negatively.

Aztec shouldn't expect such a large number of messages per (most) topics (e.g., block proposals & attestations). So we should fine-tune this properly. For example, here is how lodestar calculates these scores.

Thresholds

We should also revisit score thresholds after which GossipSub starts penalizing peers. Here are our thresholds. Which I believe are too lax. For example if the peer is not gossiping messages, the score (by default) will be deceased by -400 most. Given our thresholds (e.g. -14k) - this won't ever negatively impact bad behaving peer.

Disconnecting from peers

Negative GossipSub doesn't imply disconnect from peer, meaning GossipSub will never initiate disconnect. Nor will libp2p until we hit peer limits. Only then is the disconnect initiated. There is sorting logic defined here - how libp2p decides to disconnect. One of the factors is peer tags - one can tag peer with certain score. Bigger the score, the less likely the peer will be disconnected. This is what GossipSub also uses - peers in Mesh will be tagged with some good score. Once peer leaves the mesh, the tag is removed (source).

Still if we are not hitting these limits we could be left with peers which are not actively and altruistically participating in the network. This is why I believe we should proactively disconnect from bad peers on application level - as we already do.

Application scoring

We already score peers on application level (mainily based on Req/Resp) and we are disconnecting from bad peers. We should adjust these scores appropriately keeping in mind the GossipSub threhsolds and scoring AND merge the application-level scoring with GossipSub, as for example lodestar does.
On each heartbeat the bad peers should be disconnected.