Skip to content
This repository was archived by the owner on Oct 13, 2025. It is now read-only.

Commit b4b6b99

Browse files
awelzelawelzel
committed
devel: Add notes about the ZeroMQ cluster backend implementation
1 parent 586c051 commit b4b6b99

File tree

10 files changed

+367
-0
lines changed

10 files changed

+367
-0
lines changed

‎devel/cluster-backend-zeromq.rst‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
.. _cluster_backend_zeromq:
2+
3+
======================
4+
ZeroMQ Cluster Backend
5+
======================
6+
7+
.. versionadded:: 7.1
8+
9+
*Experimental*
10+
11+
Quickstart
12+
==========
13+
14+
To switch a Zeek cluster with a static cluster layout over to use ZeroMQ
15+
as cluster backend, add the following snippet to ``local.zeek``:
16+
17+
.. code-block:: zeek
18+
19+
@load frameworks/cluster/backend/zeromq/connect
20+
21+
22+
Note that the function :zeek:see:`Broker::publish` will be non-functional
23+
and a warning emitted when used - use :zeek:see:`Cluster::publish` instead.
24+
25+
By default, a configuration based on hard-coded endpoints and cluster layout
26+
information is created. For more customization, refer to the module documentation
27+
at :doc:`cluster/backend/zeromq/main.zeek </scripts/policy/frameworks/cluster/backend/zeromq/main.zeek>`.
28+
29+
30+
Architecture
31+
============
32+
33+
Publish-Subscribe of Zeek Events
34+
--------------------------------
35+
36+
The `ZeroMQ <https://zeromq.org/>`_ based cluster backend uses a central
37+
XPUB/XSUB broker for publish-subscribe functionality. Zeek events published
38+
via :zeek:see:`Cluster::publish` are distributed by this central broker to
39+
interested nodes.
40+
41+
.. figure:: /images/cluster/zeromq-pubsub.png
42+
43+
44+
As depicted in the figure above, each cluster node connects to the central
45+
broker twice, once via its XPUB socket and once via its XSUB socket. This
46+
results in two TCP connections from every cluster node to the central broker.
47+
This setup allows every node in the cluster to see messages from all other
48+
nodes, avoiding the need for cluster topology awareness.
49+
50+
.. note::
51+
52+
Scalability of the central broker in production setups, but for small
53+
clusters on a single node, may be fast enough.
54+
55+
On a cluster node, the XPUB socket provides notifications about subscriptions
56+
created by other nodes: For every subscription created by any node in
57+
the cluster, the :zeek:see:`Cluster::Backend::ZeroMQ::subscription` event is
58+
raised locally on every other node (unless another node had created the same
59+
subscription previously).
60+
61+
This mechanism is used to discover the existence of other cluster nodes by
62+
matching the topics with the prefix for node specific subscriptions as produced
63+
by :zeek:see:`Cluster::nodeid_topic`.
64+
65+
As of now, the implementation of the central broker calls ZeroMQ's
66+
``zmq::proxy()`` function to forward messages between the XPUB and
67+
XSUB socket.
68+
69+
While the diagram above indicates the central broker being deployed separately
70+
from Zeek cluster nodes, by default the manager node will start and run this
71+
broker using a separate thread. There's nothing that would prevent from running
72+
a long running central broker independently from the Zeek cluster nodes, however.
73+
74+
The serialization of Zeek events is done by the selected
75+
:zeek:see:`Cluster::event_serializer` and is independent of ZeroMQ.
76+
The central broker needs no knowledge about the chosen format, it is
77+
only shuffling messages between nodes.
78+
79+
80+
Logging
81+
-------
82+
83+
While remote events always pass through the central broker, nodes connect and
84+
send log writes directly to logger nodes in a cluster. The ZeroMQ cluster backend
85+
leverages ZeroMQ's pipeline pattern for this functionality. That is, logger nodes
86+
(including the manager if configured using :zeek:see:`Cluster::manager_is_logger`)
87+
open a ZeroMQ PULL socket to receive log writes. All other nodes connect their
88+
PUSH socket to all available PULL sockets. These connections are separate from
89+
the publish-subscribe setup outlined above.
90+
91+
When sending log-writes over a PUSH socket, load balancing is done by ZeroMQ.
92+
Individual cluster nodes do not have control over the decision which logger
93+
node receives log writes at any given time.
94+
95+
.. figure:: /images/cluster/zeromq-logging.png
96+
97+
While the previous paragraph used "log writes", a single message to a logger
98+
node actually contains a batch of log writes. The options :zeek:see:`Log::flush_interval`
99+
and :zeek:see:`Log::write_buffer_size` control the frequency and maximum size
100+
of these batches.
101+
102+
The serialization format used to encode such batches is controlled by the
103+
selected :zeek:see:`Cluster::log_serializer` and is independent of ZeroMQ.
104+
105+
With the default serializer (:zeek:see:`Cluster::LOG_SERIALIZER_ZEEK_BIN_V1`),
106+
every log batch on the wire has a header prepended that describes it. This allows
107+
interpretation of log writes even by non-Zeek processes. This opens the possibility
108+
to implement non-Zeek logger processes as long as the chosen serializer format
109+
is understood by the receiving process. In the future, a JSON lines serialization
110+
may be provided, allowing easier interpretation than a proprietary binary format.
111+
112+
113+
Summary
114+
-------
115+
116+
Combining the diagrams above, the connections between the different socket
117+
types in a Zeek cluster looks something like the following.
118+
119+
.. figure:: /images/cluster/zeromq-cluster.png
120+

‎devel/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,4 @@ transient, etc. compared to other documentation).
1717
Documentation Guide </README.rst>
1818
contributors
1919
maintainers
20+
cluster-backend-zeromq

‎images/cluster/Makefile‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
MMDC?=./node_modules/.bin/mmdc
2+
3+
%.png : %.mermaid
4+
$(MMDC) -i $< -e png -o $@
5+
6+
all: zeromq-cluster.png zeromq-pubsub.png zeromq-logging.png

‎images/cluster/README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
## Install mermaid-cli
2+
3+
npm install @mermaid-js/mermaid-cli
4+
5+
## Apparmor Errors
6+
7+
If running ``mmdc`` fails under Linux (e.g. with Ubuntu 24.04) with apparmor
8+
errors about ``userns_create`` in the ``demsg`` output, put the following into
9+
``/etc/apparmor.d/chrome-headless``
10+
11+
# This profile allows everything and only exists to give the
12+
# application a name instead of having the label "unconfined"
13+
abi <abi/4.0>,
14+
include <tunables/global>
15+
16+
profile chrome /home/awelzel/.cache/puppeteer/**/chrome-headless-shell flags=(unconfined) {
17+
userns,
18+
19+
# Site-specific additions and overrides. See local/README for details.
20+
include if exists <local/chrome>
21+
}
22+
23+
24+
See also: https://chromium.googlesource.com/chromium/src/+/main/docs/security/apparmor-userns-restrictions.md#option-2_a-safer-way

‎images/cluster/zeromq-cluster.mermaid‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
graph TD
2+
3+
w1-xpub-->broker-xsub
4+
broker-xpub-->w1-xsub
5+
6+
w2-xpub-->broker-xsub
7+
broker-xpub-->w2-xsub
8+
9+
w3-xpub-->broker-xsub
10+
broker-xpub-->w3-xsub
11+
12+
p1-xpub-->broker-xsub
13+
broker-xpub-->p1-xsub
14+
15+
p2-xpub-->broker-xsub
16+
broker-xpub-->p2-xsub
17+
18+
l1-xpub-->broker-xsub
19+
broker-xpub-->l1-xsub
20+
21+
l2-xpub-->broker-xsub
22+
broker-xpub-->l2-xsub
23+
24+
m-xpub-->broker-xsub
25+
broker-xpub-->m-xsub
26+
27+
%% Logging
28+
w1-push-->l1-pull
29+
w1-push-->l2-pull
30+
w2-push-->l1-pull
31+
w2-push-->l2-pull
32+
w3-push-->l1-pull
33+
w3-push-->l2-pull
34+
p1-push-->l1-pull
35+
p1-push-->l2-pull
36+
p2-push-->l1-pull
37+
p2-push-->l2-pull
38+
m-push-->l1-pull
39+
m-push-->l2-pull
40+
41+
subgraph broker ["broker"]
42+
broker-xpub((XPUB))
43+
broker-xsub((XSUB))
44+
broker-xpub-->broker-xsub
45+
broker-xsub-->broker-xpub
46+
end
47+
48+
49+
subgraph l1 [logger-1]
50+
l1-xpub((XPUB))
51+
l1-xsub((XSUB))
52+
l1-pull(PULL)
53+
end
54+
55+
subgraph l2 [logger-2]
56+
l2-xpub((XPUB))
57+
l2-xsub((XSUB))
58+
l2-pull(PULL)
59+
end
60+
61+
subgraph manager
62+
m-xpub((XPUB))
63+
m-xsub((XSUB))
64+
m-push(PUSH)
65+
end
66+
67+
subgraph p1 [proxy-1]
68+
p1-xpub((XPUB))
69+
p1-xsub((XSUB))
70+
p1-push(PUSH)
71+
end
72+
73+
subgraph p2 [proxy-2]
74+
p2-xpub((XPUB))
75+
p2-xsub((XSUB))
76+
p2-push(PUSH)
77+
end
78+
79+
subgraph w1 [worker-1]
80+
w1-xpub((XPUB))
81+
w1-xsub((XSUB))
82+
w1-push(PUSH)
83+
end
84+
85+
subgraph w2 [worker-2]
86+
w2-xpub((XPUB))
87+
w2-xsub((XSUB))
88+
w2-push(PUSH)
89+
end
90+
91+
subgraph w3 [worker-3]
92+
w3-xpub((XPUB))
93+
w3-xsub((XSUB))
94+
w3-push(PUSH)
95+
end

‎images/cluster/zeromq-cluster.png‎

46.4 KB
Loading

‎images/cluster/zeromq-logging.mermaid‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
flowchart TD
2+
3+
%% Logging
4+
w1-push-->l1-pull
5+
w1-push-->l2-pull
6+
w2-push-->l1-pull
7+
w2-push-->l2-pull
8+
w3-push-->l1-pull
9+
w3-push-->l2-pull
10+
p1-push-->l1-pull
11+
p1-push-->l2-pull
12+
p2-push-->l1-pull
13+
p2-push-->l2-pull
14+
m-push-->l1-pull
15+
m-push-->l2-pull
16+
17+
subgraph l1 [logger-1]
18+
l1-pull(PULL)
19+
end
20+
21+
subgraph l2 [logger-2]
22+
l2-pull(PULL)
23+
end
24+
25+
subgraph m [manager]
26+
m-push(PUSH)
27+
end
28+
29+
subgraph p1 [proxy-1]
30+
p1-push(PUSH)
31+
end
32+
33+
subgraph p2 [proxy-2]
34+
p2-push(PUSH)
35+
end
36+
37+
subgraph w1 [worker-1]
38+
w1-push(PUSH)
39+
end
40+
41+
subgraph w2 [worker-2]
42+
w2-push(PUSH)
43+
end
44+
45+
subgraph w3 [worker-3]
46+
47+
w3-push(PUSH)
48+
end
49+

‎images/cluster/zeromq-logging.png‎

30 KB
Loading

‎images/cluster/zeromq-pubsub.mermaid‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
graph TD
2+
3+
w1-xpub-->broker-xsub
4+
broker-xpub-->w1-xsub
5+
6+
w2-xpub-->broker-xsub
7+
broker-xpub-->w2-xsub
8+
9+
w3-xpub-->broker-xsub
10+
broker-xpub-->w3-xsub
11+
12+
p1-xpub-->broker-xsub
13+
broker-xpub-->p1-xsub
14+
15+
p2-xpub-->broker-xsub
16+
broker-xpub-->p2-xsub
17+
18+
l1-xpub-->broker-xsub
19+
broker-xpub-->l1-xsub
20+
21+
l2-xpub-->broker-xsub
22+
broker-xpub-->l2-xsub
23+
24+
m-xpub-->broker-xsub
25+
broker-xpub-->m-xsub
26+
27+
subgraph broker ["broker"]
28+
broker-xpub((XPUB))
29+
broker-xsub((XSUB))
30+
broker-xpub-->broker-xsub
31+
broker-xsub-->broker-xpub
32+
end
33+
34+
subgraph l1 [logger-1]
35+
l1-xpub((XPUB))
36+
l1-xsub((XSUB))
37+
end
38+
39+
subgraph l2 [logger-2]
40+
l2-xpub((XPUB))
41+
l2-xsub((XSUB))
42+
end
43+
subgraph m [manager]
44+
m-xpub((XPUB))
45+
m-xsub((XSUB))
46+
end
47+
48+
subgraph p1 [proxy-1]
49+
p1-xpub((XPUB))
50+
p1-xsub((XSUB))
51+
end
52+
53+
subgraph p2 [proxy-2]
54+
p2-xpub((XPUB))
55+
p2-xsub((XSUB))
56+
end
57+
58+
subgraph w1 [worker-1]
59+
w1-xpub((XPUB))
60+
w1-xsub((XSUB))
61+
end
62+
63+
subgraph w2 [worker-2]
64+
w2-xpub((XPUB))
65+
w2-xsub((XSUB))
66+
end
67+
68+
subgraph w3 [worker-3]
69+
w3-xpub((XPUB))
70+
w3-xsub((XSUB))
71+
end
72+

‎images/cluster/zeromq-pubsub.png‎

43.1 KB
Loading

0 commit comments

Comments
 (0)