Skip to content
This repository was archived by the owner on Oct 13, 2025. It is now read-only.

Commit 625c534

Browse files
awelzelawelzel
committed
Merge remote-tracking branch 'origin/topic/awelzel/add-af-packet-node-cfg-hide-pf-ring-a-bit'
* origin/topic/awelzel/add-af-packet-node-cfg-hide-pf-ring-a-bit: cluster-setup: Drop Click! Software Router cluster-setup: Add ZeekControl AF_PACKET, move PF_RING to separate page
2 parents 12ce9cc + 3979226 commit 625c534

File tree

2 files changed

+192
-138
lines changed

2 files changed

+192
-138
lines changed

‎cluster-setup.rst‎

Lines changed: 51 additions & 138 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,9 @@ above, both Zeek workers join the same fanout group.
232232
This may be visible in Zeek logs as gaps and/or duplicated connection
233233
entries produced by different Zeek workers.
234234

235+
See :ref:`cluster-configuration` for instructions how to configure AF_PACKET
236+
with ZeekControl.
237+
235238

236239
Netmap
237240
^^^^^^
@@ -333,15 +336,6 @@ For using ``lb`` or libpcap with netmap support, refer to the commands shown
333336
in the FreeBSD section - these are essentially the same.
334337

335338

336-
Click! Software Router
337-
^^^^^^^^^^^^^^^^^^^^^^
338-
339-
Click! can be used for flow based load balancing with a simple configuration.
340-
This solution is not recommended on
341-
Linux due to Zeek's PF_RING support and only as a last resort on other
342-
operating systems since it causes a lot of overhead due to context switching
343-
back and forth between kernel and userland several times per packet.
344-
345339
.. _cluster-configuration:
346340

347341
Cluster Configuration
@@ -447,151 +441,70 @@ a Zeek cluster (do this as the Zeek user on the manager host only):
447441
for information on setting up a cron job on the manager host that can
448442
monitor the cluster.
449443

450-
.. _pf-ring-config:
451-
452-
PF_RING Cluster Configuration
453-
-----------------------------
454-
455-
`PF_RING <http://www.ntop.org/products/pf_ring/>`_ allows speeding up the
456-
packet capture process by installing a new type of socket in Linux systems.
457-
It supports 10Gbit hardware packet filtering using standard network adapters,
458-
and user-space DNA (Direct NIC Access) for fast packet capture/transmission.
459-
460-
Installing PF_RING
461-
******************
462-
463-
1. Download and install PF_RING for your system following the instructions
464-
`here <http://www.ntop.org/get-started/download/#PF_RING>`_. The following
465-
commands will install the PF_RING libraries and kernel module (replace
466-
the version number 5.6.2 in this example with the version that you
467-
downloaded)::
468-
469-
cd /usr/src
470-
tar xvzf PF_RING-5.6.2.tar.gz
471-
cd PF_RING-5.6.2/userland/lib
472-
./configure --prefix=/opt/pfring
473-
make install
474-
475-
cd ../libpcap
476-
./configure --prefix=/opt/pfring
477-
make install
478-
479-
cd ../tcpdump-4.1.1
480-
./configure --prefix=/opt/pfring
481-
make install
482-
483-
cd ../../kernel
484-
make
485-
make install
486-
487-
modprobe pf_ring enable_tx_capture=0 min_num_slots=32768
488-
489-
Refer to the documentation for your Linux distribution on how to load the
490-
pf_ring module at boot time. You will need to install the PF_RING
491-
library files and kernel module on all of the workers in your cluster.
492-
493-
2. Download the Zeek source code.
494-
495-
3. Configure and install Zeek using the following commands::
496-
497-
./configure --with-pcap=/opt/pfring
498-
make
499-
make install
500-
501-
4. Make sure Zeek is correctly linked to the PF_RING libpcap libraries::
502-
503-
ldd /usr/local/zeek/bin/zeek | grep pcap
504-
libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007fa6d7d24000)
444+
AF_PACKET Cluster Configuration
445+
-------------------------------
505446

506-
5. Configure ZeekControl to use PF_RING (explained below).
447+
Since version 5.2, Zeek includes AF_PACKET as a native packet source. This
448+
provides an easy and efficient capture mechanism for Linux users.
507449

508-
6. Run "zeekctl install" on the manager. This command will install Zeek and
509-
required scripts to all machines in your cluster.
450+
Adapt the worker section in ZeekControl's ``node.cfg`` file with the
451+
following entries, assuming running four worker processes listening on ``eth0`` ::
510452

511-
Using PF_RING
512-
*************
513-
514-
In order to use PF_RING, you need to specify the correct configuration
515-
options for your worker nodes in ZeekControl's node configuration file.
516-
Edit the ``node.cfg`` file and specify ``lb_method=pf_ring`` for each of
517-
your worker nodes. Next, use the ``lb_procs`` node option to specify how
518-
many Zeek processes you'd like that worker node to run, and optionally pin
519-
those processes to certain CPU cores with the ``pin_cpus`` option (CPU
520-
numbering starts at zero). The correct ``pin_cpus`` setting to use is
521-
dependent on your CPU architecture (Intel and AMD systems enumerate
522-
processors in different ways). Using the wrong ``pin_cpus`` setting
523-
can cause poor performance. Here is what a worker node entry should
524-
look like when using PF_RING and CPU pinning::
525-
526-
[worker-1]
527-
type=worker
528-
host=10.0.0.50
529-
interface=eth0
530-
lb_method=pf_ring
531-
lb_procs=10
532-
pin_cpus=2,3,4,5,6,7,8,9,10,11
533-
534-
535-
Using PF_RING+DNA with symmetric RSS
536-
************************************
537-
538-
You must have a PF_RING+DNA license in order to do this. You can sniff
539-
each packet only once.
540-
541-
1. Load the DNA NIC driver (i.e. ixgbe) on each worker host.
542-
543-
2. Run "ethtool -L dna0 combined 10" (this will establish 10 RSS queues
544-
on your NIC) on each worker host. You must make sure that you set the
545-
number of RSS queues to the same as the number you specify for the
546-
lb_procs option in the node.cfg file.
453+
[worker-1]
454+
type=worker
455+
host=10.0.0.11
456+
interface=eth0
457+
lb_method=af_packet
458+
lb_procs=4
547459

548-
3. On the manager, configure your worker(s) in node.cfg::
460+
The specific options are ``lb_method=af_packet`` and ``lb_procs=4``.
461+
If listening on two or more interfaces on the same host is a requirement,
462+
remember to set a unique ``fanout_id`` using the node option ``af_packet_found_id``::
549463

550-
[worker-1]
551-
type=worker
552-
host=10.0.0.50
553-
interface=dna0
554-
lb_method=pf_ring
555-
lb_procs=10
464+
[worker-1-eth0]
465+
type=worker
466+
host=10.0.0.11
467+
interface=eth0
468+
lb_method=af_packet
469+
lb_procs=4
470+
af_packet_fanout_id=20
556471

472+
[worker-1-eth1]
473+
type=worker
474+
host=10.0.0.11
475+
interface=eth1
476+
lb_method=af_packet
477+
lb_procs=4
478+
af_packet_fanout_id=21
557479

558-
Using PF_RING+DNA with pfdnacluster_master
559-
******************************************
480+
Pinning the worker processes to individual CPU cores can improve performance.
481+
Use the node's option ``pin_cpus=4,5,6,7``, listing as many CPU numbers as
482+
processes at appropriate offsets.
560483

561-
You must have a PF_RING+DNA license and a libzero license in order to do
562-
this. You can load balance between multiple applications and sniff the
563-
same packets multiple times with different tools.
484+
.. _pf-ring-config:
564485

565-
1. Load the DNA NIC driver (i.e. ixgbe) on each worker host.
486+
PF_RING Cluster Configuration
487+
-----------------------------
566488

567-
2. Run "ethtool -L dna0 1" (this will establish 1 RSS queues on your NIC)
568-
on each worker host.
489+
`PF_RING <http://www.ntop.org/products/pf_ring/>`_ allows speeding up the
490+
packet capture process by installing a new type of socket in Linux systems.
491+
It supports 10Gbit hardware packet filtering using standard network adapters,
492+
and user-space DNA (Direct NIC Access) for fast packet capture/transmission.
569493

570-
3. Run the pfdnacluster_master command on each worker host. For example::
494+
.. note::
571495

572-
pfdnacluster_master -c 21 -i dna0 -n 10
496+
Unless you have evaluated to specifically require PF_RING, consider using
497+
AF_PACKET first and test if it fullfills your requirements. AF_PACKET has
498+
been integrated into Zeek since version 5.2. It's a bit easier to get
499+
started with as it does not require an out of tree Linux kernel module.
573500

574-
Make sure that your cluster ID (21 in this example) matches the interface
575-
name you specify in the node.cfg file. Also make sure that the number
576-
of processes you're balancing across (10 in this example) matches
577-
the lb_procs option in the node.cfg file.
501+
Head over to :ref:`cluster-pf-ring` for more details.
578502

579-
4. If you are load balancing to other processes, you can use the
580-
pfringfirstappinstance variable in zeekctl.cfg to set the first
581-
application instance that Zeek should use. For example, if you are running
582-
pfdnacluster_master with "-n 10,4" you would set
583-
pfringfirstappinstance=4. Unfortunately that's still a global setting
584-
in zeekctl.cfg at the moment but we may change that to something you can
585-
set in node.cfg eventually.
503+
.. toctree::
504+
:hidden:
586505

587-
5. On the manager, configure your worker(s) in node.cfg::
506+
cluster/pf_ring
588507

589-
[worker-1]
590-
type=worker
591-
host=10.0.0.50
592-
interface=dnacluster:21
593-
lb_method=pf_ring
594-
lb_procs=10
595508

596509
.. [#] Some Linux kernel versions between 3.10 and 4.7 might exhibit
597510
a bug that prevents the required symmetric hashing. The script available

‎cluster/pf_ring.rst‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
.. _cluster-pf-ring:
2+
3+
===================
4+
PF_RING Setup Guide
5+
===================
6+
7+
Installing PF_RING
8+
******************
9+
10+
1. Download and install PF_RING for your system following the instructions
11+
`here <http://www.ntop.org/get-started/download/#PF_RING>`_. The following
12+
commands will install the PF_RING libraries and kernel module (replace
13+
the version number 5.6.2 in this example with the version that you
14+
downloaded)::
15+
16+
cd /usr/src
17+
tar xvzf PF_RING-5.6.2.tar.gz
18+
cd PF_RING-5.6.2/userland/lib
19+
./configure --prefix=/opt/pfring
20+
make install
21+
22+
cd ../libpcap
23+
./configure --prefix=/opt/pfring
24+
make install
25+
26+
cd ../tcpdump-4.1.1
27+
./configure --prefix=/opt/pfring
28+
make install
29+
30+
cd ../../kernel
31+
make
32+
make install
33+
34+
modprobe pf_ring enable_tx_capture=0 min_num_slots=32768
35+
36+
Refer to the documentation for your Linux distribution on how to load the
37+
pf_ring module at boot time. You will need to install the PF_RING
38+
library files and kernel module on all of the workers in your cluster.
39+
40+
2. Download the Zeek source code.
41+
42+
3. Configure and install Zeek using the following commands::
43+
44+
./configure --with-pcap=/opt/pfring
45+
make
46+
make install
47+
48+
4. Make sure Zeek is correctly linked to the PF_RING libpcap libraries::
49+
50+
ldd /usr/local/zeek/bin/zeek | grep pcap
51+
libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007fa6d7d24000)
52+
53+
5. Configure ZeekControl to use PF_RING (explained below).
54+
55+
6. Run "zeekctl install" on the manager. This command will install Zeek and
56+
required scripts to all machines in your cluster.
57+
58+
Using PF_RING
59+
*************
60+
61+
In order to use PF_RING, you need to specify the correct configuration
62+
options for your worker nodes in ZeekControl's node configuration file.
63+
Edit the ``node.cfg`` file and specify ``lb_method=pf_ring`` for each of
64+
your worker nodes. Next, use the ``lb_procs`` node option to specify how
65+
many Zeek processes you'd like that worker node to run, and optionally pin
66+
those processes to certain CPU cores with the ``pin_cpus`` option (CPU
67+
numbering starts at zero). The correct ``pin_cpus`` setting to use is
68+
dependent on your CPU architecture (Intel and AMD systems enumerate
69+
processors in different ways). Using the wrong ``pin_cpus`` setting
70+
can cause poor performance. Here is what a worker node entry should
71+
look like when using PF_RING and CPU pinning::
72+
73+
[worker-1]
74+
type=worker
75+
host=10.0.0.50
76+
interface=eth0
77+
lb_method=pf_ring
78+
lb_procs=10
79+
pin_cpus=2,3,4,5,6,7,8,9,10,11
80+
81+
82+
Using PF_RING+DNA with symmetric RSS
83+
************************************
84+
85+
You must have a PF_RING+DNA license in order to do this. You can sniff
86+
each packet only once.
87+
88+
1. Load the DNA NIC driver (i.e. ixgbe) on each worker host.
89+
90+
2. Run "ethtool -L dna0 combined 10" (this will establish 10 RSS queues
91+
on your NIC) on each worker host. You must make sure that you set the
92+
number of RSS queues to the same as the number you specify for the
93+
lb_procs option in the node.cfg file.
94+
95+
3. On the manager, configure your worker(s) in node.cfg::
96+
97+
[worker-1]
98+
type=worker
99+
host=10.0.0.50
100+
interface=dna0
101+
lb_method=pf_ring
102+
lb_procs=10
103+
104+
105+
Using PF_RING+DNA with pfdnacluster_master
106+
******************************************
107+
108+
You must have a PF_RING+DNA license and a libzero license in order to do
109+
this. You can load balance between multiple applications and sniff the
110+
same packets multiple times with different tools.
111+
112+
1. Load the DNA NIC driver (i.e. ixgbe) on each worker host.
113+
114+
2. Run "ethtool -L dna0 1" (this will establish 1 RSS queues on your NIC)
115+
on each worker host.
116+
117+
3. Run the pfdnacluster_master command on each worker host. For example::
118+
119+
pfdnacluster_master -c 21 -i dna0 -n 10
120+
121+
Make sure that your cluster ID (21 in this example) matches the interface
122+
name you specify in the node.cfg file. Also make sure that the number
123+
of processes you're balancing across (10 in this example) matches
124+
the lb_procs option in the node.cfg file.
125+
126+
4. If you are load balancing to other processes, you can use the
127+
pfringfirstappinstance variable in zeekctl.cfg to set the first
128+
application instance that Zeek should use. For example, if you are running
129+
pfdnacluster_master with "-n 10,4" you would set
130+
pfringfirstappinstance=4. Unfortunately that's still a global setting
131+
in zeekctl.cfg at the moment but we may change that to something you can
132+
set in node.cfg eventually.
133+
134+
5. On the manager, configure your worker(s) in node.cfg::
135+
136+
[worker-1]
137+
type=worker
138+
host=10.0.0.50
139+
interface=dnacluster:21
140+
lb_method=pf_ring
141+
lb_procs=10

0 commit comments

Comments
 (0)