Merge remote-tracking branch 'origin/topic/awelzel/updates-for-7.1'

* origin/topic/awelzel/updates-for-7.1:
  broker/cluster: Changes for auto_publish() deprecation
  ldap: Note about StartTLS
  postgresql: Some notes about the PostgreSQL analyzer
  signatures: Add end_of_match parameter documentation
  telemetry: Cleanup custom configuration hunk
  telemetry: Add prometheus sd_config example
  telemetry: Telemetry::hook() invoked on demand

Author: awelzel

frameworks/broker.rst

@@ -276,19 +276,11 @@ define any event handlers for events that peers will send.
    :linenos:
    :tab-width: 4
 
-There are two different ways to send events.
-
-The first is to call the :zeek:see:`Broker::publish` function which you can
+To send an event, call the :zeek:see:`Broker::publish` function which you can
 supply directly with the event and its arguments or give it the return value of
 :zeek:see:`Broker::make_event` in case you need to send the same event/args
 multiple times.  When publishing events like this, local event handlers for
-the event are not called.
-
-The second option is to call the :zeek:see:`Broker::auto_publish` function where
-you specify a particular event that will be automatically sent to peers
-whenever the event is called locally via the normal event invocation syntax.
-When auto-publishing events, local event handlers for the event are called
-in addition to sending the event to any subscribed peers.
+the event are not called, even if a matching subscription exists.
 
 .. literalinclude:: broker/events-connector.zeek
    :caption: events-connector.zeek
@@ -301,6 +293,20 @@ to the ``zeek/events`` topic prefix you would receive events that are published
 to topic names ``zeek/events/foo`` and ``zeek/events/bar`` but not
 ``zeek/misc``.
 
+
+.. note::
+
+   In prior Zeek versions, :zeek:see:`Broker::auto_publish` was available to
+   automatically send events to peers whenever the events were called locally via
+   the normal event invocation syntax. When auto-publishing events, local
+   event handlers for the event were called in addition to sending the
+   event to any subscribed peers.
+
+   :zeek:see:`Broker::auto_publish` has been deprecated due to its
+   `implicit nature <https://github.com/zeek/zeek/discussions/3637>`_.
+
+   .. deprecated:: 7.1
+
 Remote Logging
 --------------
 

frameworks/broker/events-connector.zeek

@@ -1,20 +1,16 @@
 redef exit_only_after_terminate = T;
 global my_event: event(msg: string, c: count);
-global my_auto_event: event(msg: string, c: count);
 
 event zeek_init()
 	{
 	Broker::peer("127.0.0.1");
-	Broker::auto_publish("zeek/event/my_auto_event", my_auto_event);
 	}
 
 event Broker::peer_added(endpoint: Broker::EndpointInfo, msg: string)
 	{
 	print "peer added", endpoint;
 	Broker::publish("zeek/event/my_event", my_event, "hi", 0);
-	event my_auto_event("stuff", 88);
 	Broker::publish("zeek/event/my_event", my_event, "...", 1);
-	event my_auto_event("more stuff", 51);
 	local e = Broker::make_event(my_event, "bye", 2);
 	Broker::publish("zeek/event/my_event", e);
 	}
@@ -28,8 +24,3 @@ event my_event(msg: string, c: count)
 	{
 	print "got my_event", msg, c;
 	}
-
-event my_auto_event(msg: string, c: count)
-	{
-	print "got my_auto_event", msg, c;
-	}

frameworks/broker/events-listener.zeek

@@ -22,12 +22,3 @@ event my_event(msg: string, c: count)
 	if ( msg_count == 5 )
 		terminate();
 	}
-
-event my_auto_event(msg: string, c: count)
-	{
-	++msg_count;
-	print "got my_auto_event", msg, c;
-
-	if ( msg_count == 5 )
-		terminate();
-	}

frameworks/cluster.rst

@@ -334,12 +334,6 @@ function to publish events, including:
     - Standard function to send an event to all nodes subscribed to a given
       topic
 
-  * - :zeek:see:`Broker::auto_publish`
-    - Automatically send an otherwise generated Zeek event to any interested
-      peers whenever it is locally dispatched.
-    - Avoid, since it is somewhat “magical”, unless you’ve got code
-      compartmentalization running with ``@ifdef`` directives.
-
   * - :zeek:see:`Cluster::publish_hrw`
     - Publishes an event to a node within a pool according to
       Highest Random Weight (HRW) hashing strategy; see details below

frameworks/signatures.rst

@@ -39,6 +39,19 @@ triggered the match, ``msg`` is the string specified by the
 signature's event statement (``Found root!``), and data is the last
 piece of payload which triggered the pattern match.
 
+.. versionadded:: 7.1
+
+An alternative form of :zeek:id:`signature_match` has an additional ``end_of_match`` parameter.
+
+.. code-block:: zeek
+
+    event signature_match(state: signature_state, msg: string, data: string, end_of_match: count)
+
+The ``end_of_match`` parameter represents the offset into ``data`` where
+the match ended, or said differently, the length of the matching data within ``data``.
+If a signature matches across packet boundaries, ``data`` contains just the
+payload of the packet where a signature match triggered.
+
 To turn such :zeek:id:`signature_match` events into actual alarms, you can
 load Zeek's :doc:`/scripts/base/frameworks/signatures/main.zeek` script.
 This script contains a default event handler that raises
@@ -270,17 +283,20 @@ Actions define what to do if a signature matches. Currently, there are
 two actions defined, ``event`` and ``enable``.
 
 ``event <string>``
-    Raises a :zeek:id:`signature_match` event. The event handler has the
-    following type:
+    Raises a :zeek:id:`signature_match` event. The event handler has either
+    of the following types:
 
     .. code-block:: zeek
 
         event signature_match(state: signature_state, msg: string, data: string)
 
+        event signature_match(state: signature_state, msg: string, data: string, end_of_match: count)
+
     The given string is passed in as ``msg``, and data is the current
     part of the payload that has eventually lead to the signature
     match (this may be empty for signatures without content
-    conditions).
+    conditions). The ``end_of_match`` parameter represents the length of
+    the matching payload in ``data``.
 
 ``event event_name [string]``
 
@@ -305,6 +321,13 @@ two actions defined, ``event`` and ``enable``.
 
         event found_root(state: signature_state, data: string)
 
+    Like the :zeek:id:`signature_match` event, custom events can have an additional
+    ``end_of_match`` parameter.
+
+    .. code-block:: zeek
+
+        event found_root(state: signature_state, data: string, end_of_match: count)
+
     .. note::
 
       Matches for signatures that use custom events do not appear

frameworks/telemetry.rst

@@ -164,12 +164,24 @@ node's metrics port::
   ...
 
 To simplify telemetry collection from all nodes in a cluster, Zeek supports
-`Prometheus HTTP Service Discovery`_ on the manager node. In this approach, the
+`Prometheus HTTP Service Discovery`_ on the manager node. Using this approach, the
 endpoint ``http://<manager>:<manager-metrics-port>/services.json`` returns a
 JSON data structure that itemizes all metrics endpoints in the
 cluster. Prometheus scrapers supporting service discovery then proceed to
-collect telemetry from the listed endpoints in turn. See the `Prometheus Getting
-Started Guide`_ for additional information.
+collect telemetry from the listed endpoints in turn.
+
+The following is an example service discovery scrape config entry within
+Prometheus server's ``prometheus.yml`` configuration file::
+
+    ...
+    scrape_configs:
+      - job_name: zeek-discovery
+        scrape_interval: 5s
+        http_sd_configs:
+          - url: http://localhost:9991/services.json
+            refresh_interval: 10s
+
+See the `Prometheus Getting Started Guide`_ for additional information.
 
 .. note::
 
@@ -184,8 +196,8 @@ Started Guide`_ for additional information.
 
 If these setups aren't right for your environment, there's the possibility to
 redefine the options in ``local.zeek`` to something more suitable. For example,
-the following snippet opens an individual Prometheus port for each Zeek process
-(relative to the port used in ``cluster-layout.zeek``)::
+the following snippet selects the metrics port of each Zeek process relative
+to the cluster port used in ``cluster-layout.zeek``::
 
     @load base/frameworks/cluster
 
@@ -194,15 +206,6 @@ the following snippet opens an individual Prometheus port for each Zeek process
 
     redef Telemetry::metrics_port = my_metrics_port;
 
-As a different example, to only change the port from 9911 to 1234 on the manager
-process, but keep the export and import of metrics enabled, use the following snippet::
-
-    @load base/frameworks/cluster
-
-    @ifdef ( Cluster::local_node_type() == Cluster::MANAGER )
-    redef Telemetry::metrics_port = 1234/tcp;
-    @endif
-
 
 Examples of Metrics Application
 ===============================
@@ -276,7 +279,7 @@ directly.
    :tab-width: 4
 
 
-For metrics without labels, the metric instances can also be *cached* as global
+For metrics without labels, the metric instances can also be cached as global
 variables directly. The following example counts the number of http requests.
 
 .. literalinclude:: telemetry/global-http-counter.zeek
@@ -290,23 +293,39 @@ Sync
 ^^^^
 
 In case the scripting overhead of the previous approach is still too high,
-individual writes (or events) can be tracked in a table and then
-synchronized / mirrored during execution of the :zeek:see:`Telemetry::sync`
-hook.
+individual writes (or events) can be tracked in a table or global variable
+and then synchronized / mirrored to concrete counter and gauge instances
+during execution of the :zeek:see:`Telemetry::sync` hook.
 
 .. literalinclude:: telemetry/log-writes-sync.zeek
    :caption: log-writes-sync.zeek
    :language: zeek
    :linenos:
    :tab-width: 4
 
-For the use-case of tracking log writes, this is unlikely to be required, but
-for updating metrics within high frequency events that otherwise have very
-low processing overhead it's a valuable approach. Note, metrics will be stale
-up to the next :zeek:see:`Telemetry::sync_interval` using this method.
+For tracking log writes, this is unlikely to be required (and Zeek exposes
+various logging natively through the framework already), but for updating
+metrics within high frequency events that otherwise have low script processing
+overhead, it's a valuable approach.
+
+
+.. versionchanged:: 7.1
+
+The :zeek:see:`Telemetry::sync` hook is invoked on-demand only. Either when
+one of the :zeek:see:`Telemetry::collect_metrics`
+or :zeek:see:`Telemetry::collect_histogram_metrics` functions is invoked, or
+when querying Prometheus endpoint. It's an error to call either of the
+collection BiFs within the :zeek:see:`Telemetry::sync` hook and results
+in a reporter warning.
+
+
+.. note::
 
+   In versions before Zeek 7.1, :zeek:see:`Telemetry::sync` was invoked on a
+   fixed schedule, potentially resulting in stale metrics at collection time,
+   as well as generating small runtime overhead when metrics are not collected.
 
-Table sizes
+Table Sizes
 -----------
 
 It can be useful to expose the size of tables as metrics, as they often

frameworks/telemetry/log-writes-sync.zeek

@@ -12,6 +12,7 @@ hook Log::log_stream_policy(rec: any, id: Log::ID)
 	{
 	++log_writes[id];
 	}
+
 hook Telemetry::sync()
 	{
 	for ( id, v in log_writes )

logs/index.rst

@@ -19,9 +19,10 @@ Zeek Logs
    ntp
    smb
    irc
-   rdp
    ldap
+   postgresql
    quic
+   rdp
    traceroute
    tunnel
    dpd

logs/ldap.rst

@@ -103,6 +103,36 @@ from the log.
     }
 
 
+StartTLS
+========
+
+.. versionadded:: 7.0
+
+Zeek's LDAP analyzer supports the
+`extended StartTLS <https://datatracker.ietf.org/doc/html/rfc4511#section-4.14>`_
+operation, handing off analysis to Zeek's TLS analyzer. The following shows an
+example :file:`ldap.log` entry for the StartTLS request.
+
+.. code-block:: console
+
+    $ zeek -C LogAscii::use_json=T -r ldap-starttls.pcap
+    $ jq < ldap.log
+    {
+      "ts": 1721218680.158341,
+      "uid": "CW0qzo9A3QsrCWL4k",
+      "id.orig_h": "127.0.0.1",
+      "id.orig_p": 45936,
+      "id.resp_h": "127.0.1.1",
+      "id.resp_p": 389,
+      "message_id": 1,
+      "opcode": "extended",
+      "result": "success",
+      "object": "1.3.6.1.4.1.1466.20037 (StartTLS)"
+    }
+
+The :file:`conn.log`'s history field will contain ``ssl`` and ``ldap`` in
+the ``service`` field.
+
 Conclusion
 ==========