Merge remote-tracking branch 'origin/topic/christian/conn-ip-proto'

* origin/topic/christian/conn-ip-proto:
  Documentation for conn.log's new ip_proto field.

Author: timwoj

log-formats.rst

@@ -108,10 +108,10 @@ We use the :program:`cat` command to show the contents of each log.
   #unset_field    -
   #path   conn
   #open   2020-06-05-14-48-32
-  #fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       proto   service duration        orig_bytes      resp_bytes      conn_state    local_orig      local_resp      missed_bytes    history orig_pkts       orig_ip_bytes   resp_pkts       resp_ip_bytes   tunnel_parents
-  #types  time    string  addr    port    addr    port    enum    string  interval        count   count   string  bool    bool    count   string  count   count count    count   set[string]
-  1591367999.305988       CazOhH2qDUiJTWMCY       192.168.4.76    36844   192.168.4.1     53      udp     dns     0.066852        62      141     SF      -    -0       Dd      2       118     2       197     -
-  1591367999.430166       CLqEx41jYPOdfHF586      192.168.4.76    46378   31.3.245.133    80      tcp     http    0.254115        77      295     SF      -    -0       ShADadFf        6       397     4       511     -
+  #fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       proto   service duration        orig_bytes      resp_bytes      conn_state    local_orig      local_resp      missed_bytes    history orig_pkts       orig_ip_bytes   resp_pkts       resp_ip_bytes   tunnel_parents  ip_proto
+  #types  time    string  addr    port    addr    port    enum    string  interval        count   count   string  bool    bool    count   string  count   count count    count   set[string]    count
+  1591367999.305988       CazOhH2qDUiJTWMCY       192.168.4.76    36844   192.168.4.1     53      udp     dns     0.066852        62      141     SF      -    -0       Dd      2       118     2       197     -       17
+  1591367999.430166       CLqEx41jYPOdfHF586      192.168.4.76    46378   31.3.245.133    80      tcp     http    0.254115        77      295     SF      -    -0       ShADadFf        6       397     4       511     -       6
   #close  2020-06-05-14-48-32
 
 Next we look at Zeek’s :file:`dns.log`.
@@ -434,11 +434,17 @@ First we look at :file:`packet_filter.log`.
 ::
 
   {"ts":1591368442.854585,"node":"zeek","filter":"ip or not ip","init":true,"success":true}
+
+Next we look at :file:`conn.log` and :file:`dns.log`:
+
+.. code-block:: console
+
   zeek@zeek:~/zeek-test/json$ cat conn.log
-  {"ts":1591367999.305988,"uid":"CMdzit1AMNsmfAIiQc","id.orig_h":"192.168.4.76","id.orig_p":36844,"id.resp_h":"192.168.4.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.06685185432434082,"orig_bytes":62,"resp_bytes":141,"conn_state":"SF","missed_bytes":0,"history":"Dd","orig_pkts":2,"orig_ip_bytes":118,"resp_pkts":2,"resp_ip_bytes":197}
-  {"ts":1591367999.430166,"uid":"C5bLoe2Mvxqhawzqqd","id.orig_h":"192.168.4.76","id.orig_p":46378,"id.resp_h":"31.3.245.133","id.resp_p":80,"proto":"tcp","service":"http","duration":0.25411510467529297,"orig_bytes":77,"resp_bytes":295,"conn_state":"SF","missed_bytes":0,"history":"ShADadFf","orig_pkts":6,"orig_ip_bytes":397,"resp_pkts":4,"resp_ip_bytes":511}
 
-Next we look at :file:`dns.log`.
+::
+
+  {"ts":1591367999.305988,"uid":"CMdzit1AMNsmfAIiQc","id.orig_h":"192.168.4.76","id.orig_p":36844,"id.resp_h":"192.168.4.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.06685185432434082,"orig_bytes":62,"resp_bytes":141,"conn_state":"SF","missed_bytes":0,"history":"Dd","orig_pkts":2,"orig_ip_bytes":118,"resp_pkts":2,"resp_ip_bytes":197,"ip_proto":17}
+  {"ts":1591367999.430166,"uid":"C5bLoe2Mvxqhawzqqd","id.orig_h":"192.168.4.76","id.orig_p":46378,"id.resp_h":"31.3.245.133","id.resp_p":80,"proto":"tcp","service":"http","duration":0.25411510467529297,"orig_bytes":77,"resp_bytes":295,"conn_state":"SF","missed_bytes":0,"history":"ShADadFf","orig_pkts":6,"orig_ip_bytes":397,"resp_pkts":4,"resp_ip_bytes":511,"ip_proto":6}
 
 .. code-block:: console
 

logs/conn.rst

@@ -34,9 +34,9 @@ stored traffic. We use the :program:`jq` utility to review the contents.
 
 ::
 
-  {"ts":1591367999.305988,"uid":"CMdzit1AMNsmfAIiQc","id.orig_h":"192.168.4.76","id.orig_p":36844,"id.resp_h":"192.168.4.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.06685185432434082,"orig_bytes":62,"resp_bytes":141,"conn_state":"SF","missed_bytes":0,"history":"Dd","orig_pkts":2,"orig_ip_bytes":118,"resp_pkts":2,"resp_ip_bytes":197}
+  {"ts":1591367999.305988,"uid":"CMdzit1AMNsmfAIiQc","id.orig_h":"192.168.4.76","id.orig_p":36844,"id.resp_h":"192.168.4.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.06685185432434082,"orig_bytes":62,"resp_bytes":141,"conn_state":"SF","missed_bytes":0,"history":"Dd","orig_pkts":2,"orig_ip_bytes":118,"resp_pkts":2,"resp_ip_bytes":197,"ip_proto":17}
 
-  {"ts":1591367999.430166,"uid":"C5bLoe2Mvxqhawzqqd","id.orig_h":"192.168.4.76","id.orig_p":46378,"id.resp_h":"31.3.245.133","id.resp_p":80,"proto":"tcp","service":"http","duration":0.25411510467529297,"orig_bytes":77,"resp_bytes":295,"conn_state":"SF","missed_bytes":0,"history":"ShADadFf","orig_pkts":6,"orig_ip_bytes":397,"resp_pkts":4,"resp_ip_bytes":511}
+  {"ts":1591367999.430166,"uid":"C5bLoe2Mvxqhawzqqd","id.orig_h":"192.168.4.76","id.orig_p":46378,"id.resp_h":"31.3.245.133","id.resp_p":80,"proto":"tcp","service":"http","duration":0.25411510467529297,"orig_bytes":77,"resp_bytes":295,"conn_state":"SF","missed_bytes":0,"history":"ShADadFf","orig_pkts":6,"orig_ip_bytes":397,"resp_pkts":4,"resp_ip_bytes":511,"ip_proto":6}
 
 Alternatively, we could see each field printed on its own line:
 
@@ -64,7 +64,8 @@ Alternatively, we could see each field printed on its own line:
     "orig_pkts": 2,
     "orig_ip_bytes": 118,
     "resp_pkts": 2,
-    "resp_ip_bytes": 197
+    "resp_ip_bytes": 197,
+    "ip_proto": 17
   }
   {
     "ts": 1591367999.430166,
@@ -84,7 +85,8 @@ Alternatively, we could see each field printed on its own line:
     "orig_pkts": 6,
     "orig_ip_bytes": 397,
     "resp_pkts": 4,
-    "resp_ip_bytes": 511
+    "resp_ip_bytes": 511,
+    "ip_proto": 6
   }
 
 What an analyst derives from any log is a function of the questions that he or
@@ -119,7 +121,8 @@ the following:
     "orig_pkts": 6,
     "orig_ip_bytes": 397,
     "resp_pkts": 4,
-    "resp_ip_bytes": 511
+    "resp_ip_bytes": 511,
+    "ip_proto": 6
   }
 
 For the second log, ``192.168.4.76`` talked to ``31.3.245.133``.
@@ -264,6 +267,10 @@ In the highlighted output, we see that :program:`tshark` notes 77 bytes of data
 carried by TCP from ``192.168.4.76``. I highlighted what that data was,
 beginning with a GET request.
 
+The ``orig_pkts`` and ``resp_pkts`` fields report the number of IP packets
+transferred in the respective directions. The ``orig_ip_bytes`` and
+``resp_ip_bytes`` indicate the total IP packet-level byte counts, respectively.
+
 Another way to look at this TCP segment is to dump the hex contents using a
 different :program:`tshark` option, as shown below.
 
@@ -339,7 +346,8 @@ reference.
     "orig_pkts": 2,
     "orig_ip_bytes": 118,
     "resp_pkts": 2,
-    "resp_ip_bytes": 197
+    "resp_ip_bytes": 197,
+    "ip_proto": 17
   }
 
 For the first entry, ``192.168.4.76`` talked to ``192.168.4.1``.
@@ -370,6 +378,52 @@ the conversations as “normal establishment and termination” of the
 Similarly, the ``history`` field is simply ``Dd``, indicating that each party
 to the conversation sent data to the other.
 
+The ``ip_proto`` Field
+======================
+
+.. versionadded:: 7.1
+
+The numeric ``ip_proto`` field reports the `IP protocol number
+<https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml>`_ of
+the connection. It relates to the ``proto`` field, but while the former
+represents a :zeek:type:`transport_proto` value that exclusively covers
+*transport* protocols Zeek knows how to parse (and ties into Zeek's
+:zeek:type:`port` type), the ``ip_proto`` field is always present, including for
+non-transport IP packet flows such as IGMP or OSPF. For example, an OSPF flow
+might look as follows:
+
+::
+
+  {
+    "ts": 1098361214.420459,
+    "uid": "C9EV8R4fN8bfSj08f",
+    "id.orig_h": "192.168.170.2",
+    "id.orig_p": 0,
+    "id.resp_h": "224.0.0.6",
+    "id.resp_p": 0,
+    "proto": "unknown_transport",
+    "duration": 6.437546968460083,
+    "orig_bytes": 0,
+    "resp_bytes": 0,
+    "conn_state": "OTH",
+    "local_orig": true,
+    "local_resp": false,
+    "missed_bytes": 0,
+    "orig_pkts": 4,
+    "orig_ip_bytes": 768,
+    "resp_pkts": 0,
+    "resp_ip_bytes": 0,
+    "ip_proto": 89
+  }
+
+You can adapt this feature in several ways. Load the
+:doc:`/scripts/policy/protocols/conn/ip-proto-name-logging.zeek` policy script
+to add an ``ip_proto_name`` column with a string version of the ``ip_proto``
+value. Also, you may disable the whole feature by loading the
+:doc:`/scripts/policy/protocols/conn/disable-unknown-ip-proto-support.zeek`
+script, returning conn.log to its pre-7.1 state. Zeek's :ref:`logging framework
+<framework-logging>` supports additional customizations.
+
 The ``uid`` and Other Fields
 ============================
 

logs/dhcp.rst

@@ -346,7 +346,7 @@ source IP address, sending its search to the local network for a DHCP server.
     "orig_ip_bytes": 1323,
     "resp_pkts": 0,
     "resp_ip_bytes": 0,
-    "sensorname": "so16-enp0s8"
+    "ip_proto": 17
   }
 
 Notice that Zeek has tracked 4 “orig packets” here, which does not strictly
@@ -384,7 +384,7 @@ client, and ``192.168.4.1``, the DHCP server.
     "orig_ip_bytes": 0,
     "resp_pkts": 2,
     "resp_ip_bytes": 660,
-    "sensorname": "so16-enp0s8"
+    "ip_proto": 17
   }
 
 Here the count of 2 ``resp_pkts`` is correct.

logs/dpd.rst

@@ -248,7 +248,8 @@ Here is the :file:`conn.log` that Zeek generated for this activity:
     "orig_pkts": 6,
     "orig_ip_bytes": 776,
     "resp_pkts": 5,
-    "resp_ip_bytes": 675
+    "resp_ip_bytes": 675,
+    "ip_proto": 6
   }
 
 The :file:`conn.log` entry is fairly normal.

logs/ftp.rst

@@ -63,6 +63,7 @@ for easier viewing.
     "orig_ip_bytes": 1232,
     "resp_pkts": 17,
     "resp_ip_bytes": 1343,
+    "ip_proto": 6,
     "community_id": "1:lEESxqaSVYqFZvWNb4OccTa9sTs="
   }
   {
@@ -296,6 +297,7 @@ IP addresses too.
     "orig_ip_bytes": 216,
     "resp_pkts": 4,
     "resp_ip_bytes": 290,
+    "ip_proto": 6,
     "community_id": "1:DNwvGR6Ots6pISvsdXBUIaG8y3Q="
   }
 

logs/irc.rst

@@ -118,7 +118,8 @@ Zeek generated the following :file:`conn.log` entry for the example traffic.
     "orig_pkts": 41,
     "orig_ip_bytes": 1963,
     "resp_pkts": 185,
-    "resp_ip_bytes": 246742
+    "resp_ip_bytes": 246742,
+    "ip_proto": 6
   }
 
 We see that Zeek correctly identified this traffic as IRC. We can expect to see

logs/pe.rst

@@ -14,7 +14,7 @@ Starting with :file:`conn.log`
 ==============================
 
 This example starts with the :file:`conn.log`. It’s not strictly necessary to
-explain the :file:`pe.log`, although I wanted to include a very recent example
+explain the :file:`pe.log`, although I wanted to include a recent example
 of a modern application conducting activities via HTTP.
 
 .. literal-emph::
@@ -40,6 +40,7 @@ of a modern application conducting activities via HTTP.
     "orig_ip_bytes": 29041,
     "resp_pkts": 1367,
     "resp_ip_bytes": 2030409,
+    "ip_proto": 6
   }
 
 This example shows a host, ``192.168.4.152``, conducting a HTTP session with

logs/smb.rst

@@ -118,7 +118,8 @@ traffic.
     "orig_pkts": 78,
     "orig_ip_bytes": 19403,
     "resp_pkts": 77,
-    "resp_ip_bytes": 16812
+    "resp_ip_bytes": 16812,
+    "ip_proto": 6
   }
 
 We see that ``192.168.10.31`` initiated a connection to ``192.168.10.10``. The
@@ -376,7 +377,8 @@ The :file:`conn.log` has two entries:
     "orig_pkts": 66,
     "orig_ip_bytes": 816703,
     "resp_pkts": 91,
-    "resp_ip_bytes": 15309
+    "resp_ip_bytes": 15309,
+    "ip_proto": 6
   }
   {
     "ts": 1507565425.183882,
@@ -392,7 +394,8 @@ The :file:`conn.log` has two entries:
     "orig_pkts": 1,
     "orig_ip_bytes": 207,
     "resp_pkts": 0,
-    "resp_ip_bytes": 0
+    "resp_ip_bytes": 0,
+    "ip_proto": 17
   }
 
 The first entry shows a connection initiated by ``192.168.10.31`` to
@@ -790,7 +793,8 @@ The :file:`conn.log` for this case has only one entry:
     "orig_pkts": 74,
     "orig_ip_bytes": 8734,
     "resp_pkts": 575,
-    "resp_ip_bytes": 835740
+    "resp_ip_bytes": 835740,
+    "ip_proto": 6
   }
 
 We see the same pattern: ``192.168.10.31`` initiated a connection to
@@ -1028,7 +1032,8 @@ Looking at the :file:`conn.log`, we see one entry:
     "orig_pkts": 11,
     "orig_ip_bytes": 1595,
     "resp_pkts": 9,
-    "resp_ip_bytes": 1397
+    "resp_ip_bytes": 1397,
+    "ip_proto": 6
   }
 
 We see ``192.168.10.31`` initiated a connection to ``192.168.10.30``, port 445

logs/tunnel.rst

@@ -146,7 +146,8 @@ The :file:`conn.log` for this traffic contains the following:
     "orig_pkts": 1,
     "orig_ip_bytes": 89,
     "resp_pkts": 1,
-    "resp_ip_bytes": 137
+    "resp_ip_bytes": 137,
+    "ip_proto": 17
   }
 
 This first :file:`conn.log` entry addresses frames 2 and 3 in the original packet
@@ -172,7 +173,8 @@ shortly.
     "orig_pkts": 1,
     "orig_ip_bytes": 89,
     "resp_pkts": 0,
-    "resp_ip_bytes": 0
+    "resp_ip_bytes": 0,
+    "ip_proto": 17
   }
 
 This second :file:`conn.log` entry refers to frame 1 in the packet capture.
@@ -196,7 +198,8 @@ Note the ``uid`` field. It will appear again shortly as well.
     "resp_ip_bytes": 0,
     "tunnel_parents": [
       **"CO9T0A3FPac5ig4hud"**
-    ]
+    ],
+    "ip_proto": 1
   }
 
 Here Zeek has created a new :file:`conn.log` entry for the ICMPv6 traffic