Rename event.raw to log.message

The `event` prefix should only contain meta information about the event itself. `event.raw` contained actual event data and it was not the place I was searching for the field.

All events which make use of a raw message are log events. I don't expect metric events to require the field. Because of this putting it under `log` seemed natural to me.

The field `log.message` is a keyword to make sure it only matches exact searches and is not confused with the `message` field. It can be use to reprocessing of log events or to show log integrity.

CHANGELOG.md

@@ -6,6 +6,8 @@ All notable changes to this project will be documented in this file based on the
 
 ### Breaking changes
 
+* Rename `event.raw` to `log.message`. #3
+
 ### Bugfixes
 
 ### Added

README.md

@@ -166,7 +166,6 @@ The event fields are used for context information about the data itself.
 | <a name="event.module"></a>`event.module`  | Name of the module this data is coming from.<br/>This information is coming from the modules used in Beats or Logstash.  | keyword  |   | `mysql`  |
 | <a name="event.dataset"></a>`event.dataset`  | Name of the dataset.<br/>The concept of a `dataset` (fileset / metricset) is used in Beats as a subset of modules. It contains the information which is currently stored in metricset.name and metricset.module or fileset.name.  | keyword  |   | `stats`  |
 | <a name="event.severity"></a>`event.severity`  | Severity describes the severity of the event. What the different severity values mean can very different between use cases. It's up to the implementer to make sure severities are consistent across events.  | long  |   | `7`  |
-| <a name="event.raw"></a>`event.raw`  | Raw text message of entire event to be used to demonstrate log integrity.  | keyword  |   | `Sep 19 08:26:10 host CEF:0&#124;Security&#124; threatmanager&#124;1.0&#124;100&#124; worm successfully stopped&#124;10&#124;src=10.0.0.1 dst=2.1.2.2spt=1232`  |
 | <a name="event.hash"></a>`event.hash`  | Hash (perhaps logstash fingerprint) of raw field to be able to demonstrate log integrity.  | keyword  |   | `123456789012345678901234567890ABCD`  |
 | <a name="event.version"></a>`event.version`  | The version field contains the version an event for ECS adheres to.<br/>This field should be provided as part of each event to make it possible to detect to which ECS version an event belongs.<br/>event.version is a required field and must exist in all events. It describes which ECS version the event adheres to.<br/>The current version is 0.1.0.  | keyword  |   | `0.1.0`  |
 | <a name="event.duration"></a>`event.duration`  | Duration of the event in nanoseconds.  | long  |   |   |
@@ -261,6 +260,7 @@ Fields which are specific to log events.
 | <a name="log.level"></a>`log.level`  | Log level of the log event.<br/>Some examples are `WARN`, `ERR`, `INFO`.  | keyword  |   | `ERR`  |
 | <a name="log.line"></a>`log.line`  | Line number the log event was collected from.  | long  |   | `18`  |
 | <a name="log.offset"></a>`log.offset`  | Offset of the beginning of the log event.  | long  |   | `12`  |
+| <a name="log.message"></a>`log.message`  | This is the raw log message and contains the full unprocessed log message.<br/>In contrast to the `message` field which can contain an extracted part of the log message, this field contains the raw log message and should not be processed.<br/>The field can be used to demonstrate log integrity of for reprocessing logs with an improved processor.  | keyword  |   | `Sep 19 08:26:10 localhost My log`  |
 
 
 ## <a name="network"></a> Network fields

schema.csv

@@ -43,7 +43,6 @@ event.duration,long,0,
 event.hash,keyword,1,123456789012345678901234567890ABCD
 event.id,keyword,1,8a4f500d
 event.module,keyword,0,mysql
-event.raw,keyword,1,Sep 19 08:26:10 host CEF:0|Security| threatmanager|1.0|100| worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2spt=1232
 event.risk_score,float,0,
 event.severity,long,1,7
 event.type,keyword,0,nginx-stats-metrics
@@ -85,6 +84,7 @@ kubernetes.namespace,keyword,0,
 kubernetes.pod.name,keyword,0,
 log.level,keyword,0,ERR
 log.line,long,0,18
+log.message,keyword,1,Sep 19 08:26:10 localhost My log
 log.offset,long,0,12
 network.direction,keyword,0,inbound
 network.forwarded_ip,ip,0,192.1.1.2

schemas/event.yml

@@ -54,21 +54,6 @@
         severity values mean can very different between use cases. It's up to
         the implementer to make sure severities are consistent across events.
 
-    - name: raw
-      type: keyword
-      phase: 1
-      # Unfortunately this example is not shown correctly yet as | do not work
-      # in tables well
-      # Is | is the representation of | it works except for cases where
-      # used ticks.
-      example: "Sep 19 08:26:10 host CEF:0|Security|
-          threatmanager|1.0|100|
-          worm successfully stopped|10|src=10.0.0.1
-          dst=2.1.2.2spt=1232"
-      description: >
-          Raw text message of entire event to be used to demonstrate log
-          integrity.
-
     - name: hash
       type: keyword
       phase: 1

schemas/log.yml

@@ -21,3 +21,16 @@
       description: >
         Offset of the beginning of the log event.
       example: 12
+    - name: message
+      type: keyword
+      phase: 1
+      example: "Sep 19 08:26:10 localhost My log"
+      description: >
+        This is the raw log message and contains the full unprocessed log message.
+
+        In contrast to the `message` field which can contain an extracted
+        part of the log message, this field contains the raw log message and
+        should not be processed.
+
+        The field can be used to demonstrate log integrity of for reprocessing
+        logs with an improved processor.

template.json

@@ -222,10 +222,6 @@
               "ignore_above": 1024,
               "type": "keyword"
             },
-            "raw": {
-              "ignore_above": 1024,
-              "type": "keyword"
-            },
             "risk_score": {
               "type": "float"
             },
@@ -434,6 +430,10 @@
             "line": {
               "type": "long"
             },
+            "message": {
+              "ignore_above": 1024,
+              "type": "keyword"
+            },
             "offset": {
               "type": "long"
             }
@@ -585,12 +585,6 @@
               "type": "keyword"
             },
             "href": {
-              "fields": {
-                "analyzed": {
-                  "norms": false,
-                  "type": "text"
-                }
-              },
               "ignore_above": 1024,
               "type": "keyword"
             },