Event categorisation fields #242

tsg · 2018-12-06T10:43:33Z

This PR is a minimal change to the current ECS fields that allows us to go in the planned path for categorization without requiring breaking changes later on. The hope is that we'll get this in for Beta2. It does the following:

Updates the docs for event.action and event.category to be closer to what we are planning, but keeps their definition fairly abstract. The docs don't go as far as showing a list of examples.
Changes the event.type definition to be only reserved for now.
Adds event.kind and event.outcome, which are fairly non-contentious. I can remove them from the PR if needed to get this in before Beta2.

I plan to follow up with a more complete PR that adds more details.

ruflin · 2018-12-06T12:00:02Z

fields.yml

+
+            This gives information about what type of information the event
+            contains, without being specific to the contents of the event.  Examples
+            are `event`, `state`, `alarm`.


Instead of event, perhaps we should use log here? For me all 3 kind are an event (based on ECS).

@cwurm Also mentioned action here as an option.

log is not great, btw, in Auditbeat we have events like "process start" that we get from a netlink socket, not from a "log".

Yes, I agree "event" is better than "log" for this.

I think the definition should be a bit more forceful in that this field must contain one of these values. The goal here is super broad categorization of how this document/event came about. So the values have to be predictable:

event: just reporting something that was observed in real-time (e.g. syslog message, log entry, event on a bus such as ebpf).

state: generating an update on the state of something on a regular schedule (list of running processes, installed packages, current CPU usage).

So for example, metrics from metricbeat should all use event.kind:state.

alarm: alerting on the fact that something unwanted happened, or a state changed enough to warrant attention from someone. Could be triggered by a static threshold, ML, a rules engine, etc.

What we want to avoid here is to have people map arbitrary values from their event sources to this field, and we end up with lots of unpredictable values in event.kind.

Of course the corollary is that if people determine that their stuff doesn't fall into one of these 3, we want them to open a GH issue to discuss it, and potentially grow that list.

I added a warning like for the others. I'm not sure we want to define event, state, alarm, etc just yet, there will be more than that and we'd do a poor job if we try to do it by the freeze date.

I think the defined list of values later on should go into a different document and potentially we have 1 doc per use case and the values for each use case might be different.

log for me does not mean it must come from a file. I'm more thinking of the historical definition an official record of events during the voyage of a ship or aircraft.

@cwurm Also mentioned action here as an option.

I think it was state_restart. I wanted a way to distinguish between a state that is triggered periodically (e.g. every 1h) and one triggered by a restart of (in this case) Auditbeat. One could imagine other reasons it was sent (e.g. a threshold was breached, or that too many changes accumulated). If we had something like an event.trigger field that captures why a document was sent that would also work - but maybe that's one field too many as well.

webmat

All in all, I'd like to get this in.

However I think we need to clearly warn people that a few of these fields will soon have prescribed content, therefore not to use them (or use at their own risk). event.type is pretty clear on this already. Here are the other ones:

event.kind
event.category
event.outcome

If we don't do this and people start using them widely, then suddenly coming up with the list of prescribed values is a breaking change.

webmat · 2018-12-06T18:08:16Z

.gitignore

@@ -1,3 +1,4 @@
 .DS_Store
 *.pyc
 env
+*.swp


LOL I have this in my global gitignore, so it never came up.

Please make it *.sw?, though 🙏🏼

webmat · 2018-12-06T18:09:02Z

fields.yml

+
+            This gives information about what type of information the event
+            contains, without being specific to the contents of the event.  Examples
+            are `event`, `state`, `alarm`.


Yes, I agree "event" is better than "log" for this.

webmat · 2018-12-06T18:13:59Z

fields.yml

+
+            This gives information about what type of information the event
+            contains, without being specific to the contents of the event.  Examples
+            are `event`, `state`, `alarm`.


I think the definition should be a bit more forceful in that this field must contain one of these values. The goal here is super broad categorization of how this document/event came about. So the values have to be predictable:

event: just reporting something that was observed in real-time (e.g. syslog message, log entry, event on a bus such as ebpf).

state: generating an update on the state of something on a regular schedule (list of running processes, installed packages, current CPU usage).

So for example, metrics from metricbeat should all use event.kind:state.

alarm: alerting on the fact that something unwanted happened, or a state changed enough to warrant attention from someone. Could be triggered by a static threshold, ML, a rules engine, etc.

What we want to avoid here is to have people map arbitrary values from their event sources to this field, and we end up with lots of unpredictable values in event.kind.

webmat · 2018-12-06T18:16:40Z

fields.yml

+
+            This gives information about what type of information the event
+            contains, without being specific to the contents of the event.  Examples
+            are `event`, `state`, `alarm`.


Of course the corollary is that if people determine that their stuff doesn't fall into one of these 3, we want them to open a GH issue to discuss it, and potentially grow that list.

webmat · 2018-12-06T18:19:45Z

fields.yml

+            This contains high-level information about the contents of the event. It
+            is more generic than `event.action`, in the sense that typically a
+            category contains multiple actions.
+          example: user-management


The plan with this field is to have a predetermined set of categories (a few tens of them). Having the field now in ECS sets us up for a sort of breaking change when we come up with that list.

Therefore I think the description can remain as is, but we should add a warning telling people that they're using this field at their own risk, because it will soon be a field that should be populated with prescribed values.

It won't be the end of the world, though. It's not going to break ingestion or anything. It's just that Elastic solutions will expect this field to contain certain values, so if some sources populate this differently, they won't have the best experience...

warning added.

webmat · 2018-12-06T18:23:16Z

fields.yml


-        - name: action
+        - name: outcome


outcome is also meant to be a field with prescribed content. We should add a similar warning as what I suggest for event.category. More or less "We're about to come up with a list of prescribed values, so use with caution."

warning added.

tsg · 2018-12-06T19:12:46Z

@webmat I think I addressed your comments, please have a look again.

webmat

LGTM

webmat · 2018-12-06T19:27:25Z

Please wait for approval by @MikePaquette and @ruflin before merging. This is a big one.

ruflin · 2018-12-07T08:10:16Z

fields.yml

@@ -396,35 +396,64 @@
            Unique ID to describe the event.
          example: 8a4f500d

+        - name: kind
+          level: core


Sorry, only spotted now but these fields should go into extended for now.

I moved event.kind and event.outcome to extended.

I'll be proposing pushing these to core sometime in the future :-)

ruflin · 2018-12-07T12:06:40Z

@tsg I think you need to run make again locally to update the generated files.

tsg · 2018-12-07T12:10:20Z

@ruflin, thanks, I pushed again.

MikePaquette

LGTM

tsg requested review from ruflin and MikePaquette December 6, 2018 10:45

ruflin reviewed Dec 6, 2018

View reviewed changes

ruflin added the 1.0.0-beta2 label Dec 6, 2018

webmat suggested changes Dec 6, 2018

View reviewed changes

tsg added 4 commits December 6, 2018 20:08

Event categorization fields

a6ae7bd

updated changelog

9d9fe13

Addressed comments: added warning for some fields

acb86a0

cleanup changelog

5d372fb

tsg force-pushed the categorization-minimal branch from 3161db9 to 5d372fb Compare December 6, 2018 19:09

webmat approved these changes Dec 6, 2018

View reviewed changes

ruflin requested changes Dec 7, 2018

View reviewed changes

Move and to extended

8145ca9

make

22e190d

ruflin approved these changes Dec 7, 2018

View reviewed changes

MikePaquette approved these changes Dec 7, 2018

View reviewed changes

tsg merged commit 6439b8a into elastic:master Dec 7, 2018

ceeeekay mentioned this pull request Dec 18, 2018

Clarification for type vs event.type #290

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Event categorisation fields #242

Event categorisation fields #242

tsg commented Dec 6, 2018

ruflin Dec 6, 2018

tsg Dec 6, 2018

tsg Dec 6, 2018

webmat Dec 6, 2018

webmat Dec 6, 2018

webmat Dec 6, 2018

tsg Dec 6, 2018

ruflin Dec 7, 2018

cwurm Dec 7, 2018

webmat left a comment

webmat Dec 6, 2018

webmat Dec 6, 2018

webmat Dec 6, 2018

webmat Dec 6, 2018

webmat Dec 6, 2018

tsg Dec 6, 2018

webmat Dec 6, 2018

tsg Dec 6, 2018

tsg commented Dec 6, 2018

webmat left a comment

webmat commented Dec 6, 2018

ruflin Dec 7, 2018

tsg Dec 7, 2018

MikePaquette Dec 7, 2018

ruflin commented Dec 7, 2018

tsg commented Dec 7, 2018

MikePaquette left a comment

Event categorisation fields #242

Event categorisation fields #242

Conversation

tsg commented Dec 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg commented Dec 6, 2018

webmat left a comment

Choose a reason for hiding this comment

webmat commented Dec 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Dec 7, 2018

tsg commented Dec 7, 2018

MikePaquette left a comment

Choose a reason for hiding this comment