Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change device to observer #238

Merged
merged 6 commits into from
Dec 7, 2018
Merged

Change device to observer #238

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
07b058f
change device to observer
MikePaquette Dec 5, 2018
da4c345
Update CHANGELOG.md
MikePaquette Dec 5, 2018
1fa06fd
Update geo, os descriptions
MikePaquette Dec 5, 2018
f805cf9
Update agent, related, to change device to observer
MikePaquette Dec 5, 2018
3f62009
backed out change to CONTRIBUTING.md
MikePaquette Dec 6, 2018
8278c21
re-fixed CONTRIBUING after re-base
MikePaquette Dec 6, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ All notable changes to this project will be documented in this file based on the
## Unreleased

### Breaking changes
* Changed `device.*` fields to `observer.*` fields to eliminate user confusion. #238

* Rename `network.total.bytes` to `network.bytes` and `network.total.packets`
to `network.packets`. #179
Expand Down
9 changes: 9 additions & 0 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,15 @@ The `type` is the [Elasticsearch field type](https://www.elastic.co/guide/en/ela
`description` adds details about the field.
`example` adds an sample value.

## Phases

A field's `phase` indicates where it is in its development. Different phases exist to make it easy for you to contribute new fields, while allowing for more iteration. Here are the phases:

* 0 (alpha): The field is new and is up for discussion as to whether or not it should be added. The field might be removed at any time.
* 1 (beta): It's clear that there is value of having the field in ECS and discussions about prefixes/naming have started. It's unlikely that the field will be removed, but naming might change at any time.
* 2 (rc): The field has been accepted and is unlikely to change. It is now tested in the field.
* 3 (GA): The field is part of ECS. Any breaking changes will happen only at a major release.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So yeah, please make sure to remove this section before merging ;-)

## Guidelines for implementing ECS

* The document MUST have the `@timestamp` field.
Expand Down
42 changes: 21 additions & 21 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ ECS defines these fields.
* [Cloud fields](#cloud)
* [Container fields](#container)
* [Destination fields](#destination)
* [Device fields](#device)
* [ECS fields](#ecs)
* [Error fields](#error)
* [Event fields](#event)
Expand All @@ -63,6 +62,7 @@ ECS defines these fields.
* [HTTP fields](#http)
* [Log fields](#log)
* [Network fields](#network)
* [Observer fields](#observer)
* [Organization fields](#organization)
* [Operating System fields](#os)
* [Process fields](#process)
Expand All @@ -87,7 +87,7 @@ The base set contains all fields which are on the top level. These fields are co

## <a name="agent"></a> Agent fields

The agent fields contain the data about the software entity, if any, that collects, detects, or observes events on a host, or takes measurements on a host. Examples include beats. Agents may also run on devices. ECS agent.* fields shall be populated with details of the agent running on the host or device where the event happened or the measurement was taken.
The agent fields contain the data about the software entity, if any, that collects, detects, or observes events on a host, or takes measurements on a host. Examples include Beats. Agents may also run on observers. ECS agent.* fields shall be populated with details of the agent running on the host or observer where the event happened or the measurement was taken.


| Field | Description | Level | Type | Example |
Expand Down Expand Up @@ -151,22 +151,6 @@ Destination fields describe details about the destination of a packet/event.
| <a name="destination.packets"></a>destination.packets | Packets sent from the destination to the source. | core | long | `12` |


## <a name="device"></a> Device fields

Device fields are used to provide additional information about the device that is the source of the information. This could be a firewall, network device, etc.


| Field | Description | Level | Type | Example |
|---|---|---|---|---|
| <a name="device.mac"></a>device.mac | MAC address of the device | core | keyword | |
| <a name="device.ip"></a>device.ip | IP address of the device. | core | ip | |
| <a name="device.hostname"></a>device.hostname | Hostname of the device. | core | keyword | |
| <a name="device.vendor"></a>device.vendor | Device vendor information. | core | keyword | |
| <a name="device.version"></a>device.version | Device version. | core | keyword | |
| <a name="device.serial_number"></a>device.serial_number | Device serial number. | extended | keyword | |
| <a name="device.type"></a>device.type | The type of the device the data is coming from.<br/>There is no predefined list of device types. Some examples are `endpoint`, `firewall`, `ids`, `ips`, `proxy`. | core | keyword | `firewall` |


## <a name="ecs"></a> ECS fields

Meta-information specific to ECS.
Expand Down Expand Up @@ -241,7 +225,7 @@ A file is defined as a set of information that has been created on, or has exist
Geo fields can carry data about a specific location related to an event or geo information derived from an IP field.


The `geo` fields are expected to be nested at: `destination.geo`, `device.geo`, `host.geo`, `source.geo`.
The `geo` fields are expected to be nested at: `destination.geo`, `host.geo`, `observer.geo`, `source.geo`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏼


Note also that the `geo` fields are not expected to be used directly at the top level.

Expand Down Expand Up @@ -328,6 +312,22 @@ The network is defined as the communication path over which a host or network ev
| <a name="network.packets"></a>network.packets | Total packets transferred in both directions.<br/>If `source.packets` and `destination.packets` are known, `network.packets` is their sum. | core | long | `24` |


## <a name="observer"></a> Observer fields

An observer is defined as a special network, security, or application device used to detect, observe, or create network, security, or application-related events and metrics. This could be a custom hardware appliance or a server that has been configured to run special network, security, or application software. Examples include firewalls, intrusion detection/prevention systems, network monitoring sensors, web application firewalls, data loss prevention systems, and APM servers. The observer.* fields shall be populated with details of the system, if any, that detects, observes and/or creates a network, security, or application event or metric. Message queues and ETL components used in processing events or metrics are not considered observers in ECS.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@graphaelli Can you leave your comment on this one? APM-Server is in here :-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Elastic APM server typically operates as more as a message queue using this definition. Would logstash be considered an observer?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say even if APM server doesn't do a lot of work on the payload, compared to the library installed in each app, it could still be considered as the observer.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it makes sense for an observer simply to be an entity that works on behalf of an agent to report its data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@graphaelli I think our definition of observer needs to extend beyond what you suggest to include firewalls and network IDS, etc. that don't have an agent at all.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@graphaelli Yes, LS is in many cases also the observer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that compared to the agent the server doesn't do a lot with the payload currently. However, I am not sure that this is a general enough criteria for not considering the server an ETL component. There are some additions and changes the server does, and going forward with ECS, the server will change the structure of events even more in the future, as there need to be a mapping from the incoming events to the ES ECS structure.

My point is, I assume that

Message queues and ETL components used in processing events or metrics are not considered observers in ECS.

should clarify what to consider an observer and what not, but for me it introduces more confusion.



| Field | Description | Level | Type | Example |
|---|---|---|---|---|
| <a name="observer.mac"></a>observer.mac | MAC address of the observer | core | keyword | |
| <a name="observer.ip"></a>observer.ip | IP address of the observer. | core | ip | |
| <a name="observer.hostname"></a>observer.hostname | Hostname of the observer. | core | keyword | |
| <a name="observer.vendor"></a>observer.vendor | observer vendor information. | core | keyword | |
| <a name="observer.version"></a>observer.version | Observer version. | core | keyword | |
| <a name="observer.serial_number"></a>observer.serial_number | Observer serial number. | extended | keyword | |
| <a name="observer.type"></a>observer.type | The type of the observer the data is coming from.<br/>There is no predefined list of observer types. Some examples are `forwarder`, `firewall`, `ids`, `ips`, `proxy`, `poller`, `sensor`, `APM server`. | core | keyword | `firewall` |


## <a name="organization"></a> Organization fields

The organization fields enrich data with information about the company or entity the data is associated with. These fields help you arrange or filter data stored in an index by one or multiple organizations.
Expand All @@ -344,7 +344,7 @@ The organization fields enrich data with information about the company or entity
The OS fields contain information about the operating system.


The `os` fields are expected to be nested at: `device.os`, `host.os`, `user_agent.os`.
The `os` fields are expected to be nested at: `host.os`, `observer.os`, `user_agent.os`.

Note also that the `os` fields are not expected to be used directly at the top level.

Expand Down Expand Up @@ -379,7 +379,7 @@ These fields contain information about a process. These fields can help you corr

This field set is meant to facilitate pivoting around a piece of data. Some pieces of information can be seen in many places in ECS. To facilitate searching for them, append values to their corresponding field in `related.`.

A concrete example is IP addresses, which can be under host, device, source, destination, and network.forwarded_ip. If you append all IPs to `related.ip`, you can then search for a given IP trivially, no matter where it appeared, by querying `related.ip:a.b.c.d`.
A concrete example is IP addresses, which can be under host, observer, source, destination, and network.forwarded_ip. If you append all IPs to `related.ip`, you can then search for a given IP trivially, no matter where it appeared, by querying `related.ip:a.b.c.d`.


| Field | Description | Level | Type | Example |
Expand Down
103 changes: 51 additions & 52 deletions fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
title: Agent
group: 2
description: >
The agent fields contain the data about the software entity, if any, that collects, detects, or observes events on a host, or takes measurements on a host. Examples include beats. Agents may also run on devices. ECS agent.* fields shall be populated with details of the agent running on the host or device where the event happened or the measurement was taken.
The agent fields contain the data about the software entity, if any, that collects, detects, or observes events on a host, or takes measurements on a host. Examples include Beats. Agents may also run on observers. ECS agent.* fields shall be populated with details of the agent running on the host or observer where the event happened or the measurement was taken.
footnote: >
Examples: In the case of Beats for logs, the agent.name is filebeat. For APM, it is the
agent running in the app/service. The agent information does not change if
Expand Down Expand Up @@ -284,54 +284,6 @@
description: >
Packets sent from the destination to the source.

- name: device
title: Device
group: 2
description: >
Device fields are used to provide additional information about the device
that is the source of the information. This could be a firewall, network device, etc.
type: group
fields:
- name: mac
level: core
type: keyword
description: >
MAC address of the device
- name: ip
level: core
type: ip
description: >
IP address of the device.
- name: hostname
level: core
type: keyword
description: >
Hostname of the device.
- name: vendor
level: core
type: keyword
description: >
Device vendor information.
- name: version
level: core
type: keyword
description: >
Device version.
- name: serial_number
level: extended
type: keyword
description: >
Device serial number.
- name: type
level: core
type: keyword
description: >
The type of the device the data is coming from.

There is no predefined list of device types. Some examples are
`endpoint`, `firewall`, `ids`, `ips`, `proxy`.
example: firewall

- name: ecs
title: ECS
group: 2
Expand Down Expand Up @@ -633,7 +585,7 @@
top_level: false
expected:
- destination
- device
- observer
- host
- source
type: group
Expand Down Expand Up @@ -966,6 +918,53 @@
If `source.packets` and `destination.packets` are known, `network.packets` is their sum.
example: 24

- name: observer
title: Observer
group: 2
description: >
An observer is defined as a special network, security, or application device used to detect, observe, or create network, security, or application-related events and metrics. This could be a custom hardware appliance or a server that has been configured to run special network, security, or application software. Examples include firewalls, intrusion detection/prevention systems, network monitoring sensors, web application firewalls, data loss prevention systems, and APM servers. The observer.* fields shall be populated with details of the system, if any, that detects, observes and/or creates a network, security, or application event or metric. Message queues and ETL components used in processing events or metrics are not considered observers in ECS.
type: group
fields:
- name: mac
level: core
type: keyword
description: >
MAC address of the observer
- name: ip
level: core
type: ip
description: >
IP address of the observer.
- name: hostname
level: core
type: keyword
description: >
Hostname of the observer.
- name: vendor
level: core
type: keyword
description: >
observer vendor information.
- name: version
level: core
type: keyword
description: >
Observer version.
- name: serial_number
level: extended
type: keyword
description: >
Observer serial number.
- name: type
level: core
type: keyword
description: >
The type of the observer the data is coming from.

There is no predefined list of observer types. Some examples are
`forwarder`, `firewall`, `ids`, `ips`, `proxy`, `poller`, `sensor`, `APM server`.
example: firewall

- name: organization
title: Organization
group: 2
Expand Down Expand Up @@ -995,7 +994,7 @@
reusable:
top_level: false
expected:
- device
- observer
- host
- user_agent
type: group
Expand Down Expand Up @@ -1124,7 +1123,7 @@
searching for them, append values to their corresponding field in
`related.`.

A concrete example is IP addresses, which can be under host, device, source,
A concrete example is IP addresses, which can be under host, observer, source,
destination, and network.forwarded_ip. If you append all IPs to
`related.ip`, you can then search for a given IP trivially,
no matter where it appeared, by querying `related.ip:a.b.c.d`.
Expand Down
14 changes: 7 additions & 7 deletions schema.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,6 @@ destination.ip,ip,core,
destination.mac,keyword,core,
destination.packets,long,core,12
destination.port,long,core,
device.hostname,keyword,core,
device.ip,ip,core,
device.mac,keyword,core,
device.serial_number,keyword,extended,
device.type,keyword,core,firewall
device.vendor,keyword,core,
device.version,keyword,core,
ecs.version,keyword,core,1.0.0-beta1
error.code,keyword,core,
error.id,keyword,core,
Expand Down Expand Up @@ -101,6 +94,13 @@ network.packets,long,core,24
network.protocol,keyword,core,http
network.transport,keyword,core,TCP
network.type,keyword,core,IPv4
observer.hostname,keyword,core,
observer.ip,ip,core,
observer.mac,keyword,core,
observer.serial_number,keyword,extended,
observer.type,keyword,core,firewall
observer.vendor,keyword,core,
observer.version,keyword,core,
organization.id,keyword,extended,
organization.name,keyword,extended,
os.family,keyword,extended,debian
Expand Down
2 changes: 1 addition & 1 deletion schemas/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
title: Agent
group: 2
description: >
The agent fields contain the data about the software entity, if any, that collects, detects, or observes events on a host, or takes measurements on a host. Examples include beats. Agents may also run on devices. ECS agent.* fields shall be populated with details of the agent running on the host or device where the event happened or the measurement was taken.
The agent fields contain the data about the software entity, if any, that collects, detects, or observes events on a host, or takes measurements on a host. Examples include Beats. Agents may also run on observers. ECS agent.* fields shall be populated with details of the agent running on the host or observer where the event happened or the measurement was taken.
footnote: >
Examples: In the case of Beats for logs, the agent.name is filebeat. For APM, it is the
agent running in the app/service. The agent information does not change if
Expand Down
48 changes: 0 additions & 48 deletions schemas/device.yml
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion schemas/geo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
top_level: false
expected:
- destination
- device
- observer
- host
- source
type: group
Expand Down
Loading