Adding support for enabled in templates and beats

[jonathan-buttner]

This PR adds support for the mapping enabled field.
https://www.elastic.co/guide/en/elasticsearch/reference/current/enabled.html

If the enabled field is included and set to false in a schema file (most likely a custom schema file) the generated template and beats output will include the field. enabled is only valid for object and nested types.

The enabled field is also supported at the top level of a mapping but I have not implemented that here.

I also added index support in the beats generator.

As a side note, it would probably be useful to extend the subset format to allow for setting ecs core fields as enabled = false. Currently this change only allows for custom schema to explicitly be set to enabled = false. A user would have to edit the core ecs files temporarily to force those fields to be enabled = false. I think this can be tackled in a follow up PR if we think it'd be useful.

Example generated beats output

# WARNING! Do not edit this file directly, it was generated by the ECS project,
# based on ECS version 1.6.0-dev.
# Please visit https://github.com/elastic/ecs to suggest changes to ECS fields.

- key: ecs
  title: ECS
  description: ECS Fields.
  fields:
  - name: message
    level: core
    type: text
    description: 'For log events the message field contains the log message, optimized
      for viewing in a log viewer.

      For structured logs without an original message field, other fields can be concatenated
      to form a human-readable summary of the event.

      If multiple messages exist, they can be combined into one message.'
    example: Hello World
  - name: file
    title: File
    group: 2
    description: 'A file is defined as a set of information that has been created
      on, or has existed on a filesystem.

      File objects can be associated with host events, network events, and/or file
      events (e.g., those produced by File Integrity Monitoring [FIM] products or
      services). File fields provide details about the affected file associated with
      the event or metric.'
    type: group
    fields:
    - name: entry_modified
      level: custom
      type: object
      object_type: keyword
      description: Time of last status change.  See `st_ctim` member of `struct stat`.
      enabled: false <------ enabled added
      default_field: false
    - name: other_field
      level: custom
      type: keyword
      ignore_above: 1024
      description: other field
      index: false <------ index added
      default_field: false

[ruflin]

I wonder what the final template will look like here? Will it still index the text part but skip keyword? Or does it apply to both?

[jonathan-buttner]

Ah good catch. So I tried testing a similar scenario in ES where the outer field error.stack_trace has index: false but the inner one error.stack_trace.text does not have that. Looks like error.stack_trace is not searchable as expected, but error.stack_trace.text will be. The template that ecs generates for this schema is here: https://github.com/elastic/ecs/blob/master/generated/elasticsearch/7/template.json#L650-L661 which hasn't changed from the edits I've made here.

@webmat is it intentional to have the error.stack_trace.text be searchable while the error.stack_trace.keyword is not searchable?

[webmat]

Not intentional, no.

Also, this highlights the fact that despite flagging a few fields as index: False in ECS, this never made it to the Beats mapping. Oops 🙊. Changing this now would cause an expected breaking change for Beats, IMO.

The addition of this feature makes total sense, but we'll need to decide what we do with this Beats change.

[jonathan-buttner]

@webmat who should I reach out to get approval for making the breaking changes for beats? On the endpoint side having ecs add the index and enabled fields in beats output would be very helpful because we use the beats output to build our mapping.

[webmat]

I think @andrewkroh would be your best bet

[andrewkroh]

In elastic/beats log.original and event.original are used, but I didn't find anything that would be querying them. So I think the only effect would be if users had custom queries (like a regexp query on the event.original).

And nothing in beats populates error.stack_trace from what I could find. @simitt would anything in APM be affected by setting index: false for error.stack_trace (like does anything query error.stack_trace)? I imagine anything that needs to search it would use error.stack_trace.text.

[simitt]

Thanks for the ping; APM does not use event.original, log.original nor error.stack_trace at the moment.

[jonathan-buttner]

Thanks for taking a look @andrewkroh and @simitt. Is there anything else we could do to mitigate the breaking change risk?

[jonathan-buttner]

Any other concerns with merging this PR?

[webmat]

LGTM. Thanks @jonathan-buttner.

Just one nit, could you split the changelog in two, according to the comment below, please? You can merge right after.

[webmat]

Suggested change

	* Adding support for `enabled` and `index` fields. #824
	* Add support for Elasticsearch `enabled` field parameter. #824

And in the "Bugfixes" section

* Field parameter `index` is now correctly populated in the Beats field definition file. #824

[jonathan-buttner]

Thanks @webmat done!