[RFC] Threat Enrichment - Stage 3

[rylnd]

• Have you signed the contributor license agreement? Yes
• Have you followed the contributor guidelines? Yes
• For proposing substantial changes or additions to the schema, have you reviewed the RFC process? Yes
• If submitting code/script changes, have you verified all tests pass locally using make test? Yes
• If submitting schema/fields updates, have you generated new artifacts by running make and committed those changes? Yes
• Is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed. Yes
• Have you added an entry to the CHANGELOG.next.md? NA

Stage 3 criteria:

• Opened pull request for this candidate revising the existing candidate
• Included multiple real world example source documents
• Existing or newly raised questions and concerns are addressed
• No objections from sponsor, ECS team, or subject matter experts

Preview of markdown proposal doc

[rylnd]

Note: no significant changes have yet been identified for this PR. We've adopted these beta fields in various CTI features as part of kibana's 7.15.0 release, and the current plan is to leave this open, acquire feedback from the 7.15 release, and then close stage 3 once we've determined 7.15 to be stable with regards to these fields.

[rylnd]

@ebeahan @peasead I've updated the confidence field here to reflect the changes from #1480 (9343135). As I mention in the commit, it would be great if the tooling could support that automatically.

@ebeahan should I file a feature request for something like that? I'm not really sure what it would take in terms of implementation, but I could certainly describe it from a user perspective.

[ebeahan]

The changes to the confidence field in RFC 0008 were added to the schema in #1586 (8060855).

[rylnd]

One small change that has been brought up in recent discussions: the ability to capture when an enrichment occurred, so as to view the most recent enrichments, audit enrichments, etc.

To me this makes the most sense as a matched field; matched.at or matched.occurred, etc. Any thoughts around the most consistent name, here?

[kgeller]

@rylnd I lean matched.occurred, I think that's more in line with others we have

[devonakerr]

Looks good, Ryland.

[kgeller]

LGTM!

I'll update the date here and get this merged, and get a PR up shortly for those implementation updates.

[djptek]

Hi @ryland @devonakerr @kgeller @ebeahan

I have a usage question regarding threat.enrichments.indicator.type vs threat.enrichments.type.

I'm working on an integration where some types of alerts have multiple indicators, and others don't. At some point I expect to build a dashboard based on a TERMS aggregation across the values that may be contained in both fields.

To avoid run-time performance issues, I'd like to normalise the STIX indicators to a single field.

Would it make sense to insert a nominal indicator and then populate threat.enrichments.indicator.type for events that do not have multiple indicators?

I am inclined towards this approach since the type could differ across indicators so it would be non-trivial to resolve which value to copy from e.g. threat.enrichments.indicator.type up the hierarchy to threat.enrichments.type.

[djptek]

What kind of Aggregations are you aiming to run with the threat.enrichments[] array?

[rylnd]

@djptek I'm not aware of a threat.enrichments.type field; there is no "parent" field to represent the type of all the indicators. threat.enrichments[].indicator.type represents a given indicator's type, threat.enrichments[].matched.type represents how that indicator was found/enriched.

If you're trying to reduce multiple values to a single one, the logic to do this sounds specific to your need. Just note that those indicators are nested documents, so your query/transformation would likely need to account for that.

Hopefully this helps point you in the right direction; if not, getting more details about your problem/goals might be needed.

[djptek]

Hi @rylnd

I'm not aware of a threat.enrichments.type field

sorry, typo, I intended to refer to threat.indicator.type

my concern is that the indicators (in my use case) are really separate events and encapsulating them all in a single event using an array:

• reduces potential for aggregations across similar events
• prevents aggregation across events e.g. where you might wish to compare threat.enrichments[].indicator.type against threat.indicator.type

Thanks to @andrewkroh for directing me to how to split an event into multiple events i.e. denormalise I wasn't aware that we'd (added?) that, it has been a long time ask elsewhere in the stack

Did you consider the possibility of denormalising multiple indicators into separate events (with duplicate parent metadata) as an alternative to adding the threat.enrichments[]array?