[RFC] Create Threat Fieldset - Stage 3 Proposal

[peasead]

• Have you signed the contributor license agreement? Yes
• Have you followed the contributor guidelines? Yes
• For proposing substantial changes or additions to the schema, have you reviewed the RFC process? Yes
• If submitting code/script changes, have you verified all tests pass locally using make test? Yes
• If submitting schema/fields updates, have you generated new artifacts by running make and committed those changes? Yes
• Is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed. Yes
• Have you added an entry to the CHANGELOG.next.md? NA

Stage 3 criteria:

• Opened pull request for this candidate revising the existing candidate
• Included multiple real world example source documents
• Existing or newly raised questions and concerns are addressed
• No objections from sponsor, ECS team, or subject matter experts

Preview of markdown proposal doc

[ebeahan]

Now that more implementation work and testing has been done, do we have any resolution we can capture under Concerns?

[peasead]

For # 1 under Concerns WRT to malware name and type, can we use the threat.software.type and threat.software.name fieldsets now?

@rylnd or @dcode under Concerns, is there anything that can be resolved WRT # 2?

[peasead]

For # 1 under Concerns WRT to malware name and type, can we use the threat.software.type and threat.software.name fieldsets now?

@rylnd or @dcode under Concerns, is there anything that can be resolved WRT # 2?

Just pinging on this @rylnd / @dcode

[rylnd]

@peasead I don't think this is still true; the plan for threat.indicator is to keep it as a normal field, as is already the case with threatintel.indicator. RE the migration below, I think the only change on the data shipper side is to rename threatintel.indicator to threat.indicator.

threat.enrichments is the nested field that will cover the enrichment use case; the caveats about kibana support still apply there for Detection Alerts and users manually adopting the threat.enrichments fields, but it should not impact data shippers at this time.

[rylnd]

Correct, I believe threat.enrichments bypasses the nested support issue by allowing:

1. Data shippers to generate normal threat.indicator fields
2. Enrichment mechanisms to populate nested threat.enrichments fields

Users wishing to ingest their own CTI data into indicator documents to be consumed within the detection engine can either:

1. Populate normal threatintel.indicator fields, a la the current threatintel filebeat modules, as that is the current standard and the default for indicator match rules (for 7.14)
2. Populate normal threat.indicator fields, as that will be the default enrichment path for indicator match rules in 7.15

[peasead]

Roger.

I'll update the Concern.

@ebeahan FYI ☝️

[peasead]

@ebeahan we're ready for review

[ebeahan]

I identified a couple of small items, but otherwise LGTM.

[ebeahan]

@devonakerr Would you provide your final review here as sponsor?

[peasead]

We're ready for a final review.
@rylnd
@devonakerr
@dcode

[devonakerr]

Looks good, remarkable how far we've come with the schema in such a short period.