threat fields #505

dainperkins · 2019-07-22T19:24:01Z

added threat fields for e.g. threat reports from IDS/IPS, NGFW, etc...

added threat fields

webmat

Thanks a lot for submitting this!

There's a few things that need discussion & adjustments, but looking very good overall.

schemas/threat.yml

CHANGELOG.next.md

schemas/threat.yml

webmat · 2019-07-23T14:10:09Z

Also, please make sure to run a full make before pushing.

Right now CI is failing because schema.json hasn't been updated. I think your merge of the master branch brought in a fix to generate this file, that's why your previous commit wasn't modifying that file.

added text miulti fields, modified tactic/technique to be suggested as Mitre Att&ck, added possible qualitative options

added multifields for fts, modified tactic and technique to suggest ATT&CK, etc.

dainperkins · 2019-08-13T15:58:32Z

Works for me Mike (was thinking the same thing but was unsure of which to separate) - I'll update today

…

On Tue, Aug 13, 2019 at 9:46 AM Mike Paquette ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In schemas/threat.yml <#505 (comment)>: > + + - name: severity + level: extended + type: keyword + short: Severity of the threat as identified by the observer/ threat intel feed + description: > + Typically a qualitative description of the risk associated with the observed threat or IOC + example: high + + - name: confidence + level: extended + type: keyword + short: Predicted accuracy of the threat identification + description: > + Typically a qualitative description of the confidence that the threat identification is correct + example: medium @dainperkins <https://github.com/dainperkins> I would prefer that we add fields threat.tactic.reference and threat.technique.reference to hold hyperlinks to reference material, rather than including the URL in the description field. I also suggest that we use a .name and .id for each tactic and technique, as listed in #291 <#291>, so that would give us: - threat.tactic.id - threat.tactic.name - threat.tactic.reference - threat.technique.id - threat.technique.name - threat.technique.reference — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#505?email_source=notifications&email_token=AJMZFSPDQFRPX3YLQ2D3LELQEK3LDA5CNFSM4IF32FJKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCBMWBKA#discussion_r313404687>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJMZFSLGFPBMHFWDWDPGEG3QEK3LDANCNFSM4IF32FJA> .

dainperkins · 2019-08-13T15:59:10Z

Removed the lowercase, and updated the desc in the newest version

…

On Tue, Aug 13, 2019 at 9:46 AM Mike Paquette ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In schemas/threat.yml <#505 (comment)>: > + level: extended + type: keyword + short: Name or classification of the threat detected or reported + description: > + Threat name as reported by system or intel provider + + The field value must be normalized to lowercase for querying. See + the documentation section "Implementing ECS". + example: ntp reserved mode denial of service + + - name: tactic + level: extended + type: keyword + short: Mitre ATT&CK Matrix tactic + description: > + The Mitre ATT&CK Matrix Tactic categorization of the identified threat @webmat <https://github.com/webmat> @dainperkins <https://github.com/dainperkins> I think ECS should not "require" the MITRE ATT&CK framework, but should allow it, and as @webmat <https://github.com/webmat> suggests, making this explicit in the descriptions of the fields. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#505?email_source=notifications&email_token=AJMZFSJMJ4IVW6EFRTXFKBDQEK3KJA5CNFSM4IF32FJKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCBMWAEA#discussion_r313404572>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJMZFSIVBHCEQDT6K3IA6RTQEK3KJANCNFSM4IF32FJA> .

dainperkins · 2019-08-27T01:07:55Z

Not really sure what to do about the conflicts...

Updated threat fields per recommendations

dainperkins · 2019-08-27T01:21:16Z

I think I figured it out

MikePaquette

Dropped a few comments that need fixing but overall looks good.

schemas/threat.yml

revised short desc. to better align to tootip thoughts

webmat

Looking good!

Please make sure to address Mike's comments at the end of threat.yml as well.

schemas/threat.yml

CHANGELOG.next.md

schemas/threat.yml

webmat · 2019-09-05T15:20:26Z

Suppose someone's looking at a Suricata alert like this one:

        "suricata.eve.alert.category": "Attempted Information Leak",
        "suricata.eve.alert.gid": 1,
        "suricata.eve.alert.rev": 4,
        "suricata.eve.alert.signature": "ET POLICY curl User-Agent Outbound",
        "suricata.eve.alert.signature_id": 2013028,

How would you suggest the mapping is done?

I think alert.signature_id maps to threat.id and alert.signature maps to threat.name.

But should alert.category be mapped to threat.technique.name? Seems like we'd like to both want to allow people to keep their alert source's categorization, as it's good to have a 1:1 mapping with the source, when investigating what the alert means. But it would also be useful to let people map to a third party categorization framework. I'm sure there's already people who've mapped Suricata's alerts to the ATT&CK framework.

Or perhaps we should treat actual alerts as a separate entity than threats? Map the alert details to an eventual alert field set, and add the threat field set purely as a categorization scheme on top of the alert?

dainperkins · 2019-09-05T15:43:50Z

I'd put id/name in as described. I wouldn't be against adding threat.category to extend the basic threat concept (as reported by the observer), then leaving the tactic and technique as more specific places to record inferred tactic/technique information (mapped to ATT&CK, or other internal threat analyzation) At first I was thinking alert.category could map to threat.technique, but then people have to decide [do I want to map observed information? Or do I leverage an additional taxonomy?] given the low impact I think having both would be beneficial (if for no other reason that the ability to search for vendor specific threat names, or external classification terms, instead of one or the other) I'll add a threat.category, and take a look at a couple other detection types to see what else could go into the overall (risk is set, tho I have yet to figure out how to add threat to the risk.yml fields as a possible top level as threat doesn't exist in my risk branch...) /d

…

On Thu, Sep 5, 2019 at 11:20 AM Mathieu Martin ***@***.***> wrote: Suppose someone's looking at a Suricata alert like this one <https://github.com/elastic/beats/blob/v7.3.0/x-pack/filebeat/module/suricata/eve/test/eve-alerts.log-expected.json#L42-L46> : "suricata.eve.alert.category": "Attempted Information Leak", "suricata.eve.alert.gid": 1, "suricata.eve.alert.rev": 4, "suricata.eve.alert.signature": "ET POLICY curl User-Agent Outbound", "suricata.eve.alert.signature_id": 2013028, How would you suggest the mapping is done? I think alert.signature_id maps to threat.id and alert.signature maps to threat.name. But should alert.category be mapped to threat.technique.name? Seems like we'd like to both allow people to keep their alert source's categorization, as it's good to have a 1:1 mapping with the source, when investigating what the alert means. But it would also be useful to let people map to a third party categorization framework. I'm sure there's already people who've mapped Suricata's alerts to the ATT&CK framework. Or perhaps we should treat actual alerts as a separate entity than threats? Map the alert details to an eventual alert field set, and add the threat field set purely as a categorization scheme on top of the alert? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#505?email_source=notifications&email_token=AJMZFSMEPMQNZC6AEBSKVXDQIEPUPA5CNFSM4IF32FJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD57RBVA#issuecomment-528421076>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJMZFSI5IYCANJXTOAP46H3QIEPUPANCNFSM4IF32FJA> .

webmat · 2019-09-05T17:57:59Z

@dainperkins another thing that occurred to me. It might be worth adding a field that states which kind of categorization is being done. Is it MITRE, are people shoving QRadar in tactic & technique, or another vendor specific thing.

So perhaps something like threat.taxonomy, or threat.framework, in which you could put values such as "mitre_attack", "qradar", etc...

I can guarantee that in a big enough organization, people will be using more than one different taxonomy, because [reasons]. So being able to separate them from one another may be helpful.

dainperkins · 2019-09-05T18:15:59Z

excellent idea

…

On Thu, Sep 5, 2019 at 1:58 PM Mathieu Martin ***@***.***> wrote: @dainperkins <https://github.com/dainperkins> another thing that occurred to me. It might be worth adding a field that states which kind of categorization is being done. Is it MITRE, are people shoving QRadar in tactic & technique, or another vendor specific thing. So perhaps something like threat.taxonomy, or threat.framework, in which you could put values such as "mitre_attack", "qradar", etc... I can guarantee that in a big enough organization, people will be more than one different taxonomy, because [reasons]. So being able to separate them from one another may be helpful. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#505?email_source=notifications&email_token=AJMZFSLVKXFIYLQHG2DP3STQIFCDLA5CNFSM4IF32FJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6AE6DY#issuecomment-528502543>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJMZFSLJVNPVCO3LL26DECTQIFCDLANCNFSM4IF32FJA> .

dainperkins · 2019-09-05T19:09:40Z

was also thinking sig revision would be a good add (suricata has it, as does snort)

…

On Thu, Sep 5, 2019 at 1:58 PM Mathieu Martin ***@***.***> wrote: @dainperkins <https://github.com/dainperkins> another thing that occurred to me. It might be worth adding a field that states which kind of categorization is being done. Is it MITRE, are people shoving QRadar in tactic & technique, or another vendor specific thing. So perhaps something like threat.taxonomy, or threat.framework, in which you could put values such as "mitre_attack", "qradar", etc... I can guarantee that in a big enough organization, people will be more than one different taxonomy, because [reasons]. So being able to separate them from one another may be helpful. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#505?email_source=notifications&email_token=AJMZFSLVKXFIYLQHG2DP3STQIFCDLA5CNFSM4IF32FJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6AE6DY#issuecomment-528502543>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJMZFSLJVNPVCO3LL26DECTQIFCDLANCNFSM4IF32FJA> .

revised changelog.next, added threat.category, threat.framework, revised organization

make of previous updates

fixed spelling updated desc's

mbudge · 2019-09-12T07:33:27Z

An "outcome" or "action" field could store actions take by the observer or agent.

For example

Firewall blocking inbound or outbound connection
Web proxy blocking outbound http request
Anti-virus quarantining malware file
Endpoint software blocking suspicious file
IPS blocking connection

Not sure if this should go in threat or the event schema.

MikePaquette · 2019-09-12T10:27:06Z

@mbudge Yes, the actions such as those you list, are in many ways the "story" of the event, so I think this is better captured at the event level using the event.action field.

webmat · 2019-09-30T21:05:24Z

I'd like to close on this soon, but there's something that's been bugging me about the proposal.

According to the field set's definition, it sounds like it would be fine to either use this to map one's alerts to a taxonomy, or to use the source alert categorization as is (e.g. the Suricata alert categories). I think this field set should purely focus on making it possible to map alerts to the user's chosen taxonomy. Can be mitre, can be another taxonomy.

Corollary to that, I think it would be important to work on a sister field set that is specifically meant to capture the generic concepts about alerts. Before we have both, I think having only "threat.*" will be confusing.

I think the absence of the "alert" field set is what led us to discussing the addition .revision in here, for example. I think tracking a revision makes sense if you're talking about the version of an alert, but not if you're mapping alerts to a taxonomy. I mean it would make sense to have it in alert.revision, not in threat.revision.

Here's how I would envision the two. Let's take the same example as above to illustrate:

        "suricata.eve.alert.category": "Attempted Information Leak",
        "suricata.eve.alert.gid": 1,
        "suricata.eve.alert.rev": 4,
        "suricata.eve.alert.signature": "ET POLICY curl User-Agent Outbound",
        "suricata.eve.alert.signature_id": 2013028,

Given the existence of an alert field set, the mapping I would see would be:

ECS Field	Value	Source Field
message	ET POLICY curl User-Agent Outbound	suricata.eve.alert.signature
alert.id	2013028	suricata.eve.alert.signature_id
alert.category.id	1	suricata.eve.alert.gid
alert.category.name	Attempted Information Leak	suricata.eve.alert.category
alert.revision	4	suricata.eve.alert.rev

Then if someone is using Mitre, they would have mapped the known Suricata alerts to the mitre taxonomy, and the resulting event would also have these populated:

ECS Field	Value
threat.framework	mitre_att&ck
threat.tactic.id	TA0010
threat.tactic.name	exfiltration
threat.tactic.reference	https://attack.mitre.org/tactics/TA0010/
threat.technique.id	T1048
threat.technique.name	Exfiltration Over Alternative Protocol
threat.technique.reference	https://attack.mitre.org/techniques/T1048

If the user is actually using a different taxonomy, all of alert.* would remain exactly the same, but only what's under threat would be changing to that other taxonomy.

webmat · 2019-09-30T21:10:26Z

Heads up as well. Ray is working on a misp importer Filebeat module here elastic/beats#13805. There's some commonalities.

Worth giving this a look, while pondering all this.

webmat · 2019-10-01T12:27:01Z

schemas/threat.yml

+    - name: revision
+      format: string
+      level: extended
+      type: long


Just spotted this. It should be keyword. Not all revisions are purely numeric.

dainperkins · 2019-10-01T14:20:25Z

Forgot to commit before pull req 1 today...

webmat · 2019-10-01T14:22:28Z

CHANGELOG.next.md

@@ -11,6 +11,8 @@ Thanks, you're awesome :-) -->

 ### Added

+* Remnoved threat event fields #505


We simplified the PR, but this PR is still "Adding threat fields" :-)

webmat

I had not noticed all the places you had added the multi_fields. Since the threat fields are now solely focused on mapping to a taxonomy, it means the range of expected values will be much more limited. Therefore I think we should remove all of the multi-fields in here for now. We can add them back later if there's a need.

I'm also requesting changes on the field set short and full definitions.

Once these are in place, we can merge.

Fix the changelog
Remove all multi-field
Adjust fieldset's short description
Adjust fieldset's main description

Thanks @dainperkins!

schemas/threat.yml

simplified threat fields, limiting to taxonomy fields for ecs 1.2

updated changelog for revised threat fieldset for 1.2

schemas/threat.yml

spelling / changelog fixes

webmat · 2019-10-02T13:14:14Z

schemas/threat.yml

+   These fields are for users to classify alerts from all of their sources (e.g. IDS, NGFW, etc.) within a 
+   common taxonomy. The threat.technique.* are meant to capture the high level category of the threat 
+   (e.g. "exfiltration"). The threat.tactic.* fields are meant to capture which kind of approach is used by 
+   this detected threat, to accomplish the goal (e.g. "data compressed").


I asked you to adjust the fieldset definition, and I got tactic and technique reversed 😂 Sorry about that.

High level category is tactic, and technique is the actual method used. Please update it to undo my mistake:

These fields are for users to classify alerts from all of their sources (e.g. IDS, NGFW, etc.) within a
common taxonomy. The threat.tactic.* fields are meant to capture the high level category of the threat
(e.g. "exfiltration"). The threat.technique.* fields are meant to capture the method used by
this detected threat, to accomplish the goal (e.g. "data compressed").

fixed tactc/technique & replicated impact:endpoint DOS in top level example

webmat

Ah, I see you aligned the field set description with the examples used across the fields. Love it!

LGTM

webmat

LGTM!

Thanks for all the last minute adjustments 🙂

webmat · 2019-10-02T14:34:19Z

Oops, that last LGTM was on the wrong PR 🤦‍♂

MikePaquette

LGTM

dainperkins and others added 2 commits July 22, 2019 14:57

threat fields

087e7f7

added threat fields

Merge branch 'master' into threat

d90abfa

webmat reviewed Jul 23, 2019

View reviewed changes

schemas/threat.yml Outdated Show resolved Hide resolved

schemas/threat.yml Outdated Show resolved Hide resolved

schemas/threat.yml Show resolved Hide resolved

CHANGELOG.next.md Outdated Show resolved Hide resolved

schemas/threat.yml Outdated Show resolved Hide resolved

dainperkins added 2 commits August 12, 2019 21:48

Updates to pr 505

c8adde5

added text miulti fields, modified tactic/technique to be suggested as Mitre Att&ck, added possible qualitative options

PR #505 Update 2

54c4312

added multifields for fts, modified tactic and technique to suggest ATT&CK, etc.

Threat Updates

ac2ef12

Updated threat fields per recommendations

MikePaquette requested changes Aug 27, 2019

View reviewed changes

dainperkins added 2 commits September 5, 2019 08:21

Merge branch 'master' into threat

310db0a

Revised short descriptions

72ab6e9

revised short desc. to better align to tootip thoughts

webmat reviewed Sep 5, 2019

View reviewed changes

schemas/threat.yml Show resolved Hide resolved

CHANGELOG.next.md Outdated Show resolved Hide resolved

webmat reviewed Sep 5, 2019

View reviewed changes

schemas/threat.yml Show resolved Hide resolved

dainperkins added 3 commits September 6, 2019 09:20

Updated threat fields

1cc4be3

revised changelog.next, added threat.category, threat.framework, revised organization

make threats.yml updates

0e61bad

make of previous updates

updated spelling & descriptions

6260ef0

fixed spelling updated desc's

MikePaquette mentioned this pull request Sep 12, 2019

Added event.detection to identify suspicious activity or files #558

Closed

webmat mentioned this pull request Sep 30, 2019

Add Filebeat misp module elastic/beats#13805

Merged

webmat reviewed Oct 1, 2019

View reviewed changes

removed threat event description fields

342b432

webmat reviewed Oct 1, 2019

View reviewed changes

MikePaquette mentioned this pull request Oct 1, 2019

[SIEM][Detection Engine] Signal ECS Mappings elastic/kibana#47015

Closed

webmat suggested changes Oct 1, 2019

View reviewed changes

schemas/threat.yml Outdated Show resolved Hide resolved

schemas/threat.yml Show resolved Hide resolved

schemas/threat.yml Outdated Show resolved Hide resolved

schemas/threat.yml Outdated Show resolved Hide resolved

schemas/threat.yml Outdated Show resolved Hide resolved

dainperkins added 2 commits October 1, 2019 16:16

simplified threat fields

4ec0e5f

simplified threat fields, limiting to taxonomy fields for ecs 1.2

updated changelog

0c16d29

updated changelog for revised threat fieldset for 1.2

MikePaquette reviewed Oct 1, 2019

View reviewed changes

schemas/threat.yml Outdated Show resolved Hide resolved

final threat fixes

5acdde4

spelling / changelog fixes

webmat reviewed Oct 2, 2019

View reviewed changes

update descriptions

9630a72

fixed tactc/technique & replicated impact:endpoint DOS in top level example

webmat approved these changes Oct 2, 2019

View reviewed changes

MikePaquette approved these changes Oct 2, 2019

View reviewed changes

webmat merged commit 3f9d048 into elastic:master Oct 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

threat fields #505

threat fields #505

dainperkins commented Jul 22, 2019

webmat left a comment

webmat commented Jul 23, 2019 •

edited

Loading

dainperkins commented Aug 13, 2019 via email

dainperkins commented Aug 13, 2019 via email

dainperkins commented Aug 27, 2019

dainperkins commented Aug 27, 2019

MikePaquette left a comment

webmat left a comment

webmat commented Sep 5, 2019 •

edited

Loading

dainperkins commented Sep 5, 2019 via email

webmat commented Sep 5, 2019 •

edited

Loading

dainperkins commented Sep 5, 2019 via email

dainperkins commented Sep 5, 2019 via email

mbudge commented Sep 12, 2019 •

edited

Loading

MikePaquette commented Sep 12, 2019

webmat commented Sep 30, 2019 •

edited

Loading

webmat commented Sep 30, 2019

webmat Oct 1, 2019

dainperkins commented Oct 1, 2019

webmat Oct 1, 2019

webmat left a comment •

edited by dainperkins

Loading

webmat Oct 2, 2019

webmat left a comment

webmat left a comment

webmat commented Oct 2, 2019

MikePaquette left a comment

		@@ -11,6 +11,8 @@ Thanks, you're awesome :-) -->

		### Added

		* Remnoved threat event fields #505

threat fields #505

threat fields #505

Conversation

dainperkins commented Jul 22, 2019

webmat left a comment

Choose a reason for hiding this comment

webmat commented Jul 23, 2019 • edited Loading

dainperkins commented Aug 13, 2019 via email

dainperkins commented Aug 13, 2019 via email

dainperkins commented Aug 27, 2019

dainperkins commented Aug 27, 2019

MikePaquette left a comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

webmat commented Sep 5, 2019 • edited Loading

dainperkins commented Sep 5, 2019 via email

webmat commented Sep 5, 2019 • edited Loading

dainperkins commented Sep 5, 2019 via email

dainperkins commented Sep 5, 2019 via email

mbudge commented Sep 12, 2019 • edited Loading

MikePaquette commented Sep 12, 2019

webmat commented Sep 30, 2019 • edited Loading

webmat commented Sep 30, 2019

webmat Oct 1, 2019

Choose a reason for hiding this comment

dainperkins commented Oct 1, 2019

webmat Oct 1, 2019

Choose a reason for hiding this comment

webmat left a comment • edited by dainperkins Loading

Choose a reason for hiding this comment

webmat Oct 2, 2019

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

webmat commented Oct 2, 2019

MikePaquette left a comment

Choose a reason for hiding this comment

webmat commented Jul 23, 2019 •

edited

Loading

webmat commented Sep 5, 2019 •

edited

Loading

webmat commented Sep 5, 2019 •

edited

Loading

mbudge commented Sep 12, 2019 •

edited

Loading

webmat commented Sep 30, 2019 •

edited

Loading

webmat left a comment •

edited by dainperkins

Loading