Auto create domain observable from URL observable

[bullerdude]

Request Type

Feature Request

Work Environment

The Hive v2.11

Problem Description

Entering a URL observable and the domain contained within the URL as a second observable is currently a manually process.

Possible Solutions

It would be good to automate the creation of a domain observable when entering a URL observable as it will save the analyst from having to create the domain observable manually.

As auto-creating a domain observable may not alway be the desired outcome, it would make sense to implement this as an 'tick' option on the create observable page.

Example:

• Analyst creates a URL observable for www.theage.com.au/news/story/1/may/image.jpg, ticking the auto-create domain option.
• A second observable for www.theage.com.au is auto-created.

Complementary information

This would be a very useful feature as many analysers run against domains rather then URL's, but often it is a URL that is the subject of the incident analysis.

[saadkadhi]

Thanks @bullerdude for this request. In fact, we had it in an older version of TheHive but we had a heated internal debate as to the nature of the derived domain observable (the URL was observed but not the domain). Admittedly, this is the kind of 'expert' debate that is rather worthless.

We do see value in your proposal and we will look into implementing it.

[megan201296]

@saadkadhi in #46 we discussed having an analyzer for extracting the domain and that way the user has the option of doing the extraction if they want. If they don't want to, they don't have to.

[bullerdude]

@megan201296, I don't think having an analyser for extracting the domains is the focus of this feature request as that requires integration with Cortex for something that can be done as part of the observable creation the in The Hive.

Understand that some people may not want to auto-create domain observables which i why i suggested it should be an 'tick' box on the create observable model, so that users have the option.

[nadouani]

Totally agree with you @bullerdude

We just need to rewrite the observable creation user interface to allow this type of extractions.

I remember we thought about "linking" observables too (the url could be linked to the extracted domain), but didn't decide about it.

[megan201296]

Oh yeah. I remember now that the original feature request I created was before the cortex existed. @nadouani you can go ahead and close out my old request since it can be tracked here now and is more recent.

[bullerdude]

@nadouani, thanks :)

Just on the "linking" observables idea - creating relationships was something my team was discussing today. Our thinking is to support simple relationships of "is related to" and "is comprised of" by adding two fields to the observable details page.

Ideally these fields would operate like the tags field where you add multiple related IOCs (tags), and the existing IOC's in the case would automatically show as available tags.

[nadouani]

The problem of this type of relationships is that, using a document oriented storage (Elasticsearch), things can quickly lead to a disaster :)

[bullerdude]

Yes this is true; you would likely need an additional index to hold the relationship mappings. Though given the mapping data would be fairly simple (e.g. Sources observable ID, Target observable ID, Relationship Type) elastic should cope with the load. We are happy to work on the code to implement simple relationships, but also happy to take guidance on how best to achieve with the current elastic architecture.

…

_____________________________ From: Nabil Adouani <[email protected]<mailto:[email protected]>> Sent: Friday, May 26, 2017 12:57 am Subject: Re: [CERT-BDF/TheHive] Auto create domain observable from URL observable (#209) To: CERT-BDF/TheHive <[email protected]<mailto:[email protected]>> Cc: Mention <[email protected]<mailto:[email protected]>>, Matthew Rankin <[email protected]<mailto:[email protected]>> The problem of this type of relationships is that, using a document oriented storage (Elasticsearch), things can quickly lead to a disaster :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#209 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFKrD6YpRlO_cS_MW6atdDaQ7-arIShEks5r9ZajgaJpZM4Nd5Lx>.

[nadouani]

This issue has been added to 3.1.0 by error.

The ability to import observables from report analysis has been implemented via #246