[3.4.0-RC2] Observables are lost during Alarm to Case import

[presianbg]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	CentOS 7.6.1810 (Core)
OS version (client)	Solus
TheHive version / git hash	3.4.0-RC2
Package Type	RPM
Browser type & version	Vivaldi - latest

Problem Description

I'm using a logstash pipeline to import alerts via Thehive's API.
The alarms are created successfully with 10 observables:

but when importing the alarm into an Empty case only 8 out of 10 observables are kept:

[presianbg]

Also not sure if related, but I found another issue with observables (maybe worth to open separate Github issue for it), when the "data" value is null .

Creating alarm via logstash json like this:

and this is how final json (for different alarm than on the screen shotabove), as seen in the http body packet where "user-agent" is with NULL value:

{"data":null,"message":"UA","dataType":"user-agent"}

{
  "artifacts": [
    {      
      "data": "172.27.173.151",
      "message": "Destination IP",
      "dataType": "ip"
    },
    {
      "data": "80",
      "message": "Dest Port",
      "dataType": "dest_port"
    },
    {
      "data": "188.42.129.148",
      "message": "Source IP",
      "dataType": "ip"
    },
    {
      "data": "32122",
      "message": "Src Port",
      "dataType": "src_port"
    },
    {
      "data": "TCP",
      "message": "L4 proto",
      "dataType": "proto"
    },
    {
      "data": "http",
      "message": "L7 proto",
      "dataType": "app_proto"
    },
    {
      "data": "POST / HTTP/1.1\r\nContent-Type:application/x-www-form-urlencoded\r\nHost: rl.ammyy.com\r\nContent-Length: 243\r\nCache-Control:,
      "message": "packet payload",
      "dataType": "payload"
    },
    {
      "data": "ammyy.com",
      "message": "FQDN",
      "dataType": "domain"
    },
    {
      "data": "/",
      "message": "URL",
      "dataType": "uri_path"
    },
    {
      "data": null,
      "message": "UA",
      "dataType": "user-agent"
    }
  ],
  "date": 1563793309021,
  "title": "ETPRO POLICY RemoteAdmin Win32.Ammyy.z Checkin",
  "sourceRef": "1903828044",
  "description": "A Network Trojan was Detected %{metadata} %{alarm_message}",
  "type": "STAMUS IDS",
  "source": "XXX01",
  "severity": 1
}

Then import this alarm into case and when viewing the "Observables" tab all the observeble itmes will be gone (not only those with null value).

[ITServ-DE]

I'm running the same config, and all works well.

As far as I know, you can only import observables if the type is defined in Admin -> Observables.

You try to import data of type app_proto. Is this defined?

Check out the application log (typically /var/log/thehive/application.log) during import. You will probably see an error message like this (look at the first line):

2019-07-26 08:25:58,915 [WARN] from services.AlertSrv in application-akka.actor.default-dispatcher-16 - Create artifact error
org.elastic4play.AttributeCheckingError: [Invalid format for case_artifact.dataType: JsonInputValue("user"), expected enumeration]
		 at org.elastic4play.services.FieldsSrv.$anonfun$parse$7(FieldsSrv.scala:33)
		 at org.scalactic.Bad.transform(Or.scala:1386)
		 at org.elastic4play.services.FieldsSrv.parse(FieldsSrv.scala:33)
		 at org.elastic4play.services.CreateSrv.create(CreateSrv.scala:105)
		 at org.elastic4play.services.CreateSrv.$anonfun$apply$7(CreateSrv.scala:95)
		 at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
		 at scala.collection.immutable.List.foreach(List.scala:389)
		 at scala.collection.TraversableLike.map(TraversableLike.scala:234)
		 at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
		 at scala.collection.immutable.List.map(List.scala:295)
		 at org.elastic4play.services.CreateSrv.apply(CreateSrv.scala:93)
		 at services.ArtifactSrv.create(ArtifactSrv.scala:86)
		 at services.AlertSrv.importArtifacts(AlertSrv.scala:312)
		 at services.AlertSrv.$anonfun$createCase$4(AlertSrv.scala:224)
		 at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:303)
		 at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
		 at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
		 at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
		 at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
		 at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
		 at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
		 at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
		 at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
		 at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
		 at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
		 at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
		 at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
		 at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

You must create the observable types prior to importing.

[saadkadhi]

Indeed, it seems you have several datatypes that are not included in the stock install of TheHive. You'll have to create them as suggested by @github-pba.

[presianbg]

@saadkadhi and @github-pba you are awesome!
It turned out that I have those custom datatypes defined by me, but they were with typos :(

Shall I open a new issue for the NULL value problem described in my comment or it's a know issue ?

[ITServ-DE]

Thank you for the complement.

I think it would be the best to open a new issue on the other thing. And to be honest: I didn't understand it. It would be helpful to provide sample code that generates the problem.

Please, don't forget to close this issue (#1079).

[ITServ-DE]

Oh, now I understand your problem.

We can handle this here, I think.

Would you please explain what sense it makes to create observables of some type but without any value?

(Still I think there is really a problem, I saw a weird thing).

[ITServ-DE]

Please checkout Issue 1080 I have just opened. I think this exactly describes what you encoutered.

[presianbg]

Oh, now I understand your problem.

We can handle this here, I think.

Would you please explain what sense it makes to create observables of some type but without any value?

(Still I think there is really a problem, I saw a weird thing).

So my alerts are coming from suricata IDS and the jsons are not consistent.
For convenience I'm using logstash http output to ship them to TheHive's API.
My parsing is not dynamic - I'm not checking if such key/value exists in the original suricata alarm, instead the artifacts are constructed like this:

  ruby {
        code => '
          event.set("artifacts", [{"dataType" => "ip", "data" => event.get("src_ip"), "message" => "Source IP"},
                                  {"dataType" => "ip", "data" => event.get("dest_ip"), "message" => "Destination IP"},
                                  {"dataType" => "dest_port", "data" => event.get("dest_port"), "message" => "Dest Port"},
                                  {"dataType" => "src_port", "data" => event.get("src_port"), "message" => "Src Port"},
                                  {"dataType" => "proto", "data" => event.get("proto"), "message" => "L4 proto"},
                                  {"dataType" => "app_proto", "data" => event.get("app_proto"), "message" => "L7 proto"},
                                  {"dataType" => "payload", "data" => event.get("payload_printable"), "message" => "packet payload"},
                                  {"dataType" => "domain", "data" => event.get("[hostname_info][domain]"), "message" => "FQDN"},
                                  {"dataType" => "url_path", "data" => event.get("[http][url]"), "message" => "url path"},
                                  {"dataType" => "url", "data" => event.get("[http][hostname]") + event.get("[http][url]"), "message" => "uri + url"},
                                  {"dataType" => "other", "data" => event.get("metadata"), "message" => "metadata"},
                                  {"dataType" => "other", "data" => event.get("alarm_message"), "message" => "alarm_message"},
                                  {"dataType" => "signature_id", "data" => event.get("[alert][signature_id]"), "message" => "Signature ID"},
                                  {"dataType" => "user-agent", "data" => event.get("[http][http_user_agent]"), "message" => "UA"}])
        '
      }

This is not the optimal solution, but it saves a lot of pain which dynamic constructor will require.

Where key is missing, let's say:

{"dataType" => "payload", "data" => event.get("payload_printable"), "message" => "packet payload"},

logstash is putting "nil" value for key "payload", fair enough.
Alarms are created successfully and a observable with nil value is displayed as empty in "Alerts Preview", but when this alert is imported into case, all observables are hidden despite that the counter shows X observables for this case.

[presianbg]

Please checkout Issue 1080 I have just opened. I think this exactly describes what you encoutered.

Exactly! Thanks for testing this for me. I'm still a newbie regarding this project.