[Bug] Observable already exists

[V1D1AN]

Request Type

Bug

Work Environment

Elastalert 0.2.4
TheHive 4.1.3-1 on docker with Cassandra 3.11

Problem Description

[error] o.t.s.u.Retry [000008f2|1e3928fc] uncaught error, not retrying
org.thp.scalligraph.CreateError: Observable already exists
at org.thp.thehive.services.ObservableSrv.create(ObservableSrv.scala:95)
at org.thp.thehive.services.AlertSrv.createObservable(AlertSrv.scala:125)
at org.thp.thehive.controllers.v0.AlertCtrl.$anonfun$createObservable$6(AlertCtrl.scala:341)
at org.thp.scalligraph.package$RichSeq.$anonfun$toTry$3(package.scala:17)
at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
at scala.collection.immutable.List.foldLeft(List.scala:91)
at org.thp.scalligraph.package$RichSeq.toTry(package.scala:16)
at org.thp.thehive.controllers.v0.AlertCtrl.$anonfun$createObservable$1(AlertCtrl.scala:341)
at scala.util.Success.flatMap(Try.scala:251)
[error] o.t.s.m.Database [000008f2|1e3928fc] Exception raised, rollback (Observable already exists)
[warn] o.t.s.ErrorHandler [000008f2|1e3928fc] POST /thehive4/api/alert returned 400
[info] o.t.s.AccessLogFilter [000008f2|] 172.21.0.2 POST /thehive4/api/alert took 143ms and returned 400 60 bytes

Steps to Reproduce

This is my elastalert rule that triggers the error

alert:

• debug
  description: Suricata Level Alert
  filter:
• query:
  query_string:
  query: (event_type:"alert" AND alert.severity:"1")
  index: suricata-events-*
  category:
• Suricata
  name: Suricata Level Alert
  priority: 3
  realert:
  minutes: 0
  title: Suricata Level Alert
  type: any

aggregation:
minutes: 2
aggregation_key: alert.signature_id
summary_table_fiels:

• alert.signature_id
• src_ip
• dest_ip

alert_subject_args:

• alert.signature
  alert_subject: "{0}"
  alert_text_type: alert_text_only
  alert_text: "Detection Suricata: {0} \n\n
  SI: {7} \n\n
  Source IP: {1}:{5} \n\n
  Destination IP: {2}:{6} \n\n
  Suricata Rule: {4} \n\n
  Link Kibana: https://XXX.XXX.XXX.XXX/kibana/app/kibana#/discover?_g=(filters:!())&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'*:suricata-events-*',key:_id,negate:!f,params:(query:'{3}'),type:phrase,value:'{3}'),query:(match:(_id:(query:'{3}',type:phrase))))),index:'*:suricata-events-*',interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))) \n\n"
  alert_text_args:
• alert.signature
• src_ip
• dest_ip
• _id
• alert.rule
• src_port
• dest_port
• SI

alert: hivealerter

hive_alert_config_type: classic

hive_alert_config:
type: 'Suricata'
source: '{match[source]}'
severity: 3
tags: ['Suricata','{match[alert][category]}']
tlp: 2
status: 'New'
follow: True

hive_observable_data_mapping:

• hash: "{match[tls][ja3][hash]}"
• hash: "{match[fileinfo][md5]}"
• domain: "{match[dns][query][rrname]}"
• ip: "{match[src_ip]}"
• ip: "{match[dest_ip]}"
• filename: "{match[fileinfo][filename]}"

Complementary information

This is work with TheHive 3 but not with TheHive 4 .. Why ??

Thanks

[joseluratm]

I also got the same error.
To fix it check that the artifacts you are sending to thehive are not repeated. To do this you have to check that the same artifact with 'dataType' and 'data' does not exist in the same alert.

In case you are sending files. Even if the 'data' field has a different value, you should check that none of the files have the same hash.

I hope it helps.

[ch0wm3in]

This was not a problem pre 4.1.0 it worked in 4.05

Thehive would ignore one of the duplicate observables and continue silently.

[V1D1AN]

This was not a problem pre 4.1.0 it worked in 4.05

Thehive would ignore one of the duplicate observables and continue silently.

Yes it works on the 4.05 ...