[Bug] Sometimes able to add identical url observables to the same case

[Passimist]

Work Environment

Question	Answer
OS version (server)	Ubuntu 18.04.5 LTS
OS version (client)	Ubuntu 18.04.5 LTS
TheHive version	4.0.0-1
Scalligraph	0.1.0-SNAPSHOT
Play	2.8.2
CORTEX	3.0.1-1
Package Type	DEB

Problem Description

Sometimes I am able to add an url observable that already exists in a case the the same case again. Like in this picture:

And sometimes I get an error like this:

I can always upload the same file as observable without any errors (even with the same filename)

A while after uploading identical URLs the TheHive/application.log shows lines like this:

2020-11-11 17:11:59,818 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in application-akka.actor.default-dispatcher-17 [|59700ec0] Found duplicate entities:
Data(https://www.youtube.com/)
Data(https://www.youtube.com/)

This are the logs around the time of creating a duplicate observable (I dont see anything interesting in there):

2020-11-11 17:24:40,917 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-19 [0000002d|] 127.0.0.1 POST /api/v1/query?name=cases took 17ms and returned 200
2020-11-11 17:24:41,055 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-19 [0000002f|] 127.0.0.1 GET /api/flow?count=10&rootId=any took 131ms and returned 200 1758 bytes
2020-11-11 17:24:41,211 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-19 [0000002c|] 127.0.0.1 POST /api/v1/query?name=case-count-stats took 314ms and returned 200 2 bytes
2020-11-11 17:24:41,243 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [0000002e|] 127.0.0.1 POST /api/v1/query?name=cases.count took 334ms and returned 200 2 bytes
2020-11-11 17:24:41,495 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [0000002b|] 127.0.0.1 POST /api/v1/query?name=case-status-stats took 597ms and returned 200 45 bytes
2020-11-11 17:24:46,061 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-17 [00000030|] 127.0.0.1 POST /api/v1/query?name=get-case-41480216 took 8ms and returned 200
2020-11-11 17:24:46,176 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-13 [00000031|] 127.0.0.1 POST /api/v1/query?name=observable-stats-41480216 took 32ms and returned 200 2 bytes
2020-11-11 17:24:46,182 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [00000033|] 127.0.0.1 POST /api/v1/query?name=alert-stats-41480216 took 28ms and returned 200 1 bytes
2020-11-11 17:24:46,194 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [00000032|] 127.0.0.1 POST /api/v1/query?name=task-stats-41480216 took 49ms and returned 200 1 bytes
2020-11-11 17:24:46,201 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-17 [00000034|] 127.0.0.1 POST /api/v1/query?name=case-attachments.count took 47ms and returned 200 1 bytes
2020-11-11 17:24:46,204 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [00000035|] 127.0.0.1 POST /api/v1/query?name=case-attachments took 49ms and returned 200
2020-11-11 17:24:46,222 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-17 [00000036|] 127.0.0.1 POST /api/v1/query?name=case-actions took 40ms and returned 200
2020-11-11 17:24:46,290 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [00000038|] 127.0.0.1 GET /fonts/SourceSansPro-Semibold.otf took 5ms and returned 200 232680 bytes
2020-11-11 17:24:47,220 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [00000039|] 127.0.0.1 GET /api/flow?count=10&rootId=41480216 took 756ms and returned 200 16477 bytes
2020-11-11 17:24:47,827 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-19 [0000003a|] 127.0.0.1 POST /api/v1/query?name=observables took 15ms and returned 200
2020-11-11 17:24:47,843 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [0000003b|] 127.0.0.1 POST /api/v1/query?name=observables.count took 23ms and returned 200 2 bytes
2020-11-11 17:24:48,472 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [00000027|] 127.0.0.1 GET /api/stream/JsM19EGTbCtb66tVSl8H took 60020ms and returned 200 2 bytes
2020-11-11 17:24:49,109 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [00000037|] 127.0.0.1 GET /api/case/41480216/links took 2905ms and returned 200 112888 bytes
2020-11-11 17:24:56,079 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-21 [0000003d|] 127.0.0.1 GET /api/observable/type?range=all took 56ms and returned 200
2020-11-11 17:25:11,440 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [0000003e|] 127.0.0.1 POST /api/case/41480216/artifact took 415ms and returned 201 309 bytes
2020-11-11 17:25:11,663 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-19 [0000003c|] 127.0.0.1 GET /api/stream/JsM19EGTbCtb66tVSl8H took 22933ms and returned 200 1821 bytes
2020-11-11 17:25:11,699 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [00000040|] 127.0.0.1 POST /api/v1/query?name=observables took 23ms and returned 200
2020-11-11 17:25:11,705 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-19 [0000003f|] 127.0.0.1 POST /api/v1/query?name=observable-stats-41480216 took 33ms and returned 200 2 bytes
2020-11-11 17:25:11,714 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-17 [00000042|] 127.0.0.1 POST /api/v1/query?name=observables.count took 32ms and returned 200 2 bytes
2020-11-11 17:25:29,278 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [00000043|] 127.0.0.1 GET /api/status took 1ms and returned 200 381 bytes
2020-11-11 17:26:06,249 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-21 [00000044|] 127.0.0.1 GET /index.html took 1ms and returned 304 0 bytes
2020-11-11 17:26:06,270 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-21 [00000046|] 127.0.0.1 GET /scripts/scripts.6cde7d53.js took 1ms and returned 304 0 bytes
2020-11-11 17:26:06,271 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [00000045|] 127.0.0.1 GET /scripts/vendor.e8efd510.js took 2ms and returned 304 0 bytes
2020-11-11 17:26:06,469 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-17 [00000047|] 127.0.0.1 GET /api/status took 2ms and returned 200 381 bytes
2020-11-11 17:26:06,485 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-17 [00000049|] 127.0.0.1 POST /api/v1/query?name=get-case-41480216 took 12ms and returned 200
2020-11-11 17:26:06,492 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [0000004a|] 127.0.0.1 GET /api/config/organisation/ui.hideEmptyCaseButton took 12ms and returned 200 68 bytes
2020-11-11 17:26:06,499 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-21 [00000048|] 127.0.0.1 GET /api/v1/user/current took 30ms and returned 200 425 bytes
2020-11-11 17:26:06,552 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-21 [0000004b|] 127.0.0.1 POST /api/v1/query took 17ms and returned 200
2020-11-11 17:26:06,575 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-19 [0000004c|] 127.0.0.1 POST /api/stream took 14ms and returned 200 20 bytes
2020-11-11 17:26:06,595 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-18 [0000004e|] 127.0.0.1 POST /api/v1/query?name=my-tasks.stats took 20ms and returned 200 1 bytes
2020-11-11 17:26:06,604 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-21 [0000004f|] 127.0.0.1 POST /api/v1/query?name=waiting-tasks.stats took 16ms and returned 200 1 bytes
2020-11-11 17:26:06,604 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-18 [00000052|] 127.0.0.1 GET /images/cortex-logo.svg took 3ms and returned 304 0 bytes
2020-11-11 17:26:06,608 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-19 [00000050|] 127.0.0.1 POST /api/v1/query?name=unread-alert-count took 19ms and returned 200 1 bytes
2020-11-11 17:26:06,612 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-13 [00000051|] 127.0.0.1 GET /api/customField took 23ms and returned 200 661 bytes
2020-11-11 17:26:06,613 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [0000004d|] 127.0.0.1 POST /api/v0/query took 52ms and returned 200
2020-11-11 17:26:06,614 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-22 [00000053|] 127.0.0.1 GET /images/misp-logo.svg took 3ms and returned 304 0 bytes
2020-11-11 17:26:06,617 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-5 [00000054|] 127.0.0.1 GET /images/logo.white.svg took 4ms and returned 304 0 bytes
2020-11-11 17:26:06,737 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-21 [00000058|] 127.0.0.1 GET /api/v1/describe/_all took 6ms and returned 200 7661 bytes
2020-11-11 17:26:06,755 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [00000057|] 127.0.0.1 POST /api/v1/query?name=alert-stats-41480216 took 24ms and returned 200 1 bytes
2020-11-11 17:26:06,756 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-5 [00000055|] 127.0.0.1 POST /api/v1/query?name=task-stats-41480216 took 29ms and returned 200 1 bytes
2020-11-11 17:26:06,759 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-19 [00000056|] 127.0.0.1 POST /api/v1/query?name=observable-stats-41480216 took 24ms and returned 200 2 bytes
2020-11-11 17:26:07,066 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-5 [0000005a|] 127.0.0.1 GET /api/flow?count=10&rootId=41480216 took 231ms and returned 200 17446 bytes
2020-11-11 17:26:07,161 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-13 [0000005c|] 127.0.0.1 POST /api/v1/query?name=observables took 30ms and returned 200
2020-11-11 17:26:07,181 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [0000005e|] 127.0.0.1 POST /api/v1/query?name=observables.count took 42ms and returned 200 2 bytes
2020-11-11 17:26:07,233 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-20 [0000005d|] 127.0.0.1 GET /api/connector/cortex/analyzer?range=all took 96ms and returned 200 4007 bytes
2020-11-11 17:26:08,913 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-18 [00000059|] 127.0.0.1 GET /api/case/41480216/links took 2179ms and returned 200 112888 bytes
2020-11-11 17:26:11,287 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in application-akka.actor.default-dispatcher-22 [|7e6fd538] Found duplicate entities:
Data(https://testsafebrowsing.appspot.com/s/phishing.html)
Data(https://testsafebrowsing.appspot.com/s/phishing.html)
2020-11-11 17:26:11,701 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-13 [00000041|] 127.0.0.1 GET /api/stream/JsM19EGTbCtb66tVSl8H took 60019ms and returned 200 2 bytes

Steps to Reproduce

1. Create new Case
2. Create URL observable
3. Waiting a few minutes seems to increase the chance of this to occurr
4. Create the same URL observable in the same case again
5. wonder why it is added / is not added

Other Information

I ran TheHive 3 on this machine before TheHive4
I use thehive4py a lot but this issue occurrs with the webinterface aswell

[nadouani]

Hello @Passimist we found this one and it's fixed in 4.0.1 at least for non file observables.

I'll let @To-om confirm if there is a fix for files in 4.0.1

[To-om]

Duplicate file observable creation has been fixed in #1643.
Regardeing other observable types I can't reproduce it with 4.0.1. Note that it is still possible to have several times the same observable if the case is shared to several organisations (orgA, orgB, orgC): orgA and orgB can create the same observable without share to each other but both share it to orgC. OrgC will see 2 same observables.

[Passimist]

Thank you guys for the quick response! Is 4.0.1 already available somewhere?

[Passimist]

Hello @To-om I can still (or again?I did not experience this in 4.0.1 but I did not test a lot) add identical observables to the same case in TheHive 4.0.2.
Scalligraph 0.1.0-SNAPSHOT
TheHive 4.0.2-1
Play 2.8.5
CORTEX LOCAL CORTEX - 3.0.1-1 (OK)

This also happens to URLs that dont contain an IP but this is an example that I can publish here.