[Bug] When creating cases from alerts via API, the same case number gets assigned to multiple distinct cases

[martinr103]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Debian
OS version (client)	10.3
TheHive version / git hash	4.1.3
Package Type	Binary / in Docker
Browser type & version	If applicable

Problem Description

I observed the seriously concerning behavior, that the same case number is being assigned to multiple distinct cases.

In our system we have a script providing alerts/cases to Thehive via API.
This script runs periodically, and it can well happen that it needs to open 3,4,5 new alerts in Thehive in one "batch". Shortly one after another.
Immediately after alert creation, we also promote the newly created alerts to Cases. (because these alerts are already 'confirmed' by the source system - so they are qualified to become Cases in Thehive right from the start)

Now, I have observed that for 4 distinct alerts, the createCase API returned 4 distinct case ID's but all 4 got the same case NUMBER assigned !

Like this:

| sourceRef = 42091 | Alert-ID-in-Thehive = ~334688400 | Case-ID-in-Thehive = ~125423848 | Case-Number-in-Thehive = 21795 |
| sourceRef = 42092 | Alert-ID-in-Thehive = ~127615176 | Case-ID-in-Thehive = ~248148136 | Case-Number-in-Thehive = 21795 |
| sourceRef = 42093 | Alert-ID-in-Thehive = ~334696592 | Case-ID-in-Thehive = ~84148320 | Case-Number-in-Thehive = 21795 |
| sourceRef = 42094 | Alert-ID-in-Thehive = ~127627464 | Case-ID-in-Thehive = ~334729360 | Case-Number-in-Thehive = 21795 |

This is really scary.

Additional information:
Later on I have manually merged two of the 4 cases (sourceRef 42093 + 42094) into a brand new one, which had the effect, that the cases
got deleted (which is the new behavior by design as I understand). After the merge you are not able to find any case with the number 21795.
However you can still access the two not-merged cases by their ID, i.e. ~125423848 + ~248148136.

[nadouani]

Hello @martinr103 it's not that scary except if you rely on the case number, which is NOT the case's identifier. You cas see that 4 cases have been created out there.

TheHive has integrity check job that fix the case numbers if there are duplicate numbers. The other way would be to introduce database locks to ensure uniqueness and this leads to very poor performance in distributed databases.

[martinr103]

Oh, this is bad news! :-(
For programmatic access we rely on the identifiers, yes.
But we also need the human readable case numbers! Sometimes analysts need to search in TH GUI for a specific case by its number.
(After all you do have the Case Search box available in the top toolbar)

Question: How often is the integrity check running, that you mentioned ?

I thought Cassandra was at least as performant as Elasticsearch, if not better.
With TH3+Elastic we can at peaks open 100 new alerts/cases within just a few seconds, without a mixup in case numbers - that is never an issue.

Hmmm.... quite disappointed if that should be not possible with Cassandra anymore.

[nadouani]

It's all about data model. Elasticsearch is a set of json documents. a case in ES is a json object. In a graph system like Janusgraph (that uses Cassandra as storage) a case is a set of vertices linked by edges. It's completely incomparable.

If you want multitenancy, data integrity, constraints, transactions... then you cannot just rely on ES for our case.

Question: How often is the integrity check running, that you mentioned ?

10 min I think

[martinr103]

I kind of... understand. But it's not satisfying...

And don't get me wrong, you guys have created a really really great tool.

But this trade-off now, feels like a step... backwards.

Is there really no_way?? To always get a unique case number immediately as result from the createCase API call ?
(I asked about the interval of integrity check - but in reality it does not help, when the case-number is changed "afterwards")

Maybe these "database locks" (that you mentioned before) can be enabled optionally, by users choice ie. via config ?

Honestly, if there is absolutely no way to make it work in TH4 the way it was working in TH3, then that would be a no-go criteria for the upgrade 3-to-4. :-(

[crackytsi]

Unfortunately I have to agree with @martinr103 as once the case number is assigned and communicated to other systems a change of the number has big impact...

[crackytsi]

@nadouani / @To-om : What about storing last assigned case-number somewhere else (e.g. in akka cluster-state + file-system for starting)?

I guess that this could be a small critical-section assigning the next free number, but changing the "human-readable" number of a case is really less user intuitive and has several dependencies to connected processes.

[nadouani]

@To-om this is a duplicate of #2061 . Just putting the info here to make sure to close both issues once solved.