[Bug] Missing Alerts

[nicpenning]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu
OS version (client)	Windows 10
Virtualized Env.	True
Dedicated RAM	16 GB
vCPU	8
TheHive version / git hash	4.1.10-1
Package Type	DEB
Database	Cassandra
Index type	Elasticsearch
Attachments storage	Local
Browser type & version	Chrome 64-Bit 92.0.4515.159

Problem Description

We narrowed in on a specific set of data and this is what we are seeing:

Alerts that have been created as a case and have been merged into a case do not exist. However, the Cases that were created from the alerts do exist.

The problem is that if we created 1 case from 1 alert and 300 alerts later come and are merged into the 1 case then those 300 alerts are not accounted for and neither is the initial alert. However the case exists.

One example of a week worth of data we have 1 alert and 259 cases.

I will make one note that the version of TheHive at the time of the missing data was before 4.1.10 so we will keep an eye out to see if this is something that can be replicated.

Steps to Reproduce

1. Use TheHive to receive alerts
2. Create cases from alerts or merge alerts into existing cases
3. Search for case count and alert count to see if they align

Possible Solutions

Not sure on solutions here.

Complementary information

Here I am narrowing on 1 week of alerts:
Note: there is only 1 alert.

Here is that same time frame for cases:
Note: there are 259 cases.

Most of these cases started as an alert.

[KaanSK]

I also confirm observing the same issue in 4.1.7 . Randomly, the alerts that are merged into cases are somehow lost/removed. When this happens, cases remains and in description part we can see merged alerts. Yet, the alerts do not exists. As this was observed in our production, we currently consider this a high critical issue.

All in Openshift
TheHive - 4.1.7, single instance (no downtime observed)
Cassandra - 3 replications (no downtime observed)
Elasticsearch - Used for indexing. (No downtime observed)

[nicpenning]

What does Openshift do for you?

We use Elasticsearch as the backend for our indexing and are currently on 4.1.10 and haven't observed the issue in this version but did see it in 4.1.9 when cases / alerts were created.

[KaanSK]

What does Openshift do for you?

We use Elasticsearch as the backend for our indexing and are currently on 4.1.10 and haven't observed the issue in this version but did see it in 4.1.9 when cases / alerts were created.

Hello, we prepared custom dockerfiles based on deb installation ourselves to make it more production-ready. Utilizing Openshift to build, deploy and serve.

My first reply has been edited to include elasticsearch as indexing.

[nicpenning]

Great, thanks for that clarification!

[nicpenning]

We tried to do a Drop and Rebuild on the index, but this did not bring back any alerts so it had no effect.

This is increasingly becoming a more urgent issue for us as we cannot demonstrate the metrics on Alerts received into TheHive.

[nicpenning]

I would like to see what the best path forward is to retrieve or alerts from TheHive 3. Is it possible to migrate the alerts even after TheHive 4 has been in production for over 6 months?

My thought is migrating all the alerts and only alerts from the beginning up until the oldest alert we have in TheHive 4. That way we can close the gap on hoe many alerts we have seen since day 1 of TheHive3 and include TheHive4.

[KaanSK]

Just wanted to share my experience. Due to performance issues, I needed to actually purge the data once in 2-3 months (20k+ alerts) and start from scratch. So I dont know if migrating your alerts would actually be beneficial for you due to the fact that you could be impacted with this issue as well. The system I prepared is all on kubernetes, scaled Cassandra, Elastic instances and 32 gb ram just on hive. It just does not perform at all.

I would suggest you to plan a data (alert, case) retention and time to live while working on this. As per GDPR, if you are not a sole person working on this, you may have sensitive data and you need to remove (or at least be accountable for it) anyways. Additionally, majority of observables have time to live. Keeping those hashes and IPs does not make sense for majority of those after 3-4 weeks.

I would have been really happier if I could provide you a way to fix your exact problem but unfortunately I can not. I'm done with TheHive and its problems.

[nicpenning]

I hear your frustrations @KaanSK. We think we can handle the load and the audit requirements are useful, as for a recent example we had a request to retrieve information from a case from over 6 months ago and I am glad I had the case information available to me.

Search speed seems to be continually improving with every new release. I have seen this run on a large 5 node Elasticsearch cluster and not using any containerization with a single instance of TheHive/Cortex with 16GB and 8CPUs. The more that TheHive team uses Elasticsearch the better the performance so I am optimistic for the future as I see search improvements ahead.

I do like the idea of having policy around retention for audit, alerts, case, task log, observables, etc.

I would want the high level stats to store number or alerts/cases/etc but then can safely purge after I have a running total. It is good information to see how much we are growing in the platform and the demand from our analysts.

Our idea is to migrate the alerts and then link them using Cortex observable data when possible. If in the end, I can get a count of alerts per day and what the types/sources are, that is all I really want to see so I can move on from this.

Thank you for your input, I greatly appreciate it!

[nicpenning]

Closing as TH4 is no longer supported.