UI/UX slow due to performance issue

[mamoedo]

[Bug] UI/UX slow due to performance issue

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu
TheHive version	4.0.0-RC3

700 cases - 12 observables per case (approx)

Problem Description

The time to perform almost every action in the TheHive UI is too high. When clicking elements on the web interface, TheHive stacks more tasks than it can do. TheHive stops responding after the accumulated tasks are too much. Logs show that the API requests are slow.

Also, what's the meaning of the spinning circle on the top right of the screen? It keeps growing

Steps to Reproduce

Open the logs and measure the times for each action in the following order:

1. Create a case
2. Wait for number of observables is shown (it will be 0, as it was just created)
3. Click on "create observable"
4. Click on data type to select IP [6 seconds]
5. Create observable [9 seconds]
6. Wait for observable list to be updated (it will be 1, as it was just created) [168 seconds]
7. Go back to case list [75 seconds]

Possible Solutions

Improving database query performance

Complementary information

It seems that the slowest queries are related to artifacts search.

Note that:

• If you repeat this steps, the results will be slower each time. This was reproduced after a fresh start of TheHive.
• Sometimes, web refreshing (F5) is needed after creating an observable in order to see it on the observable list.
• Sometimes, although logs say that artifacts list was retrieved, it's not shown on the web interface.
• I removed warnings and status requests to clean the logs:

[info] o.t.s.AccessLogFilter - 172.31.0.1 GET /api/status took 0ms and returned 200 389 bytes
[warn] o.t.s.q.InputFilter - Use of filter {"status": "FString(Ok)"} is deprecated. Please use {"_is":{"status":"FString(Ok)"}}

1. Create a case

[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case took 861ms and returned 201 631 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 GET /api/case/12384?nstats=true took 23ms and returned 200 746 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/alert/_stats took 20ms and returned 200 11 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/task/log/_search?range=0-100&nparent=1 took 376ms and returned 200 2 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/connector/cortex/action/_search?range=0-100&sort=-startDate took 10ms and returned 200 2 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/task/_stats took 472ms and returned 200 11 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 GET /api/case/12384/links took 18ms and returned 200 2 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 GET /fonts/SourceSansPro-Semibold.otf took 3ms and returned 304 0 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_stats took 23062ms and returned 200 11 bytes

4. Wait for observable type list to be displayed after clicking on it

[info] o.t.s.AccessLogFilter - 172.31.0.1 GET /api/list/list_artifactDataType took 6654ms and returned 200 718 bytes

5. Create observable

[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/12384/artifact took 9718ms and returned 201 286 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_stats took 17373ms and returned 200 42839 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_stats took 17414ms and returned 200 42839 bytes

6. Observable list update: 54751ms to load from "Observable List (0 of)" to "Observable List (1of 1)" and 43450ms to load the observable

[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_search?range=all&sort=-startDate&nstats=true took 88671ms and returned 200 310 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_stats took 54016ms and returned 200 28 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_stats took 53998ms and returned 200 30 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_stats took 53649ms and returned 200 51 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_stats took 53697ms and returned 200 31 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 GET /api/stream/gSiYnwUnQCwMmKvwvL5d took 60019ms and returned 200 2 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 GET /api/stream/ZqBzeoLp2pp8aMx2kCLr took 60014ms and returned 200 2 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 GET /api/stream/Asqmz1XFZzFie88xnboP took 153847ms and returned 200 1488 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_search?range=all&sort=-startDate&nstats=true took 80449ms and returned 200 310 bytes

7. Go back to case list

[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/_search?range=0-15&sort=-flag&sort=-startDate&nstats=true took 887ms and returned 200 29883 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/_stats took 1005ms and returned 200 10356 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/_stats took 545ms and returned 200 34 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/_stats took 520ms and returned 200 13 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_stats took 74695ms and returned 200 30 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_stats took 74690ms and returned 200 74 bytes
[info] o.t.s.AccessLogFilter - 172.31.0.1 POST /api/case/artifact/_search?range=all&sort=-startDate&nstats=true took 74808ms and returned 200 914 bytes

[mamoedo]

On a side note, the migration was stopped before testing TheHive, and the machine only runs TheHive4. Migration was running since mid May. #1340 #1341

[nadouani]

Hello @mamoedo the UI un RC3 was relying on old APIs (the backward compatible APIs). Queries that were optimised in TheHive 3 are no longer optimised in TheHive 4 because of:

• the data model is not the same
• multi tenancy and rbac checks adds an overhead

Issue #1410 aims to solve those performance issues by:

• Using APIs version 1 (optimised for graph data navigation and traversal) in case, alert, task and observables listing
• Using pagination for lists that can be huge like observables (in TheHive3 and TheHive4 RC3 observables list had a client side pagination)

Same answer for the migration. I don't really understand why and how your migration was running since mid may, this sound too much. But note that the migration tool has also been refined to improved the migration procedure and performance.

Thanks for you contribution, and please be patient we are working on these topics.

[nadouani]

Hello @mamoedo #1410 aims to use the API v1 in the UI, most of the sections have been migrated to improve the performance of the queries.

With that being said, apis like /api/list/list_artifactDataType are not expected to take 6 seconds.

I would like to have your feedback about this issue using the next release 4.0.

I'll keep this issue open for tracking purposes.

[mamoedo]

Sure! I'm looking forward to test it 😄

[nadouani]

Closing this issue for now, please just reopen it if needed