Attachments download from the Hive are really slow

[cyberman6877]

Request Type

Bug - Issue

Work Environment

Question	Answer
OS version (server)	CentOS Linux release 7.4.1708 (Core) -
uname -a
3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

| OS version (client) | Windows 7 .
| TheHive version / git hash |
TheHive 3.0.9
Elastic4Play 1.4.6
Play 2.6.7
Elastic4s 5.6.0
ElasticSearch 5.6.2
CORTEX
CORTEX-PHSA - 2.0.4 (OK)
cortex1 - 2.0.4 (OK)

| Package Type | Docker - 18.03.1-ce
| Browser type & version | CHrome - Version 67.0.3396.87

Problem Description

When we want to download a document of any type from the Hive, it takes a long time to download, for example - 780KB file (less than 1mb) takes approximate 2 minutes waiting before the file download has initiated,

Steps to Reproduce

1. Login to the HIVE
2. Select any attachment
3. Click download

Docker compose settings are:

PLEASE NOTE SOME OF THE INFORMATION IN THIS FILE HAVE BEEN TRIMMED FOR SECURITY PURPOSES, SO BELOW IS ONLY A SAMPLE WITH WHAT WE THINK THE NECESSARY VALUES FOR YOU TO INVESTIGATE.

cat docker-compose.yml
version: "3"
services:
elasticsearch:
image: xxxxxx-elasticsearch:5.6.2
environment:
- http.host=0.0.0.0
- transport.host=0.0.0.0
- xpack.security.enabled=false
- cluster.name=hive
- script.inline=true
- thread_pool.index.queue_size=100000
- thread_pool.search.queue_size=100000
- thread_pool.bulk.queue_size=100000
ulimits:
nofile:
soft: 65536
hard: 65536
cortex:
image: xxxxxxxxx-cortex:2.0.4-1

depends_on:
- elasticsearch
- cortex

Possible Solutions:
''''''''''''''''''''''''''''''''''''''''''
We have eliminated the network since other docker containers share same internal subnet and file download speeds from other containers is fast (normal speed).

We suspect that we need to tune a parameter either above in the Docker configuration file or in ElastcSearch? could it be an index issue? Is there any parameters from ElasticSearch or Docker that we need to modify and increase?

I am also happy to send you an email directly from my organisation and we can setup a call to walk through it.

Thanks,

cyberman6877

[To-om]

Do you notice other slowness in TheHive, except attachment download ?
Do you have any exotic settings in your application.conf ? (datastore.chunksize or datastore.hash.extra for example)

[crackytsi]

I have the same issue... no exotic config as far I see...
All other things work fine.

[To-om]

Do you use Docker too ?

[crackytsi]

No I use dpkg package.
When I start the download the first bytes come very fast. After that it takes quite a long time and finally the download is complete.

[To-om]

Can you check that your antivirus or proxy doesn't affect the download ?
Can you get the download URL (using the developer tool of your browser, copy as cURL) and curl it locally in TheHive server ?

[crackytsi]

Hi,
The download from TheHive Server locally is slow as well...
There is no Proxy inbetween or used in any Kind.

e.g. I have a 84 KB Excel file, it takes more than 1 minutes.
Strange is, that all Bytes are received very soon, (e.g. 86005 of 86005 Bytes) but the download is not finished...

root@thehive:~# time curl -XGET -u user:pass "https://fqdn:9443/api/datastore/5f51145b2d31beb7e13c6d727607132c973b3dd5e240c7f4dad371d25e94c831?name=Filename%20Demo%20Test.xlsx" -o tmp
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 86005 0 86005 0 0 1429 0 --:--:-- 0:01:00 --:--:-- 0

real 1m0.184s
user 0m0.048s
sys 0m0.024s

[cyberman6877]

Hi,

My issue is exactly the same as the one highlighted by crackytsi. We are running on Docker.

Nothing fancy in application.conf and checked antivirus and proxy, all check out. I personally think its Elasticsearch which is not designed for for retrieval of attachments but I can't prove it.

Any ideas would be very much appreciated as it's very annoying to have to wait 2 minutes to download a 780KB file.

[To-om]

@crackytsi Can you capture the network traffic while you curl the file ? It would be interesting to see http(s) and elasticsearch connections together in the same capture.

[crackytsi]

Hi @To-om
What do you mean? Something like:
tcpdump port 9200 or port 8443

[To-om]

yes
tcpdump -ni lo -w thehive_654.pcap -f "port 9300 or port 9443"
(if TheHive uses 9443/tcp)
If you can send me this file, it helps me to understand what happens. Be careful, the network capture may contain sensitive information (the file content and login/password if https is not used).

[crackytsi]

I have the file, but I don't know how to remove all the sensitive data... :-/

[crackytsi]

[To-om]

The easiest solution is to not use sensitive data: create a dedicated user and file.
Otherwise, you can send me only the list of network packets (without the content):
tcpdump -n -r thehive_654.pcap
or better: tshark -n -r thehive_654.pcap

[crackytsi]

Ok, problem gets more complex....
On my test system I don't have the issue, only on my prod system...
Test runs 3.1 while prod rund 3.0.latest.

Can I directly do a curl on Elastic-search?

[crackytsi]

Everything happens on the same machine....
headers.txt

[To-om]

It seams that the problem is not related to ElasticSearch. I suspect SSL problem.
Can you enable http port and try to reproduce the issue ?

[crackytsi]

Yes, you seems to be right.
On http port Problem does not occour...

[To-om]

It is not a good news, it means that the problem is not in TheHive but in PlayFramework (or somewhere else)

[crackytsi]

Do not shoot the messenger for bringing bad News :-(
Sorry for that.... Let me know if I can do some debugs... I try to dig into the issue...

[cyberpescadito]

Same issue here.
The download "start" (just the request, the file size appear but the download himself isn't starting), theres a 60 seconds break, and finally the real start of the download, with a normal speed.
I got this issue since long time too

[cyberpescadito]

thanks to a workmate, he find a solution to bypass this issue.
put an nginx in front of thehive (locally) which communicate with TheHive in http.
This way, the attachments files download is instant

[crackytsi]

hmm, good to know, but technically I don't want to add an apache/nginx etc. for TheHive to make ssl to be fast ;)

[cyberpescadito]

Yes, this is just a "bypass", this isn't solving the issue

[cyberman6877]

Thank you for all the work on this but technically we can't add an apache/nginx just to make this work although I appreciate it is an option and a workaround.

Any idea on how this issue will be solved permanently in the Hive without this workaround?

Also, is this Docker environment specific only?

Thanks!

[cyberpescadito]

No this isn't Docker specific

[cyberman6877]

Hi All,

So where are we now with this?

Thanks

[To-om]

The problem seems to be related to SSL when SSL is configured on TheHive. I suspect a bug in PlayFramework (or AkkaHttp) but as long as I can not reproduce it, I will not be able to fix it or report it to the PlayFamework team.
I advise you to add a reverse proxy (nginx for example) in front of TheHive that manage SSL and forward HTTP requests on TheHive (without SSL). With this solution, it is not possible to authenticate users with certificate.

[crackytsi]

I can try to check next week if it is in any relation to data inside of thehive or related to certificate topics etc.
If you have and idear you want me to check, pls leat me know

[crackytsi]

@To-om After the upgrade from 3.0.10 to 3.1.0 suddenly the downloads are fast again. Does this bring any new idear? Or has the issue been solved indirectly in any kind? e.g. by encoding filenames

[crackytsi]

Hi,
I have to mention that it is not "fast again", instead the downloads are broken.

@To-om This is really a bad bug. Do you have any hints or comments how we could identify the bug? If it helps we could do a web-conference. Do you think that is possible? I know you are very busy, but I think this bug has several related topics (like #760, #759, #654)

Unforunately I'm don't understand scala, but maybe you can point me to the classes or lines of code that cover the https part?

[To-om]

Hi @crackytsi,
The problems with this issue are 1) the bug is likely located in PlayFramework 2) I can't reproduce it.
After my holidays (from this evening), I'll rebuild a test platform to try to reproduce the problem. I'll probably need your help for that.
Once I can reproduce the problem, I can try to identify it and create an issue on PlayFramework if needed.

[crackytsi]

@To-om You will get all my support, just let me know what I can do ;)

[crackytsi]

Hi @To-om Any update on this topic?

[teancum144]

I have the same problem -- downloading attachments is painfully slow when using the Chrome browser, but comparably fast when using the Firefox browser. Any thoughts on this?

[rj-chap]

I see this bad boy was closed. We have the same issue here in July of 2022. Was this ever rectified officially?