StreamSrv: Unexpected message : StreamNotFound

[ParanoidRat]

Request Type

Bug?

Work Environment

Question	Answer
OS version (server)	CentOS
OS version (client)	Ubuntu
TheHive version / git hash	3.0.3
Package Type	Docker
Browser type & version	Chromium 63

Problem Description

I see multiple warning notifications in UI (stack of red rectangles in the lower left corner) saying
StreamSrv: Unexpected message : StreamNotFound (class services.StreamActor$StreamNotFound$)
and fading away.

EDIT 1: When Elasticsearch index is deleted and TheHive service is reinitialized in docker (removed, recreated) it seems to create the_hive_X without giving any options to provide admin password.

EDIT 2: When TheHive service is reduced down to one replica (one app instance running in the docker swarm) everything seems to work as expected without any change in configuration (just deletion of ES index)

EDIT 3: When left to its own devices, at some point even one instance of TheHive starts to produce receive dead message in the log and show the 504 Gateway Time-out error from Nginx (has nginx/1.13.8 in the body) in the UI (red notification popups).

Full log could be viewed here

Steps to Reproduce

1. Deploy Docker Swarm 17.12.0-ce cluster with 2 worker and 3 master nodes
2. Deploy Nginx 1.13.8 reverse proxy service on both workers to act as a load-balancer and TLS termination (Let's Encrypt certs)
3. Deploy TheHive as a service replicated to both worker nodes and load-balanced by Nginx
4. Login to TheHive (data index pre-existed from previous deployment)
5. Navigate to a case or other UI element or just wait

Complementary information

TheHive config

play.crypto.secret="************"

search {
  index = the_hive
  cluster = ****
  host = ["****0:9300", "****1:9300", "****2:9300", "****3:9300"]
  keepalive = 1m
  pagesize = 50
}

auth {
	type = [local]

	ad {
	}

	ldap {
	}
}

cortex {
}

misp {
  interval = 1h
}

Nginx config

server {
  listen 443;
  server_name *****.org;

  resolver 127.0.0.11 valid=30s;

  ssl on;
  ssl_certificate /etc/letsencrypt/live/*****/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/*****/privkey.pem;

  ssl_stapling on;
  ssl_stapling_verify on;
  ssl_trusted_certificate /etc/letsencrypt/live/*****/chain.pem;

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers HIGH:!aNULL:!MD5;

  location / {
    set $docker_service thehive_internal;
    set $docker_port 9000;

    proxy_pass http://$docker_service:$docker_port;

    proxy_http_version 1.1;
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header Proxy-Connection "Keep-Alive";
  }
}

[ParanoidRat]

For those who might face similar problem. I have an answer to the problem of 504 Gateway Time-out.

After dropping Keep-Alive header sporadic gateway time-outs stopped. I also stopped using variables for service port (vars apparently evaluated on each request in nginx and thus, expensive), added some proxy timeout settings and some proxy headers for logging, so nginx config would look like below.

 location / {
    set $docker_service thehive_internal;

    proxy_pass http://$docker_service:9000;

    # time out settings
    proxy_connect_timeout 159s;
    proxy_send_timeout 600;
    proxy_read_timeout 600;
    proxy_buffer_size 64k;
    proxy_buffers 16 32k;
    proxy_busy_buffers_size 64k;
    proxy_temp_file_write_size 64k;
    
    proxy_http_version 1.1; # needed for Play Framework chunked responses to function
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }

I still get StreamNotFound error when scale TheHive service to more than one worker.

[To-om]

If you want to setup TheHive in cluster, you must wait version 3.1 (including the patch
21590aa) and add some configuration. You can find here some documentation.