Can't secure ElasticSearch connection

[1earch]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	RedHat 7
TheHive version / git hash	3.3.1-1, d8f0290
Package Type	Binary

Problem Description

TheHive can't contact ElasticSearch cluster when SearchGuard is enabled (plaintext tcp connection is opened by TheHive each time).

Note: when disabling SearchGuard in ElasticSearch (searchguard.disabled: true), the communication between TheHive and ElasticSearch was working (but not secure).

Steps to Reproduce

1. Fresh install ElasticSearch, and secure it with SearchGuard (see config bellow)
2. Fresh install TheHive, using install guide
3. Copy /opt/thehive/conf/logback.conf in /etc/thehive, as mentionned in Systemd: thehive.service: Failed with result 'exit-code'. #811
4. Configure TheHive as bellow

Complementary information

TheHive config (SearchGuard tentative)

search {
  index = the_hive
  cluster = "cluster-name"
  host = ["xxx.xxx.xxx.xxx:9300"]

  nbshards = 5
  nbreplicas = 1

  search.guard.keyStore.path = "/path/to/keystore.jks"
  search.guard.keyStore.password = "password"
  search.guard.trustStore.path = "/path/to/truststore.jks"
  search.guard.trustStore.password = "password"
  search.guard.hostVerification = false
  search.guard.hostVerificationResolveHostname = false
}

TheHive config (XPack tentative)

search {
  index = the_hive
  cluster = "cluster-name"
  host = ["xxx.xxx.xxx.xxx:9300"]

  nbshards = 5
  nbreplicas = 1

  search.username = "admin"
  search.password = "admin"
  search.ssl.enabled = true
  search.ssl.ca = "/path/to/CA.pem"
  search.ssl.certificate = "/path/to/cert.pem"
  search.ssl.key = "/path/to/key.pem"
}

ElasticSearch SearchGuard config

searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemkey_filepath: /path/to/key.pem
searchguard.ssl.transport.pemcert_filepath: /path/to/cert.pem
searchguard.ssl.transport.pemtrustedcas_filepath: /path/to/CA.pem

searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemkey_filepath: /path/to/key.pem
searchguard.ssl.http.pemcert_filepath: /path/to/cert.pem
searchguard.ssl.http.pemtrustedcas_filepath: /path/to/CA.pem

TheHive log

[debug] o.e.c.t.TransportClientNodesService - failed to connect to node [{#transport#-1}{6BGW0fmWQYmsnNAouqS8Ow}{xxx.xxx.xxx.xxx}{xxx.xxx.xxx.xxx:9300}], ignoring...
org.elasticsearch.transport.ConnectTransportException: [][xxx.xxx.xxx.xxx:9300] general node connection failure
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:595)
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:117)
        at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:361)
        at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:407)
        at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:357)
        at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:390)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.IllegalStateException: handshake failed
        at org.elasticsearch.transport.TcpTransport.executeHandshake(TcpTransport.java:1636)
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:577)
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:117)
        at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:361)
        at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:407)
        at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:357)
        at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:390)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Caused by: org.elasticsearch.transport.TransportException: connection reset
        at org.elasticsearch.transport.TcpTransport.onChannelClosed(TcpTransport.java:1678)
        at org.elasticsearch.transport.TcpTransport.lambda$openConnection$1(TcpTransport.java:558)
        at org.elasticsearch.transport.netty4.Netty4Transport.lambda$connectToChannels$2(Netty4Transport.java:356)
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
        at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
        at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
        at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1058)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:686)

ElasticSearch log

This line was generated both when we tried configuring TheHive with SearchGuard or with XPack.

[2019-07-09T09:24:26,003][WARN ][c.f.s.s.t.SearchGuardSSLNettyTransport] [HOSTNAME] Someone (/xxx.xxx.xxx.xxx:42650) speaks transport plaintext instead of ssl, will close the channel

[1earch]

After an in-depth study and debugging (adding some debugging message in elastic4play), I finally found the solution. In the default config, all config parameters related to ElasticSearch are under a search block. However, all parameters related to SearchGuard/X-Pack re-mentionned the search block in their name. Thus, elastic4play don't found them and defaults to non-TLS connection.

The config file should be:

search {
  index = the_hive
  cluster = "cluster-name"
  host = ["xxx.xxx.xxx.xxx:9300"]

  nbshards = 5
  nbreplicas = 1

  # X-Pack config
  username = "admin"
  password = "admin"
  ssl.enabled = true
  ssl.ca = "/path/to/CA.pem"
  ssl.certificate = "/path/to/cert.pem"
  ssl.key = "/path/to/key.pem"

  # SearchGuard config
  guard.keyStore.path = "/path/to/keystore.jks"
  guard.keyStore.password = "password"
  guard.trustStore.path = "/path/to/truststore.jks"
  guard.trustStore.password = "password"
  guard.hostVerification = false
  guard.hostVerificationResolveHostname = false
}

I will prepare a PR to fix this.