Akka Crashing

[JamesHolmes1981]

Akka Crashing

This is potentially related to (#877) when trying to configure TheHive in a clustered configuration to run within kubernetes.

Request Type

Bug

Work Environment

Kubernetes

Question	Answer
OS version (server)	docker
OS version (client)	Ubuntu
TheHive version / git hash	thehive:latest docker image
Package Type	docker

Problem Description

Describe the problem/bug as clearly as possible.

Steps to Reproduce

1. Configured Akka clustering with the following config. The hostname resolves to a list of all IPs associated with the Akka cluster.

akka {
  loglevel = DEBUG

  remote {
    netty.tcp {
      port = 2552
    }
  }

  cluster.seed-nodes = ["akka.tcp://[email protected]:2552"]
}

Possible Solutions

The error in the log suggest some sort of issue related to mismatched Akka module versions.

"Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.5.17] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries.Possibly conflicting versions [2.5.17, 2.5.11] in libraries [akka-protobuf:2.5.17, akka-actor:2.5.17, akka-slf4j:2.5.17, akka-remote:2.5.11, akka-cluster:2.5.11, akka-stream:2.5.17, akka-cluster-tools:2.5.11]"

Complementary information

Using secret: XXXXXXXXXXXXXXXXXx
Warning automatic elasticsearch host config fails
elasticsearch host not configured
[info] o.r.Reflections - Reflections took 177 ms to scan 5 urls, producing 122 keys and 1336 values 
[info] module - Loading model class models.CaseTemplateModel
[info] module - Loading model class models.CaseModel
[info] module - Loading model class org.elastic4play.services.AttachmentModel
[info] module - Loading model class models.UserModel
[info] module - Loading model class models.ArtifactModel
[info] module - Loading model class models.DashboardModel
[info] module - Loading model class org.elastic4play.services.DBListModel
[info] module - Loading model class connectors.cortex.models.ReportTemplateModel
[info] module - Loading model class models.AuditModel
[info] module - Loading model class models.AlertModel
[info] module - Loading model class connectors.cortex.models.ActionModel
[info] module - Loading model class models.TaskModel
[info] module - Loading model class connectors.cortex.models.JobModel
[info] module - Loading model class models.LogModel
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/thehive/lib/com.google.inject.guice-4.2.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[info] a.e.s.Slf4jLogger - Slf4jLogger started
[info] a.r.Remoting - Starting remoting
[info] a.r.Remoting - Remoting started; listening on addresses :[akka.tcp://[email protected]:2552]
[info] a.r.Remoting - Remoting now listens on addresses: [akka.tcp://[email protected]:2552]
[info] a.c.Cluster(akka://application) - Cluster Node [akka.tcp://[email protected]:2552] - Starting up...
[info] a.c.Cluster(akka://application) - Cluster Node [akka.tcp://[email protected]:2552] - Registered cluster JMX MBean [akka:type=Cluster]
[info] a.c.Cluster(akka://application) - Cluster Node [akka.tcp://[email protected]:2552] - Started up successfully
[warn] a.u.ManifestInfo - Detected possible incompatible versions on the classpath. Please note that a given Akka version MUST be the same across all modules of Akka that you are using, e.g. if you use [2.5.17] all other modules that are released together MUST be of the same version. Make sure you're using a compatible set of libraries.Possibly conflicting versions [2.5.17, 2.5.11] in libraries [akka-protobuf:2.5.17, akka-actor:2.5.17, akka-slf4j:2.5.17, akka-remote:2.5.11, akka-cluster:2.5.11, akka-stream:2.5.17, akka-cluster-tools:2.5.11]
Uncaught error from thread [application-akka.remote.default-remote-dispatcher-5]: Receiver class akka.cluster.ClusterActorRefProvider does not define or inherit an implementation of the resolved method abstract serializationInformation()Lakka/serialization/Serialization$Information; of interface akka.actor.ActorRefProvider., shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[application]
java.lang.AbstractMethodError: Receiver class akka.cluster.ClusterActorRefProvider does not define or inherit an implementation of the resolved method abstract serializationInformation()Lakka/serialization/Serialization$Information; of interface akka.actor.ActorRefProvider.
	at akka.serialization.Serialization.serializationInformation(Serialization.scala:161)
	at akka.serialization.Serialization.withTransportInformation(Serialization.scala:167)
	at akka.serialization.Serialization.deserializeByteArray(Serialization.scala:221)
	at akka.serialization.Serialization.$anonfun$deserialize$3(Serialization.scala:211)
	at scala.util.Try$.apply(Try.scala:209)
	at akka.serialization.Serialization.deserialize(Serialization.scala:205)
	at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:33)
	at akka.remote.DefaultMessageDispatcher.payload$lzycompute$1(Endpoint.scala:64)
	at akka.remote.DefaultMessageDispatcher.payload$1(Endpoint.scala:64)
	at akka.remote.DefaultMessageDispatcher.payloadClass$1(Endpoint.scala:65)
	at akka.remote.DefaultMessageDispatcher.dispatch(Endpoint.scala:108)
	at akka.remote.EndpointReader$$anonfun$receive$2.applyOrElse(Endpoint.scala:991)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:454)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Akka Config. The domain thehive-discovery.default.svc.cluster.local is a headless service that returns a list of IPs of all running instances of TheHive

akka {
  loglevel = DEBUG

  remote {
    netty.tcp {
      port = 2552
    }
  }

  cluster.seed-nodes = ["akka.tcp://[email protected]:2552"]
}

[zpriddy]

Look at #877 it may be the same. What version of AKKA are you running?

[crackytsi]

I guess that this was related to a bug in akka. With TheHive 3.4 Akka seems to work more stable.
Did you test latest version?