Skip to content
This repository was archived by the owner on Mar 3, 2023. It is now read-only.

Deploy a production instance of The Hive #1

Closed
7 tasks done
jakubgs opened this issue Oct 5, 2020 · 79 comments
Closed
7 tasks done

Deploy a production instance of The Hive #1

jakubgs opened this issue Oct 5, 2020 · 79 comments
Assignees
@jakubgs

Comments

@jakubgs
Copy link
Member

jakubgs commented Oct 5, 2020 •
edited
Loading

The instructions are available here:
https://github.com/TheHive-Project/TheHiveDocs/blob/master/TheHive4/Installation/Install_deb.md
https://github.com/TheHive-Project/CortexDocs/blob/master/installation/install-guide.md

@corpetty already configured a test instance at master-01.gc-us-central1-a.thehive.misc: https://github.com/status-im/infra-misc/commit/44e26d18

Here are his notes: https://notes.status.im/1cGF__y4Q1GuKCt2OdONgw

Steps:

  • Create repo for The Hive infra configuration
  • Deploy The Hive frontend and backend hosts
  • Extract Cassandra rope into separate repo
  • Deploy Cassandra cluster
  • Deploy ElasticSearch cluster
  • Deploy Cortex service
  • Deploy TheHive service
@jakubgs jakubgs self-assigned this Oct 5, 2020
@jakubgs
Copy link
Member Author

jakubgs commented Oct 5, 2020

I've extracted the Ansible role for Cassandra cluster configuration into a separate repo in https://github.com/status-im/infra-hq/commit/fb7efeb3:
https://github.com/status-im/infra-role-cassandra

The ElasticSearch cluster role is already available:
https://github.com/status-im/infra-role-elasticsearch

@jakubgs
Copy link
Member Author

jakubgs commented Oct 7, 2020

While deploying hosts for Cassandra and ElasticSearch i found some weird bug in Terraform:

module.hive_slave.google_compute_instance.host[2]: Creating...
module.hive_slave.google_compute_instance.host[0]: Creating...
module.hive_slave.google_compute_instance.host[1]: Creating...

Error: Error creating instance: googleapi: Error 400: The disk resource 'projects/russia-servers/zones/us-central1-a/disks/data-slave-03-gc-us-central1-a-thehive-prod' is already being used by 'projects/russia-servers/zones/us-central1-a/instances/slave-03-gc-us-central1-a-thehive-prod', resourceInUseByAnotherResource


Error: Error creating instance: googleapi: Error 400: The disk resource 'projects/russia-servers/zones/us-central1-a/disks/data-slave-02-gc-us-central1-a-thehive-prod' is already being used by 'projects/russia-servers/zones/us-central1-a/instances/slave-02-gc-us-central1-a-thehive-prod', resourceInUseByAnotherResource


Error: Error creating instance: googleapi: Error 400: The disk resource 'projects/russia-servers/zones/us-central1-a/disks/data-slave-01-gc-us-central1-a-thehive-prod' is already being used by 'projects/russia-servers/zones/us-central1-a/instances/slave-01-gc-us-central1-a-thehive-prod', resourceInUseByAnotherResource

It say that:

The disk resource 'projects/russia-servers/zones/us-central1-a/disks/data-slave-03-gc-us-central1-a-thehive-prod'
is already being used by 'projects/russia-servers/zones/us-central1-a/instances/slave-03-gc-us-central1-a-thehive-prod'

But that's clearly not true if I look at the Google Console:
google_console_the_hive_disks
So this is some seriously weird shit.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 7, 2020 •
edited
Loading

I figured it out. We had a bug in how we were attaching the data volume and I fixed it in: status-im/infra-tf-google-cloud@da60baab

@jakubgs
Copy link
Member Author

jakubgs commented Oct 7, 2020

Our current role for Cassandra setup doesn't configure any authentication. But considering this will be handling more sensitive data it will make sense to change that. If we look at The Hive configuration we can see it allows for user/password auth:

db.janusgraph {
  storage {
    ## Cassandra configuration
    # More information at https://docs.janusgraph.org/basics/configuration-reference/#storagecql
    backend: cql
    // hostname: ["ip1", "ip2"]
    # Cassandra authentication (if configured)
    // username: "thehive"
    // password: "password"
    cql {
      cluster-name: thp
      keyspace: thehive
    }
  }
}

And Cassandra allows for configuration including user/password auth by editing the config to include:

authenticator: PasswordAuthenticator

And then creating users using CQL commands:

CREATE ROLE dba WITH SUPERUSER = true AND LOGIN = true AND PASSWORD = 'super';

https://cassandra.apache.org/doc/latest/operating/security.html#enabling-password-authentication

@jakubgs
Copy link
Member Author

jakubgs commented Oct 7, 2020

Though apparently:

Note that using PasswordAuthenticator also requires the use of CassandraRoleManager.

Which means:

Cassandra uses database roles, which may represent either a single user or a group of users, in both authentication and permissions management. Role management is an extension point in Cassandra and may be configured using the role_manager setting in cassandra.yaml. The default setting uses CassandraRoleManager, an implementation which stores role information in the tables of the system_auth keyspace.

https://cassandra.apache.org/doc/latest/operating/security.html#operation-roles

@jakubgs
Copy link
Member Author

jakubgs commented Oct 7, 2020 •
edited
Loading

While working on this I found a few other bugs I fixed:

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020 •
edited
Loading

I've adjusted the Cassandra role to handle user authentication too:

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020

Okay, we have a Cassandra cluster up with user auth:

[email protected]:~ % /opt/cassandra/bin/cqlsh 10.2.1.90 -u thehive 
Password: 
Connected to thp at 10.2.1.90:9042.
[cqlsh 5.0.1 | Cassandra 3.11.7 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
thehive@cqlsh> DESCRIBE CLUSTER

Cluster: thp
Partitioner: Murmur3Partitioner

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020

What I would like to understand is how Cortex ties into The Hive, because the installation docs do not reference Cortex even once.

Based on the Docker Compose they have in their repo TheHive uses Cortex.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020

The config file contains a section for Cortex:

## CORTEX configuration
# More information at https://github.com/TheHive-Project/TheHiveDocs/TheHive4/Administration/Connectors.md
# Enable Cortex connector
// play.modules.enabled += org.thp.thehive.connector.cortex.CortexModule
// cortex {
//  servers: [
//    {
//      name: "local"                # Cortex name
//      url: "http://localhost:9001" # URL of Cortex instance
//      auth {
//        type: "bearer"
//        key: "***"                 # Cortex API key
//      }
//      ws {}                        # HTTP client configuration (SSL and proxy)
//    }
//  ]
// }

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020

The ElasticSearch version in Cortex installation docs is 7.8.1 but our Ansible role is still on 7.8.0:
https://github.com/status-im/infra-role-elasticsearch/blob/9a429d3c44747a32abc85243b18b127ed99575c7/defaults/main.yml#L11
So I'll have to upgrade that first.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020

The Cortex config does provide an option to setup ElasticSearch with auth:

## ElasticSearch
search {
  # Name of the index
  index = cortex
  # ElasticSearch instance address.
  # For cluster, join address:port with ',': "http://ip1:9200,ip2:9200,ip3:9200"
  uri = "http://127.0.0.1:9200"

  ## Advanced configuration
  # Scroll keepalive.
  #keepalive = 1m
  # Scroll page size.
  #pagesize = 50
  # Number of shards
  #nbshards = 5
  # Number of replicas
  #nbreplicas = 1
  # Arbitrary settings
  #settings {
  #  # Maximum number of nested fields
  #  mapping.nested_fields.limit = 100
  #}

  ## Authentication configuration
  #search.username = ""
  #search.password = ""

https://github.com/TheHive-Project/Cortex/blob/a0f9053c4fcabe6a1eb46c1eee6ae32c1fa3fbef/conf/application.sample#L11-L49

But the Open Source version of ElasticSearch does not provide Auth, so we'll skip that.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020

Another fix for file permissions: status-im/infra-role-elasticsearch@32ac6e39

@corpetty
Copy link

corpetty commented Oct 8, 2020

we can handle cortex later. Getting TheHive up and running allows me to start inviting people and creating case templates, which I can add Cortex analyzers to later.

You're planning on putting the Cortex instance on the same machine?

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020

Yes, that was the idea.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020

I don't see any flags for specifying config location:

 > d run --rm -it thehiveproject/thehive:3.4.2 --help       
    Available options:
    --no-config                           | do not try to configure TheHive (add secret and elasticsearch)
    --no-config-secret                    | do not add random secret to configuration
    --secret <secret>                     | secret to secure sessions
    --show-secret                         | show the generated secret
    --no-config-es                        | do not add elasticsearch hosts to configuration
    --es-uri <uri>                        | use this string to configure elasticsearch hosts (format: http(s)://host:port,host:port(/prefix)?querystring)
    --es-hostname <host>                  | resolve this hostname to find elasticsearch instances
    --no-config-cortex                    | do not add Cortex configuration
    --cortex-proto <proto>                | define protocol to connect to Cortex (default: http)
    --cortex-port <port>                  | define port to connect to Cortex (default: 9000)
    --cortex-url <url>                    | add Cortex connection
    --cortex-hostname <host>              | resolve this hostname to find Cortex instances
    --cortex-key <key>                    | define Cortex key
    --auto-migration                      | migrate the database, if needed
    --create-admin <user> <password>      | create the first admin user, if not exist yet
    --create-user <user> <role> <password>| create a user, only in conjunction with admin creation

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020 •
edited
Loading

According to the doc:

Following configurations are required to start TheHive successfully:

  • Secret key configuration - /etc/thehive/secret.conf
  • Database configuration - /etc/thehive/application.conf
  • File storage configuration - /etc/thehive/application.conf
  • Logging configuration - /etc/thehive/logback.xml

@jakubgs
Copy link
Member Author

jakubgs commented Oct 8, 2020 •
edited
Loading

According to Docker docs:

You can specify a custom TheHive configuration file (application.conf) by adding the following lines in the thehive section of your docker-compose file:

   - /path/to/application.conf:/etc/thehive/application.conf

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020

It appears there is no Docker image for 4.0.0: https://hub.docker.com/r/thehiveproject/thehive/tags

The newest one there is is for 3.4.2, which appears to be a few months old and uses a different format for configuration.
And their install docs don't even state for which version they are...

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020

There is a Dockerfile available tho:
https://github.com/TheHive-Project/TheHive/blob/4.0.0/package/docker/Dockerfile

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020

Building the image fails on downloading some JAR:

Downloading sbt launcher for 1.3.0:
  From  http://repo.scala-sbt.org/scalasbt/maven-releases/org/scala-sbt/sbt-launch/1.3.0/sbt-launch.jar
    To  /root/.sbt/launchers/1.3.0/sbt-launch.jar
Download failed. Obtain the jar manually and place it at /root/.sbt/launchers/1.3.0/sbt-launch.jar

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020 •
edited
Loading

I fixed that by adding:

  mkdir -p /root/.sbt/launchers/1.3.0 && \
  curl -sL https://repo1.maven.org/maven2/org/scala-sbt/sbt-launch/1.3.0/sbt-launch-1.3.0.jar -o /root/.sbt/launchers/1.3.0/sbt-launch.jar && \

To the Dockerfile. But then it fails on something else:

[error] sbt.librarymanagement.ResolveException: Error downloading org.thehive-project:elastic4play_2.12:1.12.2-SNAPSHOT
[error]   Not found
[error]   Not found
[error]   not found: /root/.ivy2/local/org.thehive-project/elastic4play_2.12/1.12.2-SNAPSHOT/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/org/thehive-project/elastic4play_2.12/1.12.2-SNAPSHOT/elastic4play_2.12-1.12.2-SNAPSHOT.pom
[error]   not found: https://dl.bintray.com/thehive-project/maven/org/thehive-project/elastic4play_2.12/1.12.2-SNAPSHOT/elastic4play_2.12-1.12.2-SNAPSHOT.pom
[error]   not found: https://artifacts.elastic.co/maven/org/thehive-project/elastic4play_2.12/1.12.2-SNAPSHOT/elastic4play_2.12-1.12.2-SNAPSHOT.pom

I opened an issue about it: TheHive-Project/TheHive#1573

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020

I guess I'm forced to use their APT repo:

curl https://raw.githubusercontent.com/TheHive-Project/TheHive/master/PGP-PUBLIC-KEY | sudo apt-key add -
echo 'deb https://deb.thehive-project.org stable main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list
sudo apt-get update
sudo apt-get install thehive4

https://github.com/TheHive-Project/TheHiveDocs/blob/master/TheHive4/Installation/Install_deb.md#stable-versions

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020

They clearly have 4.0.0 out on GitHub, and yet when you use the APT repo you get:

[email protected]:/docker/thehive/logs % qapt info thehive
Package: thehive
Version: 3.4.2-1
Priority: optional
Section: java
Source: thehive
Maintainer: TheHive Project <[email protected]>
Installed-Size: 77.5 MB
Depends: java8-runtime-headless
Recommends: elasticsearch
Download-Size: 77.6 MB
APT-Manual-Installed: yes

So they don't consider 4.0.0 as stable? Or they didn't get around to packaging it yet? But it's been out for over 2 months.

I'm confused.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020 •
edited
Loading

They have a 3.5.1 Release release out, but if you look at the tag it's an 3.5.0-RC1:
https://github.com/TheHive-Project/TheHive/releases/tag/3.5.0-RC1

They appear to be VERY sloppy with their versioning.
Issue: TheHive-Project/TheHive#1574

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020 •
edited
Loading

When I look at config samples for TheHive 3.4.2 and Cortex 3.0.1 I see no references to Cassandra:
https://github.com/TheHive-Project/TheHive/blob/3.4.2/conf/application.sample
https://github.com/TheHive-Project/Cortex/blob/3.0.1/conf/application.sample
I see only configuration for ElasticSearch under search key.

But then why is the instance Corey configured using Cassandra? And why does the sample not contain a db key at all?

What the hell is going on?

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020

It does appear like the config on the hoist configured by Corey is using the 4.0.0 format:
https://github.com/TheHive-Project/TheHive/blob/4.0.0/conf/application.sample.conf
But then how it was installed? Since the APT version appears to be 3.4.2.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020

Ooooh, it was never installed using APT:

[email protected]:/opt/thehive % sudo dpkg-query -L thehive 
dpkg-query: package 'thehive' is not installed
Use dpkg --contents (= dpkg-deb --contents) to list archive files contents.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 9, 2020

It appears that the version available through the beta channel of their APT repo isn't 4.0.0 but 3.5.0-RC1:

[email protected]:~ % qapt info thehive
Package: thehive
Version: 3.5.0-0.1RC1
Priority: optional
Section: java
Source: thehive
Maintainer: TheHive Project <[email protected]>
Installed-Size: 87.1 MB
Depends: java8-runtime-headless
Recommends: elasticsearch
Download-Size: 87.3 MB
APT-Sources: https://deb.thehive-project.org beta/main amd64 Packages

@jakubgs
Copy link
Member Author

jakubgs commented Oct 12, 2020

But these /api/stream/* requests don't appear in application or Nginx logs, so they never reach the server.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 12, 2020

Oh wait, I'm seeing something in Nginx logs:

 upstream timed out (110: Connection timed out) while reading response header from upstream,
  client: 162.158.75.223,
  server: hive.status.im,
  request: "GET /api/stream/oOfnsbZpnpNfcGAfCbkK HTTP/1.1",
  upstream: "http://127.0.0.1:9000/api/stream/oOfnsbZpnpNfcGAfCbkK",
  host: "hive.status.im",
  referrer: "https://hive.status.im/index.html"

@jakubgs
Copy link
Member Author

jakubgs commented Oct 12, 2020

My understanding is that the Web UI opens a stream at /api/stream/* to handle user actions, but if no actions take place then the connection times out after 60 seconds which is the default for the proxy_read_timeout Nginx proxy setting.

I've set it to 3600 seconds - 1 hour - in c12b93dc.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 12, 2020

We had issues with Corey logging in initially. He didn't have a public email on his GitHub profile and that might have been the issue.

I had to re-create the account and create a fresh password too.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 12, 2020

Adjusted The Hive role to download specifically 4.0.0-1: 7e74e1af

@jakubgs
Copy link
Member Author

jakubgs commented Oct 12, 2020

According to TheHive-Project/TheHive#1105 just removing these lines form service definition should fix logs:

StandardOutput=null
StandardError=null

Which I did: 891304ed
And it works:

[email protected]:~ % s status thehive
● thehive.service - Scalable, Open Source and Free Security Incident Response Solutions
     Loaded: loaded (/lib/systemd/system/thehive.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2020-10-12 14:16:41 UTC; 1min 25s ago
       Docs: https://thehive-project.org
   Main PID: 160931 (java)
      Tasks: 57 (limit: 8939)
     Memory: 453.9M
     CGroup: /system.slice/thehive.service
             └─160931 java -Duser.dir=/opt/thehive4-4.0.0-1 -Dconfig.file=/data/thehive/conf/application.conf -Dlogger.file=/data/thehive/conf/logback.xml -Dpidfile.path=/dev/null -cp /opt/thehive4-4.0.0-1/lib/../conf/:/opt/thehive4-4.0.0-1/>

Oct 12 14:17:04 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] a.c.s.ClusterSingletonProxy [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
Oct 12 14:17:04 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] play.api.Play [|] Application started (Prod) (no global state)
Oct 12 14:17:05 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] a.c.s.ClusterSingletonProxy [|] Singleton identified at [akka://application/user/flowSingletonManager/singleton]
Oct 12 14:17:05 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] p.c.s.AkkaHttpServer [|] Listening for HTTP on /[0:0:0:0:0:0:0:0]:9000
Oct 12 14:17:07 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] o.t.s.AccessLogFilter [00000001|] 127.0.0.1 GET / took 18ms and returned 308 0 bytes
Oct 12 14:17:07 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] o.t.s.AccessLogFilter [00000002|] 127.0.0.1 GET /index.html took 120ms and returned 200 1191 bytes
Oct 12 14:17:25 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] o.t.s.AccessLogFilter [00000003|] 162.158.102.129 GET /api/status took 119ms and returned 200 286 bytes
Oct 12 14:17:37 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] o.t.s.AccessLogFilter [00000004|] 127.0.0.1 GET / took 1ms and returned 308 0 bytes
Oct 12 14:17:37 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] o.t.s.AccessLogFilter [00000005|] 127.0.0.1 GET /index.html took 3ms and returned 200 1191 bytes
Oct 12 14:17:49 master-01.gc-us-central1-a.thehive.prod thehive[160931]: [info] o.t.s.AccessLogFilter [00000006|] 162.158.75.203 GET /api/status took 2ms and returned 200 286 bytes

@jakubgs
Copy link
Member Author

jakubgs commented Oct 12, 2020

Looks like this needs the ES cluster to already have an index present:

[error] o.e.d.DBConfiguration [|] ElasticSearch request failure: POST:/cortex_4/_search?scroll=60000ms
StringEntity({"version":"true","query":{"bool":{"must":[{"term":{"relations":{"value":"worker"}}},{"match_all":{}}]}},"from":0,"sort":[{"_id":{"order":"desc"}}]},Some(application/json))
 => ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),None,List(ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),None,null,None)),None)
[warn] o.e.d.SearchWithScroll [|] Search error
org.elastic4play.IndexNotFoundException$: null
	at org.elastic4play.IndexNotFoundException$.<clinit>(Errors.scala)
	at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:145)
	at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)

@jakubgs
Copy link
Member Author

jakubgs commented Oct 12, 2020

I created the index and now I'm just getting some warnings:

[warn] o.e.c.RestClient [|] request [POST http://10.2.1.88:9200/cortex_4/_search?scroll=60000ms] returned 1 warnings:
[
  299
  Elasticsearch-7.8.0-757314695644ea9a1dc2fecd26d1a43856725e65
  "Loading the fielddata on the _id field is deprecated and will be removed in future versions. If you require sorting or aggregating on this field you should also include the id in the body of your documents, and map this field as a keyword field that has [doc_values] enabled"
]

And a search error that appears as a warning:

[warn] o.e.d.SearchWithScroll [|] Search error
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `long` out of START_OBJECT token
 at [Source: (String)"{"_scroll_id":"FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoARRNNERaSFhVQm0tNVVjeTBjaVZNeAAAAAAAAAACFlNnQW9PbGFLUWVDQk1WWkZpdEtDNEE=","took":70,"timed_out":false,"_shards":{"total":3,"successful":3,"skipped":2,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}"; line: 1, column: 242] (through reference chain: com.sksamuel.elastic4s.http.search.SearchResponse["hits"]->com.sksamuel.elastic4s.http.search.SearchHits["total"])
        at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
        at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)
        at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$LongDeserializer._parseLong(NumberDeserializers.java:593)
        at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$LongDeserializer.deserialize(NumberDeserializers.java:557)
        at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$LongDeserializer.deserialize(NumberDeserializers.java:535)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:530)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:528)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:417)

So I'm not sure if it's actually an error or a warning.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 12, 2020

And when trying to log in via WebUI I get:

[info] o.t.c.s.ErrorHandler [|] GET /api/user/current returned 500
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `long` out of START_OBJECT token
 at [Source: (String)"{"took":10,"timed_out":false,"_shards":{"total":3,"successful":3,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}"; line: 1, column: 106] (through reference chain: com.sksamuel.elastic4s.http.search.SearchResponse["hits"]->com.sksamuel.elastic4s.http.search.SearchHits["total"])
        at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
        at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)
        at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$LongDeserializer._parseLong(NumberDeserializers.java:593)
        at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$LongDeserializer.deserialize(NumberDeserializers.java:557)
        at com.fasterxml.jackson.databind.deser.std.NumberDeserializers$LongDeserializer.deserialize(NumberDeserializers.java:535)
        at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:530)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:528)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:417)

This service produces a lot of errors for a brand new install...

@jakubgs
Copy link
Member Author

jakubgs commented Oct 13, 2020

Opened an issue about lack of index: TheHive-Project/Cortex#305

@jakubgs
Copy link
Member Author

jakubgs commented Oct 13, 2020

Opened another issue about 500 errors: TheHive-Project/Cortex#306

@jakubgs
Copy link
Member Author

jakubgs commented Oct 13, 2020

Configured and Nginx proxy for Cortex: 9ef85cee

@jakubgs
Copy link
Member Author

jakubgs commented Oct 15, 2020

After a bunch of a research in TheHive-Project/Cortex#305 I realized that Cortex requires ElasticSearch 5.6.0:
https://github.com/TheHive-Project/CortexDocs/blob/master/installation/install-guide.md#1-pre-requisites

Which is ridiculous because its end of life was in 2019-03-11:
https://www.elastic.co/support/eol

@jakubgs
Copy link
Member Author

jakubgs commented Oct 15, 2020

They do have have an 3.1.0-RC1 release, which is of course misnamed as 3.0.1 Release...
https://github.com/TheHive-Project/Cortex/tree/3.1.0-RC1
Which does introduce support for ElasticSearch 7: TheHive-Project/Cortex#279

@jakubgs
Copy link
Member Author

jakubgs commented Oct 15, 2020

There appears to be no binary release of 3.1.0-RC1 so I'll have to build it.

This section of docs does say how to build the project:

git clone https://github.com/TheHive-Project/Cortex.git
cd Cortex
./sbt clean stage
sudo cp -r Cortex/target/universal/stage /opt/cortex

https://github.com/TheHive-Project/CortexDocs/blob/master/installation/install-guide.md#23-cortex

@jakubgs
Copy link
Member Author

jakubgs commented Oct 15, 2020

I built it using an ubuntu:20.04 image by first installing the following packages:

apt install git curl openjdk-8-jdk-headless nodejs npm zip

And here it is: https://status-misc.ams3.digitaloceanspaces.com/cortex-3.1.0-RC1.zip

@jakubgs
Copy link
Member Author

jakubgs commented Oct 22, 2020

Fixed logging configuration: 9b3a50d5

And upgraded to 3.1.0-RC1: 38a8ef79

@jakubgs
Copy link
Member Author

jakubgs commented Oct 22, 2020

But it's still failing with:

[ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-7
  - ElasticSearch request failure: POST:/cortex_4/_search?scroll=60000ms
    StringEntity(
      {
        "version":"true",
        "query":{
          "bool":{
            "must":[
              {"term":{"relations":{"value":"job"}}},
              {"term":{"status":{"value":"Waiting"}}}
            ]
          }
        },
        "from":0,
        "sort":[
          {"_doc":{"order":"desc"}}
        ]
      },
      Some(application/json)
    )
   => ElasticError(
        index_not_found_exception,
        no such index [cortex_4],Some(_na_),
        Some(cortex_4),
        None,
        List(ElasticError(index_not_found_exception,no such index [cortex_4],Some(_na_),Some(cortex_4),
        None,null,None,None,None,null)),None,None,None,null
    )

@jakubgs
Copy link
Member Author

jakubgs commented Oct 22, 2020

Okay, i created the index by hand and it starts now:

[info] o.t.c.s.DockerJobRunnerSrv - Docker is available:
Info{architecture=x86_64, clusterStore=, cgroupDriver=cgroupfs, containers=0, containersRunning=0, containersStopped=0, containersPaused=0, cpuCfsPeriod=true, cpuCfsQuota=true, debug=false, dockerRootDir=/var/lib/docker/100000.100000, storageDriver=overlay2, driverStatus=[[Backing Filesystem, extfs], [Supports d_type, true], [Native Overlay Diff, true]], executionDriver=null, experimentalBuild=false, httpProxy=, httpsProxy=, id=KNBK:S3L6:YYAG:KF54:24FK:543D:YLHM:DQAW:IBM6:TSXO:N5ZD:L77P, ipv4Forwarding=true, images=1, indexServerAddress=https://index.docker.io/v1/, initPath=null, initSha1=null, kernelMemory=true, kernelVersion=5.4.0-1025-gcp, labels=[], memTotal=7819628544, memoryLimit=true, cpus=2, eventsListener=0, fileDescriptors=22, goroutines=36, name=master-01.gc-us-central1-a.thehive.prod, noProxy=, oomKillDisable=true, operatingSystem=Ubuntu 20.04.1 LTS, osType=linux, plugins=Plugins{volumes=[local], networks=[bridge, host, ipvlan, macvlan, null, overlay]}, registryConfig=RegistryConfig{indexConfigs={docker.io=IndexConfig{name=docker.io, mirrors=[], secure=true, official=true}}, insecureRegistryCidrs=[127.0.0.0/8]}, serverVersion=19.03.13, swapLimit=false, swarm=SwarmInfo{cluster=null, controlAvailable=false, error=, localNodeState=inactive, nodeAddr=, nodeId=, nodes=null, managers=null, remoteManagers=null}, systemStatus=[], systemTime=Thu Oct 22 15:53:41 UTC 2020}
[info] o.t.c.s.JobRunnerSrv - The package cortexutils for python has valid version: 2.0.0
[warn] o.t.c.s.JobRunnerSrv - The package cortexutils for python2 hasn't been found
[info] o.t.c.s.JobRunnerSrv - The package cortexutils for python3 has valid version: 2.0.0
[info] play.api.Play - Application started (Prod) (no global state)
[info] p.c.s.AkkaHttpServer - Enabling HTTP/2 on Akka HTTP server...
[info] p.c.s.AkkaHttpServer - Listening for HTTP on /[0:0:0:0:0:0:0:0]:9001

But when I try to open the side I get:

[email protected]:~ % curl -s localhost:9001/index.html 
A client error occurred on GET /index.html : Resource not found by Assets controller

But logs show no error. So something is still wrong.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 22, 2020

For now I just added index creation to the role: 04699b5f

@jakubgs
Copy link
Member Author

jakubgs commented Oct 22, 2020 •
edited
Loading

But I'm not sure I was supposed to create the index,because I'm looking at it now and there's nothing in there.

cortex_4_index_empty

Maybe because I created it by hand no migrations ran, but then why didn't Cortex create it in the first place?

This software is so frustrating.

@jakubgs
Copy link
Member Author

jakubgs commented Oct 22, 2020

I had a conversation with Jerome Leonard from the project in their Gitter and I was told that:

but I think there is a problemb with the RC1
with the creation of the index

Which confirms my issues. He also said that:

be aware that we are about to release a stable version 3.1.0

And it's supposed to appear within days. So I think it would be sensible to take a break form this and wait for their stable release.

@jakubgs
Copy link
Member Author

jakubgs commented Nov 4, 2020

I tried 3.1.0 today and it did not create the index: TheHive-Project/Cortex#305 (comment)

@jakubgs
Copy link
Member Author

jakubgs commented Nov 13, 2020

Based on TheHive-Project/Cortex#305 (comment) comment I opened the site available at localhost:9001 to see the http://127.0.0.1:9001/index.html#!/maintenance page which greeted me with an Update Database button:
cortex_update_database
Which when clicked makes a POST request to http://localhost:9001/api/maintenance/migrate:
cortex_post_migrate
Which results in the index being created:

[email protected]:~ % curl -s localhost:9200/_cat/indices     
green open cortex_5   b62BDfKkTOSqiFh_yidepg 3 1 1 0 6.4kb  624b
green open .elastichq OtyN7XiTQDa2yAi1h6Awcw 1 1 1 0  13kb 6.5kb

I cannot for the life of me comprehend why this software doesn't run the migration by itself at startup, but it doesn't.

@jakubgs
Copy link
Member Author

jakubgs commented Nov 13, 2020

Applied a fix in: 6d25ba9c

@jakubgs
Copy link
Member Author

jakubgs commented Nov 13, 2020

Opened an issue about automated creation of superuser: TheHive-Project/Cortex#316

@jakubgs
Copy link
Member Author

jakubgs commented Nov 13, 2020

I've configured Cortex in f84f9c6a.
And connected The Hive to it in 98350319.

But I'm not entirely sure how to verify that it's working correctly.

@jakubgs
Copy link
Member Author

jakubgs commented Nov 13, 2020

There's this documentation for Cortex connector:
https://github.com/TheHive-Project/TheHiveDocs/tree/master/api/connectors/cortex
And you can query jobs:
https://github.com/TheHive-Project/TheHiveDocs/blob/master/api/connectors/cortex/job.md
Which seems to work:

 > curl -si localhost:9000/api/connector/cortex/job/_search -H "Authorization: Bearer $API_KEY"                        
HTTP/1.1 404 Not Found
Request-Time: 67
Date: Fri, 13 Nov 2020 17:13:55 GMT
Content-Type: application/json
Content-Length: 54

{"type":"NotFoundError","message":"RichJob not found"}

Based on the lack of error from the request and in the logs I assume it's working as intended.

[info] o.t.s.AccessLogFilter [0000009b|] 127.0.0.1 GET /api/connector/cortex/job/_search took 67ms and returned 404 54 bytes

@jakubgs jakubgs closed this as completed Nov 13, 2020
@jakubgs
Copy link
Member Author

jakubgs commented Nov 13, 2020

Also added OAuth for good measure: d01e6f73

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@jakubgs jakubgs

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jakubgs @corpetty