#70 update and customized for Cortex only

TheHive-Project · Oct 24, 2020 · 52f1a19 · 52f1a19
1 parent e7f7850
commit 52f1a19
Show file tree

Hide file tree

Showing 2 changed files with 377 additions and 0 deletions.
diff --git a/admin/upgrade_to_cortex_3_1_and_es7_x.md b/admin/upgrade_to_cortex_3_1_and_es7_x.md
@@ -0,0 +1,317 @@
+# Migration from Elasticsearch 6.8.2 to ES 7.x
+
+---
+⚠️ IMPORTANT NOTE
+
+- This migration process is intended for single node of Elasticsearch database
+- The current version of this document is provided for testing purpose **ONLY!**  
+- This guide has been written and tested to migrate data from ES 6.8.2 to ES 7.8.1, and Cortex 3.0.1 to Cortex 3.1.0 **only!**
+- This guide starts with Elasticsearch version 6.8.2  up and running, indexes and data. To test this guide, we recommend using a backup of you production server. (see Backup and Restore page for more information)
+- This guide is illustrated with Cortex index. The process is identical for Cortex, you just have to adjust index names.
+---
+
+## Prerequisite
+
+The software `jq` is required to manipulate JSON and create new indexes. More information at [https://stedolan.github.io/jq/](). 
+
+## Identify if your index should be reindexed
+
+You can easily identify if indexes should be reindexed or not. On the index named `cortex_4` run the following command: 
+
+```
+curl -s http://127.0.0.1:9200/cortex_4?human | jq '.cortex_4.settings.index.version.created'
+```
+
+if the output is similar to `"5xxxxxx"`  then reindexing is required, you should follow this guide. 
+
+If it is   `"6xxxxxx"` then the index can be read by Elasticsearch 7.8.x. Upgrade Elasticsearch, and Cortex 3.1.0.
+
+## Migration guide
+
+### Current status
+
+Current context is: 
+- Elasticsearch 6.8.2
+- Cortex 3.0.1
+
+All up and running. 
+
+Start by identifying indices on you Elasticsearch instance.
+
+```
+curl  http://localhost:9200/_cat/indices\?v
+```
+
+The output should look like this: 
+
+```
+health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
+yellow open   cortex_4    1CGSt9R3Rlm0bc9E-qEkXg       5   1       8531            8       19mb            9mb 
+```
+
+
+The index name is `cortex_4`. Record this somewhere.
+
+### Stop services
+
+Before starting updating the database, lets stop applications:
+
+```
+sudo service cortex stop 
+```
+
+### Create a new index
+
+
+The First operation lies in creating a new index named `new_the_hive_15` with settings from current index `the_hive_15` (ensure to keep index version, needed for future upgrade).
+
+```bash
+curl -XPUT 'http://localhost:9200/new_cortex_4' \
+  -H 'Content-Type: application/json' \
+  -d "$(curl http://localhost:9200/cortex_4 |\
+   jq '.the_hive_15 |
+   del(.settings.index.provided_name,
+    .settings.index.creation_date,
+    .settings.index.uuid,
+    .settings.index.version,
+    .settings.index.mapping.single_type,
+    .mappings.doc._all)'
+    )"
+```
+
+
+Check the new index is well created: 
+
+```
+curl -XGET http://localhost:9200/_cat/indices\?v
+```
+
+The output should look like this: 
+
+```
+health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
+yellow open   new_the_hive_15 A2KLoZPpSXygutlfy_RNCQ   5   1          0            0      1.1kb          1.1kb
+green  open   the_hive_15     Oap-I61ySgyv6EAI1ZUTFQ   5   0      30977           36     33.2mb         33.2mb
+```
+
+### Proceed to Reindex 
+
+Next operation lies in running the reindex command in the newly created index:
+
+```bash
+curl -XPOST -H 'Content-Type: application/json' http://localhost:9200/_reindex -d '{
+  "conflicts": "proceed",
+  "source": {
+    "index": "cortex_4"
+  },
+  "dest": {
+    "index": "new_cortex_4"
+  }
+}'
+```
+
+After a moment, you should get a similar output:  
+
+```json
+{
+    "took": 5119,
+    "timed_out": false,
+    "total": 5889,
+    "updated": 0,
+    "created": 5889,
+    "deleted": 0,
+    "batches": 6,
+    "version_conflicts": 0,
+    "noops": 0,
+    "retries": {
+        "bulk": 0,
+        "search": 0
+    },
+    "throttled_millis": 0,
+    "requests_per_second": -1.0,
+    "throttled_until_millis": 0,
+    "failures": []
+}
+```
+
+### Ensure new index has been created
+
+Run the following command, and ensure the new index is like the current one (size can vary):
+
+```
+curl -XGET http://localhost:9200/_cat/indices\?v
+```
+
+The output should look like this: 
+
+```
+health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
+green  open   new_the_hive_15 GV-3Y8QjTjWw0F-p2sjW6Q   5   0      30977            0       26mb           26mb
+green  open   the_hive_15     Oap-I61ySgyv6EAI1ZUTFQ   5   0      30977           36     33.2mb         33.2mb
+```
+
+### Delete old indices
+
+This is the thrilling part. 
+Now the new index `new_cortex_4` is created and similar to `cortex_4`,  older indexes **should be completely deleted** from the database. To delete index named `cortex_4`, run the following command:  
+
+```bash
+curl -XDELETE http://localhost:9200/cortex_4
+```
+
+Run the same command for older indexes if exist (cortex_3, cortex_2....). Elasticsearch 7.x cannot run with index created with Elasticsearch 5.x.
+
+### Create an alias 
+
+Before stopping Elasticsearch service, let’s create an alias to keep index names in the future.  
+
+```bash
+curl -XPOST -H 'Content-Type: application/json'  'http://localhost:9200/_aliases' -d '{
+    "actions": [
+        {
+            "add": {
+                "index": "new_cortex_4",
+                "alias": "cortex_4"
+            }
+        }
+    ]
+}'
+```
+
+
+Doing so will allow Cortex 3.1.0  to find the index without updating the configuration file. 
+
+Check the alias has been well created by running the following command
+
+```bash
+curl -XGET http://localhost:9200/_alias?pretty
+```
+
+The output should look like:
+
+```json
+{
+  "new_cortex_4" : {
+    "aliases" : {
+      "cortex_4" : { }
+    }
+  }
+}
+```
+
+
+## Stop Elasticsearch version 6.8.2
+
+```bash
+sudo service elasticsearch stop 
+```
+
+
+## Update Elasticsearch 
+
+Update the configuration of Elastisearch. Configuration file should look like this:
+
+```
+[..]
+http.host: 127.0.0.1
+discovery.type: single-node
+cluster.name: hive
+script.allowed_types: inline
+thread_pool.search.queue_size: 100000
+thread_pool.write.queue_size: 10000    
+```
+
+Now, upgrade Elasticsearch to version 7.x following the documentation for your Operating System, and ensure the service start successfully.
+
+## Install or update to TheHive 3.5.0
+
+### DEB package
+
+If using Debian based Linux operating system, configure it to follow our beta repository:
+
+```bash
+curl https://raw.githubusercontent.com/TheHive-Project/TheHive/master/PGP-PUBLIC-KEY | sudo apt-key add -
+echo 'deb https://deb.thehive-project.org stable main es7' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list
+sudo apt-get update
+```
+Then install it by running:
+
+```bash
+sudo apt install cortex
+```
+
+or
+
+```bash
+sudo apt install cortex=3.1.0-1
+```
+
+### RPM
+
+Setup your system to connect the RPM repository. Create and edit the file  `/etc/yum.repos.d/thehive-project.repo` :
+
+```
+[thehive-project]
+enabled=1
+priority=1
+name=TheHive-Project RPM repository
+baseurl=http://rpm.thehive-project.org/stable/es7
+gpgcheck=1
+```
+
+Then install it by running:
+
+```bash
+sudo yum install cortex
+```
+
+or 
+
+```bash
+sudo yum install cortex-3.1.0-1
+```
+
+### Install binaries
+
+```bash
+cd /opt
+wget https://download.thehive-project.org/cortex-3.1.0-1.zip
+unzip cortex-3.1.0-1.zip
+ln -s cortex-3.1.0-1 cortex
+```
+
+### Docker images
+
+Docker images are also provided on Dockerhub. 
+
+```bash
+docker pull thehiveproject/cortex:3.1.0-1
+```
+
+
+⚠️  Starting from this version, docker image doesn't contain analyzers anymore. _Analyzers__/__Responders_ and Cortex have different life-cycles, their update including their dependencies should not be correlated to Cortex update. 
+
+It is recommended to use docker version of analyzers : this can be done by binding docker service docket inside cortex container (run with `-v /var/run/docker.sock:/var/run/docker.sock`).
+
+
+### Update Database
+
+Connect to TheHive (and Cortex), the maintenance page should ask to update. 
+
+![](/images/thehive-first-access_screenshot.png)
+
+Once updated, ensure a new index named `cortex_5` has been created.
+
+
+```bash
+curl -XGET http://localhost:9200/_cat/indices\?v
+```
+
+The output should look like this: 
+
+```
+health status index           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
+green  open   new_the_hive_15 GV-3Y8QjTjWw0F-p2sjW6Q   5   0      30977            0       26mb           26mb
+yellow open   the_hive_16     Nz0vCKqhRK2xkx1t_WF-0g   5   1      30977            0     26.1mb         26.1mb
+```
+
diff --git a/migration_guide.md b/migration_guide.md
@@ -1,5 +1,65 @@
 # Migration Guide
 
+## From 3.0.x to 3.1.0
+
+Taking into account the EoL of version 6.x. of Elasticsearch, Cortex 3.1.0 is the first version to support Elasticsearch 7.x. This version introduce breaking changes.  This time,  we had no choice, we were not able to make TheHive support smoothly the ES upgrade. 
+
+Cortex 3.1.0 supports Elasticsearch 7.x **ONLY**.
+
+This first steps before starting the upgrade process are: 
+
+- Identify the version of Elasticsearch which created your index
+- Stop Cortex service
+- Stop Elasticsearch service
+
+### How to identify the version of Elasticsearch which created your database index ?
+
+---
+
+The software `jq` is required to manipulate JSON and create new indexes. More information at [https://stedolan.github.io/jq/](). 
+
+---
+
+Run the following command : 
+
+```bash
+curl -s http://127.0.0.1:9200/cortex_4?human | jq '.cortex_4.settings.index.version.created'
+```
+
+- if the output is similar to `"5xxxxxx"`  then your database index has been created with Elasticsearch 5.x  reindexing is required, you should follow [a dedicated process to upgrade](../admin/upgrade_to_thehive_3_5_and_es_7_x.md). 
+- If it is   `"6xxxxxx"` then your database has been created with Elasticsearch 6.
+
+### Your database was created with Elasticsearch 5.x or earlier
+
+This is where things might be complicated. This upgrade progress  requires handling the database index by updating parameters, and reindex before updating Elasticsearch, and updating TheHive.
+
+Read carefully [the dedicated documentation](../admin/upgrade_to_cortex_3_1_and_es7_x). It should help you run this specific actions on your Elasticsearch database, and also install or update application whether you are using DEB, RPM or binary packages, and even docker images.
+
+### Your database was created with Elasticsearch 6.x
+
+If you started using TheHive with Elasticsearch 6.x, then you just need to update the configuration of Elasticsearch to reflect this one: 
+
+```
+[..]
+http.host: 127.0.0.1
+discovery.type: single-node
+cluster.name: hive
+script.allowed_types: inline
+thread_pool.search.queue_size: 100000
+thread_pool.write.queue_size: 10000
+```
+
+Following parameters are **not accepted anymore** by Elasticsearch 7: 
+
+- `thread_pool.index.queue_size`
+- `thread_pool.bulk.queue_size` 
+
+With TheHive service stopped, ensure the new version of Elasticsearch starts.
+
+If everything is ok, then Cortex 3.1.0 can be installed. To run this operation successfully, you need to **update your repository configuration**  if you are using DEB and RPM packages, or specify the right version to install if using docker. Read carefully the [installation guide](../installation/install-guide.md). 
+
+
+
 ## From 2.x to 3.x
 
 ### Elasticsearch Changes