Skip to content

Commit

Permalink
[docs] Adapt rollover.md documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
richard-julien committed Sep 7, 2024
1 parent 66ba549 commit 4b7e117
Show file tree
Hide file tree
Showing 5 changed files with 9 additions and 120 deletions.
Binary file removed docs/deployment/assets/add-policy.png
Binary file not shown.
Binary file removed docs/deployment/assets/ilm-policy.png
Binary file not shown.
Binary file removed docs/deployment/assets/index-template.png
Binary file not shown.
Binary file removed docs/deployment/assets/templates.png
Binary file not shown.
129 changes: 9 additions & 120 deletions docs/deployment/rollover.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,5 @@
# Indices and rollover policies

!!! note "Default rollover policies"

Since OpenCTI 5.9.0, rollover policies are automatically created when the platform is initialized for the first time. If your platform has been initialized using an older version of OpenCTI or if you would like to understand (and customize) rollover policies please read the following documentation.

## Introduction

ElasticSearch and OpenSearch both support rollover on indices. OpenCTI has been designed to be able to use aliases for indices and so supports index lifecycle policies very well. Thus, by default OpenCTI initializes indices with a suffix of `-00001` and uses wildcards to query indices. When rollover policies are implemented (default starting OCTI 5.9.X if you initialized your platform at this version), indices are splitted to keep a reasonable volume of data in shards.
Expand Down Expand Up @@ -64,129 +60,22 @@ For your information, the indices which can grow rapidly are:

### Default implemented lifecycle policy

Here is the recommended policy (initialized starting 5.9.X):
Here is the recommended policy (for 1 shard per index) (initialized starting 5.9.X):

* Maximum primary shard size: `50 GB`
* Maximum age: `365 days`
* Maximum documents: `75,000,000`

### Applying rollover policies on existing indices
## Adapt platform initialized before the 6+ version

!!! warning "Procedure information"

Please read the following only if your platform has been initialized before 5.9.0, otherwise lifecycle policies has been created (but you can still cutomize them).

Unfortunately, to be able to implement rollover policies on ElasticSearch / OpenSearch indices, it will be needed to re-index all the data in new indices using ElasticSearch capabilities.

#### Shutdown

First step is to shutdown your OpenCTI platform.

#### Change configuration

Then, in the [OpenCTI configuration](configuration.md), change the ElasticSearch / OpenSearch default prefix to `octi` (default is `opencti`).

#### Create the rollover policy

Create a rollover policy named `octi-ilm-policy` (in Kibana, `Management > Index Lifecycle Policies`):

* Maximum primary shard size: `50 GB`
* Maximum age: `365 days`
* Maximum documents: `75,000,000`

![ILM Policy](assets/ilm-policy.png)

#### Create index templates

In Kibana, clone the `opencti-index-template` to have one index template by OpenCTI index with the appropriate rollover policy, index pattern and rollover alias (in Kibana, `Management > Index Management > Index Templates`).

![Index template](assets/index-template.png)

Create the following index templates:

* `octi_deleted_objects`
* `octi_files`
* `octi_history`
* `octi_inferred_entities`
* `octi_inferred_relationships`
* `octi_internal_objects`
* `octi_internal_relationships`
* `octi_stix_core_relationships`
* `octi_stix_cyber_observable_relationships`
* `octi_stix_cyber_observables`
* `octi_stix_domain_objects`
* `octi_stix_meta_objects`
* `octi_stix_meta_relationships`
* `octi_stix_sighting_relationships`
If your platform has been initialized before 5.9.0, your platform will be not configured to automatically managed the indices

Here is the overview of all templates (you should have something with `octi_` instead of `opencti_`).
Unfortunately, to be able to implement rollover policies on ElasticSearch / OpenSearch indices, it will be needed to:
- Upgrade your platform to the latest version
- Check that policy and templates are available after the migration
- split all the shards > 50Gb in new indices using ElasticSearch capabilities.
- Reconfigure the writing alias to the latest index and ensure that the policy is running correctly.

![Index templates](assets/templates.png)
If you need any help for this migration, please join the slack community.

#### Apply rollover policy on all index templates

Then, going back in the index lifecycle policies screen, you can click on the "+" button of the `octi-ilm-policy` to `Add the policy to index template`, then add the policy to add previously created template with the proper "Alias for rollover index".

![Add policy to template](assets/add-policy.png)

#### Bootstrap all new indices

Before we can re-index, we need to create the new indices with aliases.

```
PUT octi_history-000001
{
"aliases": {
"octi_history": {
"is_write_index": true
}
}
}
```

Repeat this step for all indices:

* `octi_deleted_objects`
* `octi_files`
* `octi_history`
* `octi_inferred_entities`
* `octi_inferred_relationships`
* `octi_internal_objects`
* `octi_internal_relationships`
* `octi_stix_core_relationships`
* `octi_stix_cyber_observable_relationships`
* `octi_stix_cyber_observables`
* `octi_stix_domain_objects`
* `octi_stix_meta_objects`
* `octi_stix_meta_relationships`

#### Re-index all indices

Using the `reindex` API, re-index all indices one by one:

```
curl -X POST "localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d'
{
"source": {
"index": "opencti_history-000001"
},
"dest": {
"index": "octi_history"
}
}
'
```

You will see the rollover policy to be applied and the new indices are automatically rolled-over during re-indexation.

#### Delete all old indices

Then just delete all indices with the prefix `opencti_`.

#### Start your platform

Start your platform, using the new indices.

!!! note "Rollover documentation"

To have more details about automatic rollover and lifecycle policies, please read the [official ElasticSearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-rollover.html).

0 comments on commit 4b7e117

Please sign in to comment.