ES Mapping bug

[s4vgR]

Hi,

Request Type: Bug

Work Environment

Question	Answer
OS version (client)	CentOS 7 (Linux lab.centos7 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64)
TheHive version	thehive.noarch 3.0.10-1
Package Type	RPM

Problem Description

I installed The Hive and Cortex using RPM. When i go to local web of both Hive and Cortex I get the same error after clicking Update Database:

2018-08-20 17:37:40,950 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-5 - POST /api/maintenance/migrate returned 400
org.elasticsearch.transport.RemoteTransportException: [fQRBr16][127.0.0.1:9300][indices:admin/create]
Caused by: java.lang.IllegalArgumentException: Rejecting mapping update to [cortex_1] as the final mapping would have more than 1 type: [artifact, dblist, data, audit, analyzer, organization, report, job, user, analyzerConfig]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:408)
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:356)
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:280)
at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:443)
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630)
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267)
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197)
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132)
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:626)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Looks like the mapping removal in ES 6+ disabled updating and with that any usage of Hive and Cortex. Any ideas? Thank you for help.

Steps to Reproduce

1. Install latest CentOS 7
2. Install The Hive
3. Install Cortex
4. Visit corresponding web sites

[nadouani]

Hello, are you using ES6? TheHive and Cortex are only supporting ES 5.6 max.

[saadkadhi]

https://github.com/TheHive-Project/TheHiveDocs/blob/master/FAQ.md#do-you-support-elasticsearch-6x-or-later

[s4vgR]

Oh, thank you! Always the easiest solution...
Maybe it would be good to highlight it in the install docs

[s4vgR]

Tried with 5.6.10
Again the same issue!

[nadouani]

Oh, thank you! Always the easiest solution...
Maybe it would be good to highlight it in the install docs

Well, take a breath and read the docs: https://github.com/TheHive-Project/TheHiveDocs/blob/master/FAQ.md#do-you-support-elasticsearch-6x-or-later

We already highlighted the fact that ES 6 is not supported

[s4vgR]

In the FAQ, not under install guide.
Btw read my comment above.

[3c7]

Yeah, except for both examples given in the install guide.

Weird that you get the same issue using 5.6.x. Is there something strange with the ES install on CentOS?

[s4vgR]

I don't want to complain because this is a awesome project, but regarding ES in this guide:

https://github.com/TheHive-Project/TheHiveDocs/blob/master/installation/install-guide.md

Elasticsearch is mentioned 67 times. Only 2 times the version 5.x is mentioned: once under Binary (Minimal Ubuntu Installation) and once under "Build it Yourself".

Table of Contents:
Installation Options
RPM
DEB
Docker
-> Binary
-> Build it Yourself
Elasticsearch Installation
System Package
Start the Service
Elasticsearch inside a Docker

So, if I use RPM install and follow the guide, there is no information about ES version Hive requires.
Just my 5 cent contribution :)

[nadouani]

Well, fair enough, we will specify it under the ES install section.

That said, if you follow the instructions we’ve written, you should end up with the latest ES 5.x not with a 6.x version I think.

[s4vgR]

And "5. First start" should be a separate heading, not under Binary IMHO :)

[saadkadhi]

Guys,

Are we really going to go down this bean counter route?

@s4vgR please show us your willingness to help and submit a PR for the documentation with the modifications you see fit to clearly indicate that ES 5.x is required to prevent other users from installing ES 6+. Thanks in advance.

[s4vgR]

Np, I'll send a PR probably today.

Regarding my issue, this fixed it:

• reinstalling Hive
• restarting ES service
  There were no nodes set up, logically ES should be installed before Hive xD

It's not bean counting, if I install for RPM why should I bother to read install for Ubuntu or from binaries?