MISP Sync - Files limited to 32KB

[arnydo]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	Ubuntu (Docker)
OS version (client)	Windows 10, Ubuntu
TheHive version / git hash	2.13.1 / ?
Package Type	Docker
Browser type & version	Firefox Nightly

Problem Description

During a sync with MISP I am receiving an error stating that "...UTF8 encoding is longer than the max length 32766" as seen in the log below:

Steps to Reproduce

1. Create new MISP event with attribute/attachment greater than 32766 bytes
2. Attempt sync with TheHive

Possible Solutions

Looks similar to issue #237

Complementary information

2017-10-19 08:42:15,334 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-6826 - Update of MISP events is starting ...
2017-10-19 08:42:15,338 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-6826 - Synchronize MISP MISP-SERVER-ID from Some(Wed Oct 18 11:15:00 EDT 2017)
2017-10-19 08:42:15,392 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-6825 - getting MISP event MISP-SERVER-ID:120
2017-10-19 08:42:15,416 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-6832 - MISP event MISP-SERVER-ID:120 has no related alert, create it with 15 observable(s)
2017-10-19 08:42:15,432 [WARN] from org.elastic4play.database.DBCreate in application-akka.actor.default-dispatcher-6790 - create error
java.lang.IllegalArgumentException: Document contains at least one immense term in field="artifacts.data" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[66, 46, 32, 75, 121, 108, 101, 32, 80, 97, 114, 114, 105, 115, 104, 10, 83, 101, 114, 118, 101, 114, 32, 79, 112, 101, 114, 97, 116, 105]...', original message: bytes can be at most 32766 in length; got 33461

[nadouani]

Well, how many attributes does the MISP event have?

This is a limitation with Elasticsearch where a document cannot be bigger than 32KB, in our case, converting a MISP event to a TheHive alert produces a document bigger than that limit.

This is unfortunately unsolvable :(

[arnydo]

This particular event has 15 attributes. None of them being files.

[nadouani]

What type of attributes?

[nadouani]

Files are not stored on the alert, but on a datastore, and the alert includes only links to the datastore, so files are not an issue on alerts.

[arnydo]

Urls, domains and the body (text-only) of an email.

I removed the email-body attribute and the sync was successful.