[Bug] Observables not present in some events imported from MISP

[Tyrell20]

Request Type

Bug

Work Environment

Question	Answer
OS version (server)	RedHat 7.9
TheHive version / git hash	Version: 4.0.5-1
Package Type	RPM

Problem Description

For some MISP events, TheHive fails to import the related observables.

Steps to Reproduce

1. Publish an event on MISP that contains a "malware-sample" object but without reference to the attachment file;
2. Wait for MISP sync on TheHive;
3. The alert is imported and visible on TheHive but it does not contain any observable, even if present on MISP.

Possible Solutions

The problem seems to be only when the MISP event that TheHive try to import contains a "malware-sample" without the related attachment file.
TheHive goes into error, without continuing to import the other observables.
As a possible solution, TheHive could try to retrieve the "malware-sample" and the related attachment and, if it fails, it could proceed to import the other observables.

Complementary information

On application.log, I have the following errors regarding the ID of the event for which TheHive fails to import observables:

2021-03-05 16:08:49,325 [WARN] from play.api.libs.ws.ahc.StandaloneAhcWSClient$ in application-akka.actor.default-dispatcher-12 - blockingToByteString is a $
2021-03-05 16:08:49,347 [ERROR] from org.thp.thehive.connector.misp.services.QueueIterator in application-akka.actor.default-dispatcher-4 - Stream fails
org.thp.client.ApplicationError: ApplicationError(404):
{
  "name" : "File '/var/www/MISP/app/files/33805/10034471' does not exists.",
  "message" : "File '/var/www/MISP/app/files/33805/10034471' does not exists.",
  "url" : "/attributes/download/10034471"
}
        at org.thp.client.ApplicationError$.apply(BaseClient.scala:14)
        at org.thp.misp.client.MispClient.$anonfun$downloadAttachment$1(MispClient.scala:231)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at org.thp.scalligraph.ContextPropagatingDisptacher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:56)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.thp.scalligraph.DiagnosticContext$.$anonfun$withDiagnosticContext$2(ContextPropagatingDisptacher.scala:91)
        at org.thp.scalligraph.DiagnosticContext$.saveDiagnosticContext(ContextPropagatingDisptacher.scala:106)
        at org.thp.scalligraph.DiagnosticContext$.withDiagnosticContext(ContextPropagatingDisptacher.scala:89)
        at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:74)
        at org.thp.scalligraph.ContextPropagatingDisptacher$$anon$1.$anonfun$execute$1(ContextPropagatingDisptacher.scala:56)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
       at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2021-03-05 16:08:49,348 [INFO] from org.thp.thehive.connector.misp.services.MispImportSrv in application-akka.actor.default-dispatcher-4 - Removing old obse$
2021-03-05 16:08:49,348 [WARN] from play.api.libs.ws.ahc.StandaloneAhcWSClient$ in application-akka.actor.default-dispatcher-12 - blockingToByteString is a $
2021-03-05 16:08:49,522 [WARN] from play.api.libs.ws.ahc.StandaloneAhcWSClient$ in application-akka.actor.default-dispatcher-10 - blockingToByteString is a $
2021-03-05 16:08:49,522 [WARN] from play.api.libs.ws.ahc.StandaloneAhcWSClient$ in application-akka.actor.default-dispatcher-31 - blockingToByteString is a $
2021-03-05 16:08:49,524 [ERROR] from org.thp.thehive.connector.misp.services.QueueIterator in application-akka.actor.default-dispatcher-4 - Stream fails

Instead, when the MISP event imported does not contain any malware-sample object it is imported correctly, with all observables and there aren't errors on the log.

[nadouani]

@To-om Added in 4.1 milestone for investigation