Merge branch 'release/3.1.0'

TheHive-Project · Sep 25, 2018 · a19feb3 · a19feb3
2 parents 7f8c812 + f53bee5
commit a19feb3
Show file tree

Hide file tree

Showing 28 changed files with 1,803 additions and 1,697 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,27 +1,57 @@
 # Change Log
 
+## [3.1.0](https://github.com/TheHive-Project/TheHive/tree/3.1.0) (2018-09-25)
+[Full Changelog](https://github.com/TheHive-Project/TheHive/compare/3.1.0-RC3...3.1.0)
+
+**Implemented enhancements:**
+
+- Add MarkAlertAsRead action to responders [\#729](https://github.com/TheHive-Project/TheHive/issues/729)
+- AddCustomField responder operation [\#724](https://github.com/TheHive-Project/TheHive/issues/724)
+- 3.1.0RC3: Browsing to negative case ids is possible [\#713](https://github.com/TheHive-Project/TheHive/issues/713)
+
+**Fixed bugs:**
+
+- TheHive Hyperlinking  [\#723](https://github.com/TheHive-Project/TheHive/issues/723)
+- Multiple responder actions does not seem to be handled [\#722](https://github.com/TheHive-Project/TheHive/issues/722)
+- API allows alert creation with duplicate artifacts [\#720](https://github.com/TheHive-Project/TheHive/issues/720)
+- 3.0.1RC3: certificate based authentication failes as attributes are not correctly lowercased [\#714](https://github.com/TheHive-Project/TheHive/issues/714)
+- Fix PAP labels [\#711](https://github.com/TheHive-Project/TheHive/issues/711)
+- Observables not being displayed [\#655](https://github.com/TheHive-Project/TheHive/issues/655)
+
+**Closed issues:**
+
+- TheHive:Alerts don't send observables to Responders [\#725](https://github.com/TheHive-Project/TheHive/issues/725)
+- Cortex Connector [\#721](https://github.com/TheHive-Project/TheHive/issues/721)
+- Markdown syntex not rendered correctly [\#718](https://github.com/TheHive-Project/TheHive/issues/718)
+- 3.1.0RC3: Search produces errors on screen [\#712](https://github.com/TheHive-Project/TheHive/issues/712)
+
+**Merged pull requests:**
+
+- CloseTask responder operation [\#728](https://github.com/TheHive-Project/TheHive/pull/728) ([srilumpa](https://github.com/srilumpa))
+- Add AddTagToArtifact action to responders [\#717](https://github.com/TheHive-Project/TheHive/pull/717) ([srilumpa](https://github.com/srilumpa))
+
 ## [3.1.0-RC3](https://github.com/TheHive-Project/TheHive/tree/3.1.0-RC3) (2018-09-06)
 [Full Changelog](https://github.com/TheHive-Project/TheHive/compare/3.1.0-RC2...3.1.0-RC3)
 
 **Implemented enhancements:**
 
+- Extend Case Description Field [\#81](https://github.com/TheHive-Project/TheHive/issues/81)
 - Display task description via a collapsible row [\#709](https://github.com/TheHive-Project/TheHive/issues/709)
 - Allow task group auto complete in case template admin section [\#707](https://github.com/TheHive-Project/TheHive/issues/707)
 - Display task group in global task lists [\#705](https://github.com/TheHive-Project/TheHive/issues/705)
 - Make task group input optional [\#696](https://github.com/TheHive-Project/TheHive/issues/696)
 - Related Cases: See \(x\) more links [\#690](https://github.com/TheHive-Project/TheHive/issues/690)
 - Search section: Search for a string over all types of objects [\#689](https://github.com/TheHive-Project/TheHive/issues/689)
 - Filter on computedHandlingDuration in SearchDialog fails [\#688](https://github.com/TheHive-Project/TheHive/issues/688)
-- Extend Case Description Field [\#81](https://github.com/TheHive-Project/TheHive/issues/81)
 - Change layout of observable creation form [\#706](https://github.com/TheHive-Project/TheHive/pull/706) ([srilumpa](https://github.com/srilumpa))
 
 **Fixed bugs:**
 
+- Adding new observables to an alert retrospectively is impossible [\#511](https://github.com/TheHive-Project/TheHive/issues/511)
 - .sbt build of current git version fails with x-pack-transport error [\#710](https://github.com/TheHive-Project/TheHive/issues/710)
 - PKI authentication fails if user name in certificate has the wrong case [\#700](https://github.com/TheHive-Project/TheHive/issues/700)
 - Error handling deletion and re creation of file observables [\#699](https://github.com/TheHive-Project/TheHive/issues/699)
 - Start waiting tasks when adding task logs [\#695](https://github.com/TheHive-Project/TheHive/issues/695)
-- Adding new observables to an alert retrospectively is impossible [\#511](https://github.com/TheHive-Project/TheHive/issues/511)
 
 ## [3.1.0-RC2](https://github.com/TheHive-Project/TheHive/tree/3.1.0-RC2) (2018-08-27)
 [Full Changelog](https://github.com/TheHive-Project/TheHive/compare/3.1.0-RC1...3.1.0-RC2)
@@ -35,20 +65,21 @@
 - TheHive 3.1RC1: Slow reaction if Cortex is \(unclear\) unreachable [\#664](https://github.com/TheHive-Project/TheHive/issues/664)
 - TheHive 3.1RC1: Add status to cases and tasks in new search page [\#663](https://github.com/TheHive-Project/TheHive/issues/663)
 - TheHive 3.1RC1: Add Username that executes an active response to json data field of responder [\#662](https://github.com/TheHive-Project/TheHive/issues/662)
-- Application.conf needs clarifications [\#606](https://github.com/TheHive-Project/TheHive/issues/606)
 - Ability to set custom fields as mandatory [\#652](https://github.com/TheHive-Project/TheHive/issues/652)
+- Application.conf needs clarifications [\#606](https://github.com/TheHive-Project/TheHive/issues/606)
 - Observable type boxes doesn't line break on alert preview pane [\#593](https://github.com/TheHive-Project/TheHive/issues/593)
 -  On branch betterDescriptions [\#660](https://github.com/TheHive-Project/TheHive/pull/660) ([secdecompiled](https://github.com/secdecompiled))
 
 **Fixed bugs:**
 
+- The hive docker image has no latest tag [\#670](https://github.com/TheHive-Project/TheHive/issues/670)
+- case metrics unordered in cases [\#419](https://github.com/TheHive-Project/TheHive/issues/419)
 - 3.1.0-RC1- Tasks list is limited to 10 items. [\#679](https://github.com/TheHive-Project/TheHive/issues/679)
 - WebUI inaccessible after upgrading to 3.1.0-0-RC1 \(elastic4play and Play exceptions\) [\#674](https://github.com/TheHive-Project/TheHive/issues/674)
 - play.crypto.secret is depecrated [\#671](https://github.com/TheHive-Project/TheHive/issues/671)
-- The hive docker image has no latest tag [\#670](https://github.com/TheHive-Project/TheHive/issues/670)
 - 'Tagged as' displayed in Related Cases even if cases are untagged [\#594](https://github.com/TheHive-Project/TheHive/issues/594)
 - Horizontal Scrolling and Word-Wrap options for Logs [\#573](https://github.com/TheHive-Project/TheHive/issues/573)
-- case metrics unordered in cases [\#419](https://github.com/TheHive-Project/TheHive/issues/419)
+- Dashboard visualizations do not work with custom fields [\#478](https://github.com/TheHive-Project/TheHive/issues/478)
 
 **Closed issues:**
 
@@ -58,9 +89,9 @@
 
 **Merged pull requests:**
 
-- Move input group addons from right to left for better usage [\#672](https://github.com/TheHive-Project/TheHive/pull/672) ([srilumpa](https://github.com/srilumpa))
 - Update Cortex reference.conf [\#668](https://github.com/TheHive-Project/TheHive/pull/668) ([ErnHem](https://github.com/ErnHem))
 - Fix some minor typos [\#658](https://github.com/TheHive-Project/TheHive/pull/658) ([srilumpa](https://github.com/srilumpa))
+- Move input group addons from right to left for better usage [\#672](https://github.com/TheHive-Project/TheHive/pull/672) ([srilumpa](https://github.com/srilumpa))
 
 ## [3.1.0-RC1](https://github.com/TheHive-Project/TheHive/tree/3.1.0-RC1) (2018-07-31)
 [Full Changelog](https://github.com/TheHive-Project/TheHive/compare/3.0.10...3.1.0-RC1)

diff --git a/project/Common.scala b/project/Common.scala
@@ -9,6 +9,7 @@ object Common {
     licenses += "AGPL-V3" → url("https://www.gnu.org/licenses/agpl-3.0.html"),
     organizationHomepage := Some(url("http://thehive-project.org/")),
     resolvers += Resolver.bintrayRepo("thehive-project", "maven"),
+    resolvers += "elasticsearch-releases" at "https://artifacts.elastic.co/maven",
     scalaVersion := Dependencies.scalaVersion,
     scalacOptions ++= Seq(
       "-deprecation", // Emit warning and location for usages of deprecated APIs.

diff --git a/project/Dependencies.scala b/project/Dependencies.scala
@@ -20,7 +20,7 @@ object Dependencies {
 
     val reflections = "org.reflections" % "reflections" % "0.9.11"
     val zip4j = "net.lingala.zip4j" % "zip4j" % "1.3.2"
-    val elastic4play = "org.thehive-project" %% "elastic4play" % "1.6.1"
+    val elastic4play = "org.thehive-project" %% "elastic4play" % "1.6.2"
     val akkaCluster = "com.typesafe.akka" %% "akka-cluster" % "2.5.11"
     val akkaClusterTools = "com.typesafe.akka" %% "akka-cluster-tools" % "2.5.11"
   }

diff --git a/thehive-backend/app/services/AlertSrv.scala b/thehive-backend/app/services/AlertSrv.scala
@@ -1,8 +1,8 @@
 package services
 
 import java.nio.file.Files
-import javax.inject.{ Inject, Singleton }
 
+import javax.inject.{ Inject, Singleton }
 import scala.collection.immutable
 import scala.concurrent.{ ExecutionContext, Future }
 import scala.util.matching.Regex
@@ -23,6 +23,7 @@ import org.elastic4play.database.ModifyConfig
 import org.elastic4play.services.JsonFormat.attachmentFormat
 import org.elastic4play.services.QueryDSL.{ groupByField, parent, selectCount, withId }
 import org.elastic4play.services._
+import org.elastic4play.utils.Collection
 
 trait AlertTransformer {
   def createCase(alert: Alert, customCaseTemplate: Option[String])(implicit authContext: AuthContext): Future[Case]
@@ -103,7 +104,14 @@ class AlertSrv(
         case a ⇒ Future.successful(a)
       }
     artifactsFields.flatMap { af ⇒
-      createSrv[AlertModel, Alert](alertModel, fields.set("artifacts", JsArray(af)))
+      /* remove duplicate artifacts */
+      val distinctArtifacts = Collection.distinctBy(af) { a ⇒
+        val data = (a \ "data").asOpt[String]
+        val attachment = (a \ "attachment" \ "id").asOpt[String]
+        val dataType = (a \ "dataType").asOpt[String]
+        data.orElse(attachment).map(_ -> dataType).getOrElse(a)
+      }
+      createSrv[AlertModel, Alert](alertModel, fields.set("artifacts", JsArray(distinctArtifacts)))
     }
   }
 

diff --git a/thehive-cortex/app/connectors/cortex/services/ActionOperation.scala b/thehive-cortex/app/connectors/cortex/services/ActionOperation.scala
@@ -2,20 +2,22 @@ package connectors.cortex.services
 
 import scala.concurrent.{ ExecutionContext, Future }
 
+import play.api.Logger
 import play.api.libs.json._
 
 import akka.actor.ActorSystem
 import akka.stream.Materializer
 import akka.stream.scaladsl.Sink
-import javax.inject.{ Inject, Singleton }
-import models.{ Alert, Case }
-import services.CaseSrv
+import javax.inject.{ Inject, Provider, Singleton }
+import models._
+import org.elasticsearch.index.engine.VersionConflictEngineException
+import services.{ AlertSrv, ArtifactSrv, CaseSrv, TaskSrv }
 
 import org.elastic4play.controllers.Fields
 import org.elastic4play.database.ModifyConfig
 import org.elastic4play.models.{ BaseEntity, ChildModelDef, HiveEnumeration }
 import org.elastic4play.services.{ AuthContext, FindSrv }
-import org.elastic4play.utils.RetryOnError
+import org.elastic4play.utils.Retry
 import org.elastic4play.{ BadRequestError, InternalError }
 
 object ActionOperationStatus extends Enumeration with HiveEnumeration {
@@ -34,34 +36,72 @@ case class AddTagToCase(tag: String, status: ActionOperationStatus.Type = Action
   def updateStatus(newStatus: ActionOperationStatus.Type, newMessage: String): AddTagToCase = copy(status = newStatus, message = newMessage)
 }
 
+case class AddTagToArtifact(tag: String, status: ActionOperationStatus.Type = ActionOperationStatus.Waiting, message: String = "") extends ActionOperation {
+  def updateStatus(newStatus: ActionOperationStatus.Type, newMessage: String): AddTagToArtifact = copy(status = newStatus, message = newMessage)
+}
+
 case class CreateTask(fields: JsObject, status: ActionOperationStatus.Type = ActionOperationStatus.Waiting, message: String = "") extends ActionOperation {
   def updateStatus(newStatus: ActionOperationStatus.Type, newMessage: String): CreateTask = copy(status = newStatus, message = newMessage)
 }
 
+case class AddCustomFields(name: String, tpe: String, value: JsValue, status: ActionOperationStatus.Type = ActionOperationStatus.Waiting, message: String = "") extends ActionOperation {
+  override def updateStatus(newStatus: ActionOperationStatus.Type, newMessage: String): AddCustomFields = copy(status = newStatus, message = newMessage)
+}
+
+case class CloseTask(status: ActionOperationStatus.Type = ActionOperationStatus.Waiting, message: String = "") extends ActionOperation {
+  def updateStatus(newStatus: ActionOperationStatus.Type, newMessage: String): CloseTask = copy(status = newStatus, message = newMessage)
+}
+
+case class MarkAlertAsRead(status: ActionOperationStatus.Type = ActionOperationStatus.Waiting, message: String = "") extends ActionOperation {
+  def updateStatus(newStatus: ActionOperationStatus.Type, newMessage: String): MarkAlertAsRead = copy(status = newStatus, message = newMessage)
+}
+
 object ActionOperation {
   val addTagToCaseWrites = Json.writes[AddTagToCase]
+  val addTagToArtifactWrites = Json.writes[AddTagToArtifact]
   val createTaskWrites = Json.writes[CreateTask]
+  val addCustomFieldsWrites = Json.writes[AddCustomFields]
+  val closeTaskWrites = Json.writes[CloseTask]
+  val markAlertAsReadWrites = Json.writes[MarkAlertAsRead]
   implicit val actionOperationReads: Reads[ActionOperation] = Reads[ActionOperation](json ⇒
     (json \ "type").asOpt[String].fold[JsResult[ActionOperation]](JsError("type is missing in action operation")) {
-      case "AddTagToCase" ⇒ (json \ "tag").validate[String].map(tag ⇒ AddTagToCase(tag))
-      case "CreateTask"   ⇒ JsSuccess(CreateTask(json.as[JsObject] - "type"))
-      case other          ⇒ JsError(s"Unknown operation $other")
+      case "AddTagToCase"     ⇒ (json \ "tag").validate[String].map(tag ⇒ AddTagToCase(tag))
+      case "AddTagToArtifact" ⇒ (json \ "tag").validate[String].map(tag ⇒ AddTagToArtifact(tag))
+      case "CreateTask"       ⇒ JsSuccess(CreateTask(json.as[JsObject] - "type"))
+      case "AddCustomFields" ⇒ for {
+        name ← (json \ "name").validate[String]
+        tpe ← (json \ "tpe").validate[String]
+        value ← (json \ "value").validate[JsValue]
+      } yield AddCustomFields(name, tpe, value)
+      case "CloseTask"       ⇒ JsSuccess(CloseTask())
+      case "MarkAlertAsRead" ⇒ JsSuccess(MarkAlertAsRead())
+      case other             ⇒ JsError(s"Unknown operation $other")
     })
   implicit val actionOperationWrites: Writes[ActionOperation] = Writes[ActionOperation] {
-    case a: AddTagToCase ⇒ addTagToCaseWrites.writes(a)
-    case a: CreateTask   ⇒ createTaskWrites.writes(a)
-    case a               ⇒ Json.obj("unsupported operation" → a.toString)
+    case a: AddTagToCase     ⇒ addTagToCaseWrites.writes(a)
+    case a: AddTagToArtifact ⇒ addTagToArtifactWrites.writes(a)
+    case a: CreateTask       ⇒ createTaskWrites.writes(a)
+    case a: AddCustomFields  ⇒ addCustomFieldsWrites.writes(a)
+    case a: CloseTask        ⇒ closeTaskWrites.writes(a)
+    case a: MarkAlertAsRead  ⇒ markAlertAsReadWrites.writes(a)
+    case a                   ⇒ Json.obj("unsupported operation" → a.toString)
   }
 }
 
 @Singleton
 class ActionOperationSrv @Inject() (
     caseSrv: CaseSrv,
+    taskSrv: TaskSrv,
+    alertSrvProvider: Provider[AlertSrv],
     findSrv: FindSrv,
+    artifactSrv: ArtifactSrv,
     implicit val system: ActorSystem,
     implicit val ec: ExecutionContext,
     implicit val mat: Materializer) {
 
+  lazy val logger = Logger(getClass)
+  lazy val alertSrv: AlertSrv = alertSrvProvider.get
+
   def findCaseEntity(entity: BaseEntity): Future[Case] = {
     import org.elastic4play.services.QueryDSL._
 
@@ -70,25 +110,74 @@ class ActionOperationSrv @Inject() (
       case (a: Alert, _) ⇒ a.caze().fold(Future.failed[Case](BadRequestError("Alert hasn't been imported to case")))(caseSrv.get)
       case (_, model: ChildModelDef[_, _, _, _]) ⇒
         findSrv(model.parentModel, "_id" ~= entity.parentId.getOrElse(throw InternalError(s"Child entity $entity has no parent ID")), Some("0-1"), Nil)
-          ._1.runWith(Sink.head).flatMap(findCaseEntity)
+          ._1.runWith(Sink.head).flatMap(findCaseEntity _)
       case _ ⇒ Future.failed(BadRequestError("Case not found"))
     }
   }
 
+  def findTaskEntity(entity: BaseEntity): Future[Task] = {
+    import org.elastic4play.services.QueryDSL._
+
+    (entity, entity.model) match {
+      case (a: Task, _) ⇒ Future.successful(a)
+      case (_, model: ChildModelDef[_, _, _, _]) ⇒
+        findSrv(model.parentModel, "_id" ~= entity.parentId.getOrElse(throw InternalError(s"Child entity $entity has no parent ID")), Some("0-1"), Nil)
+          ._1.runWith(Sink.head).flatMap(findTaskEntity _)
+      case _ ⇒ Future.failed(BadRequestError("Task not found"))
+    }
+  }
+
   def execute(entity: BaseEntity, operation: ActionOperation)(implicit authContext: AuthContext): Future[ActionOperation] = {
     if (operation.status == ActionOperationStatus.Waiting) {
-      val updatedOperation = operation match {
-        case AddTagToCase(tag, _, _) ⇒
-          RetryOnError() { // FIXME find the right exception
+      Retry()(classOf[VersionConflictEngineException]) {
+        operation match {
+          case AddTagToCase(tag, _, _) ⇒
             for {
-              caze ← findCaseEntity(entity)
+              initialCase ← findCaseEntity(entity)
+              caze ← caseSrv.get(initialCase.id)
               _ ← caseSrv.update(caze, Fields.empty.set("tags", Json.toJson((caze.tags() :+ tag).distinct)), ModifyConfig(retryOnConflict = 0, version = Some(caze.version)))
             } yield operation.updateStatus(ActionOperationStatus.Success, "")
-          }
-        case _ ⇒ Future.successful(operation)
+          case AddTagToArtifact(tag, _, _) ⇒
+            entity match {
+              case initialArtifact: Artifact ⇒
+                for {
+                  artifact ← artifactSrv.get(initialArtifact.artifactId())
+                  _ ← artifactSrv.update(artifact.artifactId(), Fields.empty.set("tags", Json.toJson((artifact.tags() :+ tag).distinct)), ModifyConfig(retryOnConflict = 0, version = Some(artifact.version)))
+                } yield operation.updateStatus(ActionOperationStatus.Success, "")
+              case _ ⇒ Future.failed(BadRequestError("Artifact not found"))
+            }
+          case CreateTask(fields, _, _) ⇒
+            for {
+              caze ← findCaseEntity(entity)
+              _ ← taskSrv.create(caze, Fields(fields))
+            } yield operation.updateStatus(ActionOperationStatus.Success, "")
+          case AddCustomFields(name, tpe, value, _, _) ⇒
+            for {
+              initialCase ← findCaseEntity(entity)
+              caze ← caseSrv.get(initialCase.id)
+              customFields = caze.customFields().asOpt[JsObject].getOrElse(JsObject.empty) ++ Json.obj(name -> Json.obj(tpe -> value))
+              _ ← caseSrv.update(caze, Fields.empty.set("customFields", customFields), ModifyConfig(retryOnConflict = 0, version = Some(caze.version)))
+            } yield operation.updateStatus(ActionOperationStatus.Success, "")
+          case CloseTask(_, _) ⇒
+            for {
+              initialTask ← findTaskEntity(entity)
+              task ← taskSrv.get(initialTask.id)
+              _ ← taskSrv.update(task, Fields.empty.set("status", TaskStatus.Completed.toString).set("flag", JsFalse), ModifyConfig(retryOnConflict = 0, version = Some(task.version)))
+            } yield operation.updateStatus(ActionOperationStatus.Success, "")
+          case MarkAlertAsRead(_, _) ⇒
+            entity match {
+              case alert: Alert ⇒ alertSrv.markAsRead(alert).map(_ ⇒ operation.updateStatus(ActionOperationStatus.Success, ""))
+              case _            ⇒ Future.failed(BadRequestError("Alert not found"))
+            }
+          case o ⇒ Future.successful(operation.updateStatus(ActionOperationStatus.Failure, s"Operation $o not supported"))
+        }
       }
-      updatedOperation.recover { case error ⇒ operation.updateStatus(ActionOperationStatus.Failure, error.getMessage) }
+        .recover {
+          case error ⇒
+            logger.error("Operation execution fails", error)
+            operation.updateStatus(ActionOperationStatus.Failure, error.getMessage)
+        }
     }
     else Future.successful(operation)
   }
-}
+}