Merge branch 'release/2.9.1'

AUTHORS

@@ -11,6 +11,11 @@ Contributors
 * CERT Banque de France (CERT-BDF)
 * Nabil Adouani
 
+Contributed Analyzers
+---------------------
+
+* URLCategory: Eric Capuano
+
 Copyright (C) 2014-2016 Thomas Franco
 Copyright (C) 2014-2016 Saâd Kadhi
 Copyright (C) 2014-2016 Jérôme Leonard

CHANGELOG.md

@@ -0,0 +1,42 @@
+# Change Log
+
+## [2.9.1](https://github.com/CERT-BDF/TheHive/tree/2.9.1)
+
+**Implemented enhancements:**
+
+- Update logo and favicon [\#45](https://github.com/CERT-BDF/TheHive/issues/45)
+- Inconsistent wording between the login and user management pages [\#44](https://github.com/CERT-BDF/TheHive/issues/44)
+- MaxMind Analyzer 'Short Report' has hard-coded language [\#23](https://github.com/CERT-BDF/TheHive/issues/23)
+- Don't update imported case from MISP if it is deleted or merged [\#22](https://github.com/CERT-BDF/TheHive/issues/22)
+
+**Fixed bugs:**
+
+- NPE occurs at startup if conf directory doesn't exists [\#41](https://github.com/CERT-BDF/TheHive/issues/41)
+- Resource not found by Assets controller [\#38](https://github.com/CERT-BDF/TheHive/issues/38)
+- Systemd startup script does not work [\#29](https://github.com/CERT-BDF/TheHive/issues/29)
+- MISP event parsing error when it doesn't contain any attribute [\#25](https://github.com/CERT-BDF/TheHive/issues/25)
+- Phantom tabs [\#18](https://github.com/CERT-BDF/TheHive/issues/18)
+- The Action button of observables list is blank [\#15](https://github.com/CERT-BDF/TheHive/issues/15)
+- Description becomes empty when you cancel an edition [\#13](https://github.com/CERT-BDF/TheHive/issues/13)
+- Metric Labels Not Showing in Case View [\#10](https://github.com/CERT-BDF/TheHive/issues/10)
+- chrome on os x - header alignment [\#5](https://github.com/CERT-BDF/TheHive/issues/5)
+- Tags not saving when creating observable. [\#4](https://github.com/CERT-BDF/TheHive/issues/4)
+
+**Closed issues:**
+
+- Statistics based on Tags [\#37](https://github.com/CERT-BDF/TheHive/issues/37)
+- Statistics on a per case template name / prefix basis [\#31](https://github.com/CERT-BDF/TheHive/issues/31)
+- Observable Viewing Page [\#17](https://github.com/CERT-BDF/TheHive/issues/17)
+- Case merging [\#14](https://github.com/CERT-BDF/TheHive/issues/14)
+- Give us something to work with! [\#2](https://github.com/CERT-BDF/TheHive/issues/2)
+
+**Merged pull requests:**
+
+- New analyzer to check URL categories [\#24](https://github.com/CERT-BDF/TheHive/pull/24) ([ecapuano](https://github.com/ecapuano))
+- Fix "Run from Docker" [\#9](https://github.com/CERT-BDF/TheHive/pull/9) ([2xyo](https://github.com/2xyo))
+- Fixing a Simple Typo [\#6](https://github.com/CERT-BDF/TheHive/pull/6) ([swannysec](https://github.com/swannysec))
+- Fixed broken link to Wiki [\#1](https://github.com/CERT-BDF/TheHive/pull/1) ([Neo23x0](https://github.com/Neo23x0))
+
+
+
+\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*

README.md

@@ -41,13 +41,14 @@ TheHive is written in Scala and uses ElasticSearch 2.x for storage. Its REST API
 ![](images/Architecture.png)
 
 ## Analyzers
-The first public release of TheHive is provided with 7 analyzers:
+TheHive 2.9.1 is provided with 8 analyzers:
 + DNSDB*: leverage Farsight's [DNSDB](https://www.dnsdb.info/) for pDNS.
 + DomainTools*: look up domain names, IP addresses, WHOIS records, etc. using the popular [DomainTools](http://domaintools.com/) service API.
 + Hippocampe: query threat feeds through [Hippocampe](https://github.com/CERT-BDF/Hippocampe), a FOSS tool that centralizes feeds and allows you to associate a confidence level to each one of them (that can be changed over time) and get a score indicating the data quality.
 + MaxMind: geolocation.
 + Olevba: parse OLE and OpenXML files using [olevba](http://www.decalage.info/python/olevba) to detect VBA macros, extract their source code etc.
 + Outlook MsgParser: this analyzer allows to add an Outlook message file as an observable and parse it automatically.
++ URLCategory: checks the Fortinet categories of URLs.
 + VirusTotal*: look up files, URLs and hashes through [VirusTotal](https://www.virustotal.com/).
 
 The star (*) indicates that the analyzer needs an API key to work correctly. We do not provide API keys. You have to use your own.

analyzers/MaxMind/report/success_short.html

@@ -1 +1 @@
-<span class="label label-info">IP location: {{content.country.names.fr}} / {{content.continent.names.fr}}</span>
+<span class="label label-info">IP location: {{content.country.name}} / {{content.continent.name}}</span>

analyzers/URLCategory/report/success_long.html

@@ -0,0 +1,18 @@
+<div class="panel panel-info">
+    <div class="panel-heading">
+        URL Categories of <strong>{{artifact.data}}</strong>
+    </div>
+    <div class="panel-body">
+	<dl class="dl-horizontal" ng-if="content.fortinet_category">
+                <dt>Fortinet URL Category: </dt>
+                <dd class="wrap">{{content.fortinet_category}}&nbsp; 
+			<i class="fa fa-search"></i>
+                        <a ng-href="http://www.fortiguard.com/iprep?data={{artifact.data}}&lookup=Lookup" target="_blank">
+                                View Full Report</a>
+			<i class="fa fa-recycle"></i>    
+			<a ng-href="http://www.fortiguard.com/iprep_form?data={{artifact.data}}" target="_blank">
+				Request Recategorization</a> 
+		</dd>
+        </dl>
+    </div>
+</div>

analyzers/URLCategory/report/success_short.html

@@ -0,0 +1,4 @@
+<span class="label label-info">
+	URLCat:
+	<span ng-if="content.fortinet_category">{{content.fortinet_category}}&nbsp;</span>
+</span>

analyzers/URLCategory/urlcategory.py

@@ -0,0 +1,85 @@
+#!/usr/bin/env python
+# encoding: utf-8
+import sys
+import os
+import json
+import codecs
+import time
+import re
+import requests
+
+if sys.stdout.encoding != 'UTF-8':
+	if sys.version_info.major == 3:
+		sys.stdout = codecs.getwriter('utf-8')(sys.stdout.buffer, 'strict')
+	else:
+		sys.stdout = codecs.getwriter('utf-8')(sys.stdout, 'strict')
+if sys.stderr.encoding != 'UTF-8':
+	if sys.version_info.major == 3:
+		sys.stderr = codecs.getwriter('utf-8')(sys.stderr.buffer, 'strict')
+	else:
+		sys.stderr = codecs.getwriter('utf-8')(sys.stderr, 'strict')
+
+# load artifact
+artifact = json.load(sys.stdin)
+
+def error(message):
+	print('{{"errorMessage":"{}"}}'.format(message))
+	sys.exit(1)
+
+def get_param(name, default=None, message=None, current=artifact):
+	if isinstance(name, str):
+		name = name.split('.')
+	if len(name) == 0:
+		return current
+	else:
+		value = current.get(name[0])
+		if value == None:
+			if message != None:
+				error(message)
+			else:
+				return default
+		else:
+			return get_param(name[1:], default, message, value)
+
+def debug(msg):
+	#print >> sys.stderr, msg
+	pass
+
+def fortinet_category(data):
+	debug('>> fortinet_category ' + str(data))
+	pattern = re.compile("(?:Category: )([\w\s]+)")
+	baseurl = 'http://www.fortiguard.com/iprep?data='
+	tailurl = '&lookup=Lookup'
+	url = baseurl + data + tailurl
+	r = requests.get(url)
+	category_match = re.search(pattern, r.content, flags=0)
+	return category_match.group(1)
+
+http_proxy     = get_param('config.proxy.http')
+https_proxy    = get_param('config.proxy.https')
+max_tlp        = get_param('config.max_tlp', 1)
+tlp            = get_param('tlp', 2) # amber by default
+data_type      = get_param('dataType', None, 'Missing dataType field')
+service        = get_param('config.service', None, 'Service parameter is missing')
+
+# run  only if TLP condition is met
+if tlp > max_tlp:
+	error('Error with TLP value ; see max_tlp in config or tlp value in input data')
+
+# setup proxy
+if http_proxy != None:
+	os.environ['http_proxy'] = http_proxy
+if https_proxy != None:
+	os.environ['https_proxy'] = https_proxy
+
+if service == 'query':
+	if data_type == 'url' or data_type == 'domain':
+		data = get_param('data', None, 'Data is missing')
+	        json.dump({
+                        'fortinet_category': fortinet_category(data)
+                	}, sys.stdout, ensure_ascii=False)
+	else:
+		error('Invalid data type')
+else:
+	error('Invalid service')
+

analyzers/URLCategory_1.0.json

@@ -0,0 +1,13 @@
+{
+	"name": "URLCategory",
+	"version": "1.0",
+	"report": "URLCategory/report",
+	"description": "URL Category query: checks the category of a specific URL or domain",
+	"dataTypeList": ["url", "domain"],
+	"baseConfig" : "URLCategory",
+	"config": {
+		"service": "query",
+		"max_tlp": 10
+	},
+	"command": "URLCategory/urlcategory.py"
+}

install/thehive.service

@@ -5,15 +5,19 @@ Wants=network-online.target
 After=network-online.target
 
 [Service]
+Environment=PID_DIR=/var/run/thehive
 WorkingDirectory=/opt/thehive
+
 User=thehive
 Group=thehive
 
-ExecStart=/opt/thehive/bin/thehive \
-			-Dconfig.file=/etc/thehive/application.conf \
-			-Dhttp.port=9000 \
-			-Dpidfile.path=/var/run/thehive/pid
+RuntimeDirectory=thehive
+RuntimeDirectoryMode=0750
 
+ExecStart=/opt/thehive/bin/thehive \
+                        -Dconfig.file=/etc/thehive/application.conf \
+                        -Dhttp.port=9000 \
+			-Dpidfile.path=${PID_DIR}/pid
 
 StandardOutput=journal
 StandardError=inherit

project/BuildSettings.scala

@@ -7,7 +7,7 @@ object BasicSettings extends AutoPlugin {
   override def projectSettings = Seq(
     organization := "org.cert-bdf",
     licenses += "AGPL-V3" -> url("https://www.gnu.org/licenses/agpl-3.0.html"),
-    version := "2.9.0",
+    version := "2.9.1",
     resolvers += Resolver.bintrayRepo("cert-bdf", "elastic4play"),
     scalaVersion := Dependencies.scalaVersion,
     scalacOptions ++= Seq(

project/Dependencies.scala

@@ -29,7 +29,7 @@ object Dependencies {
     val reflections = "org.reflections" % "reflections" % "0.9.10"
     val zip4j = "net.lingala.zip4j" % "zip4j" % "1.3.2"
     val akkaTest = "com.typesafe.akka" %% "akka-stream-testkit" % "2.4.4"
-    val elastic4play = "org.cert-bdf" %% "elastic4play" % "1.1.0"
+    val elastic4play = "org.cert-bdf" %% "elastic4play" % "1.1.1"
 
     object Elastic4s {
       private val version = "2.3.0"

thehive-backend/app/controllers/Case.scala

@@ -24,10 +24,13 @@ import org.elastic4play.services.JsonFormat.{ aggReads, queryReads }
 
 import models.{ Case, CaseStatus }
 import services.{ CaseSrv, TaskSrv }
+import services.CaseMergeSrv
+import scala.util.Try
 
 @Singleton
 class CaseCtrl @Inject() (
     caseSrv: CaseSrv,
+    caseMergeSrv: CaseMergeSrv,
     taskSrv: TaskSrv,
     auxSrv: AuxSrv,
     authenticated: Authenticated,
@@ -39,46 +42,53 @@ class CaseCtrl @Inject() (
   val log = Logger(getClass)
 
   @Timed
-  def create() = authenticated(Role.write).async(fieldsBodyParser) { implicit request =>
+  def create() = authenticated(Role.write).async(fieldsBodyParser) { implicit request ⇒
     caseSrv.create(request.body)
-      .map(caze => renderer.toOutput(CREATED, caze))
+      .map(caze ⇒ renderer.toOutput(CREATED, caze))
   }
 
   @Timed
-  def get(id: String) = authenticated(Role.read).async { implicit request =>
-    caseSrv.get(id)
-      .map(caze => renderer.toOutput(OK, caze))
+  def get(id: String) = authenticated(Role.read).async { implicit request ⇒
+    val withStats = for {
+      statsValues <- request.queryString.get("nstats")
+      firstValue <- statsValues.headOption
+    } yield Try(firstValue.toBoolean).getOrElse(firstValue == "1")
+
+    for {
+      caze ← caseSrv.get(id)
+      casesWithStats ← auxSrv.apply(caze, 0, withStats.getOrElse(false))
+    } yield renderer.toOutput(OK, casesWithStats)
   }
 
   @Timed
-  def update(id: String) = authenticated(Role.write).async(fieldsBodyParser) { implicit request =>
+  def update(id: String) = authenticated(Role.write).async(fieldsBodyParser) { implicit request ⇒
     val isCaseClosing = request.body.getString("status").filter(_ == CaseStatus.Resolved.toString).isDefined
 
     for {
       // Closing the case, so lets close the open tasks
-      caze <- caseSrv.update(id, request.body)
-      closedTasks <- if (isCaseClosing) taskSrv.closeTasksOfCase(id) else Future.successful(Nil) // FIXME log warning if closedTasks contains errors
+      caze ← caseSrv.update(id, request.body)
+      closedTasks ← if (isCaseClosing) taskSrv.closeTasksOfCase(id) else Future.successful(Nil) // FIXME log warning if closedTasks contains errors
     } yield renderer.toOutput(OK, caze)
   }
 
   @Timed
-  def bulkUpdate() = authenticated(Role.write).async(fieldsBodyParser) { implicit request =>
+  def bulkUpdate() = authenticated(Role.write).async(fieldsBodyParser) { implicit request ⇒
     val isCaseClosing = request.body.getString("status").filter(_ == CaseStatus.Resolved.toString).isDefined
-    
-    request.body.getStrings("ids").fold(Future.successful(Ok(JsArray()))) { ids =>
+
+    request.body.getStrings("ids").fold(Future.successful(Ok(JsArray()))) { ids ⇒
       if (isCaseClosing) taskSrv.closeTasksOfCase(ids: _*) // FIXME log warning if closedTasks contains errors
-      caseSrv.bulkUpdate(ids, request.body.unset("ids")).map(multiResult => renderer.toMultiOutput(OK, multiResult))
+      caseSrv.bulkUpdate(ids, request.body.unset("ids")).map(multiResult ⇒ renderer.toMultiOutput(OK, multiResult))
     }
   }
 
   @Timed
-  def delete(id: String) = authenticated(Role.write).async { implicit request =>
+  def delete(id: String) = authenticated(Role.write).async { implicit request ⇒
     caseSrv.delete(id)
-      .map(_ => NoContent)
+      .map(_ ⇒ NoContent)
   }
 
   @Timed
-  def find() = authenticated(Role.read).async(fieldsBodyParser) { implicit request =>
+  def find() = authenticated(Role.read).async(fieldsBodyParser) { implicit request ⇒
     val query = request.body.getValue("query").fold[QueryDef](QueryDSL.any)(_.as[QueryDef])
     val range = request.body.getString("range")
     val sort = request.body.getStrings("sort").getOrElse(Nil)
@@ -91,26 +101,33 @@ class CaseCtrl @Inject() (
   }
 
   @Timed
-  def stats() = authenticated(Role.read).async(fieldsBodyParser) { implicit request =>
+  def stats() = authenticated(Role.read).async(fieldsBodyParser) { implicit request ⇒
     val query = request.body.getValue("query").fold[QueryDef](QueryDSL.any)(_.as[QueryDef])
     val aggs = request.body.getValue("stats").getOrElse(throw BadRequestError("Parameter \"stats\" is missing")).as[Seq[Agg]]
-    caseSrv.stats(query, aggs).map(s => Ok(s))
+    caseSrv.stats(query, aggs).map(s ⇒ Ok(s))
   }
 
   @Timed
-  def linkedCases(id: String) = authenticated(Role.read).async { implicit request =>
+  def linkedCases(id: String) = authenticated(Role.read).async { implicit request ⇒
     caseSrv.linkedCases(id)
       .runWith(Sink.seq)
-      .map { cases =>
+      .map { cases ⇒
         val casesList = cases.sortWith {
-          case ((c1, _), (c2, _)) => c1.startDate().after(c2.startDate())
+          case ((c1, _), (c2, _)) ⇒ c1.startDate().after(c2.startDate())
         }.map {
-          case (caze, artifacts) =>
+          case (caze, artifacts) ⇒
             Json.toJson(caze).as[JsObject] - "description" +
               ("linkedWith" -> Json.toJson(artifacts)) +
               ("linksCount" -> Json.toJson(artifacts.size))
         }
         renderer.toOutput(OK, casesList)
       }
   }
+
+  @Timed
+  def merge(caseId1: String, caseId2: String) = authenticated(Role.read).async { implicit request ⇒
+    caseMergeSrv.merge(caseId1, caseId2).map { caze ⇒
+      renderer.toOutput(OK, caze)
+    }
+  }
 }