Merge branch 'release/4.0.2' into master-th4

TheHive-Project · Nov 20, 2020 · 42dd59b · 42dd59b
2 parents bd0632b + 22cbf23
commit 42dd59b
Show file tree

Hide file tree

Showing 83 changed files with 812 additions and 443 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,30 @@
 # Change Log
 
+## [4.0.2](https://github.com/TheHive-Project/TheHive/milestone/64) (2020-11-20)
+
+**Implemented enhancements:**
+
+- [Feature Request] Add a dedicated permission to give access to TheHiveFS [\#1655](https://github.com/TheHive-Project/TheHive/issues/1655)
+- [Feature Request] Normalize editable input fields [\#1669](https://github.com/TheHive-Project/TheHive/issues/1669)
+
+**Fixed bugs:**
+
+- [Bug] Unable to list Cases [\#1598](https://github.com/TheHive-Project/TheHive/issues/1598)
+- [Bug] Alert to case merge is broken in v4.0.1 [\#1648](https://github.com/TheHive-Project/TheHive/issues/1648)
+- [Bug] Attachment.* filters are broken under observable search in v4.0.1  [\#1649](https://github.com/TheHive-Project/TheHive/issues/1649)
+- [Bug] Result of observable update API v0 is empty [\#1652](https://github.com/TheHive-Project/TheHive/issues/1652)
+- [Bug] Display issue of custom fields [\#1653](https://github.com/TheHive-Project/TheHive/issues/1653)
+- [Bug] Persistent AuditSrv:undefined error on 4.0.1 [\#1656](https://github.com/TheHive-Project/TheHive/issues/1656)
+- [Bug] Issues with case attachments section [\#1657](https://github.com/TheHive-Project/TheHive/issues/1657)
+- [Bug] API method broken: /api/case/artifact/_search in 4.0.1 [\#1659](https://github.com/TheHive-Project/TheHive/issues/1659)
+- [Bug] API method broken: /api/case/task/log/_search in 4.0.1 [\#1660](https://github.com/TheHive-Project/TheHive/issues/1660)
+- [Bug] Unable to define ES index on migration [\#1661](https://github.com/TheHive-Project/TheHive/issues/1661)
+- [Bug] Dashboard max aggregation on custom-integer field does not work [\#1662](https://github.com/TheHive-Project/TheHive/issues/1662)
+- [Bug] Missing the fix for errorMessage [\#1666](https://github.com/TheHive-Project/TheHive/issues/1666)
+- [Bug] Fix alert details dialog [\#1672](https://github.com/TheHive-Project/TheHive/issues/1672)
+- [Bug] error 500 with adding an empty file in Observables of an Alert [\#1673](https://github.com/TheHive-Project/TheHive/issues/1673)
+- [Bug] Fix migration of audit logs [\#1676](https://github.com/TheHive-Project/TheHive/issues/1676)
+
 ## [4.0.1](https://github.com/TheHive-Project/TheHive/milestone/60) (2020-11-13)
 
 **Implemented enhancements:**
@@ -24,6 +49,7 @@
 
 **Fixed bugs:**
 
+- [Bug] MISP->THEHIVE4 'ExportOnly' and 'Exceptions' ignored in application.conf file [\#1482](https://github.com/TheHive-Project/TheHive/issues/1482)
 - [Bug] Mobile-responsive Hamburger not visible [\#1290](https://github.com/TheHive-Project/TheHive/issues/1290)
 - [Bug] Unable to start TheHive after migration [\#1450](https://github.com/TheHive-Project/TheHive/issues/1450)
 - [Bug] Expired session should show a dialog or login page on pageload [\#1456](https://github.com/TheHive-Project/TheHive/issues/1456)
@@ -34,7 +60,6 @@
 - [Bug] Dashboard shared/private [\#1474](https://github.com/TheHive-Project/TheHive/issues/1474)
 - [Bug]Migration tool date/number/duration params don't work [\#1478](https://github.com/TheHive-Project/TheHive/issues/1478)
 - [Bug] AuditSrv: undefined on non-case page(s), thehive4-4.0.0-1, Ubuntu [\#1479](https://github.com/TheHive-Project/TheHive/issues/1479)
-- [Bug] MISP->THEHIVE4 'ExportOnly' and 'Exceptions' ignored in application.conf file [\#1482](https://github.com/TheHive-Project/TheHive/issues/1482)
 - [Bug] Unable to enumerate tasks via API [\#1483](https://github.com/TheHive-Project/TheHive/issues/1483)
 - [Bug] Case close notification displays "#undefined" instead of case number [\#1488](https://github.com/TheHive-Project/TheHive/issues/1488)
 - [Bug] Task under "Waiting tasks" and "My tasks" do not display the case number [\#1489](https://github.com/TheHive-Project/TheHive/issues/1489)

diff --git a/ScalliGraph b/ScalliGraph
diff --git a/build.sbt b/build.sbt
@@ -2,7 +2,7 @@ import Dependencies._
 import com.typesafe.sbt.packager.Keys.bashScriptDefines
 import org.thp.ghcl.Milestone
 
-val thehiveVersion         = "4.0.1-1"
+val thehiveVersion         = "4.0.2-1"
 val scala212               = "2.12.12"
 val scala213               = "2.13.1"
 val supportedScalaVersions = List(scala212, scala213)

diff --git a/docker/README.md b/docker/README.md
@@ -1,14 +1,14 @@
 ## Example of docker-compose (not for production)
 With this docker-compose.yml you will be able to run the following images:
-- The Hive 4
+- The Hive 4.0.1-1
 - Cassandra 3.11
 - Cortex 3.1.0-1
 - Elasticsearch 7.9.3
 - Kibana 7.9.3
-- MISP 2.4.133
+- MISP 2.4.134
 - Mysql 8.0.22
 - Redis 6.0.9
-- Shuffle 0.7.1
+- Shuffle 0.7.6
 
 ## Some Hint
 
@@ -17,47 +17,46 @@ In docker-compose version is set 3.8, to run this version you need at least Dock
 ```
 Compose file format    Docker Engine release
 3.8                    19.03.0+
-3.7	               18.06.0+
-3.6	               18.02.0+
-3.5	               17.12.0+
-3.4	               17.09.0+
+3.7	                   18.06.0+
+3.6	                   18.02.0+
+3.5	                   17.12.0+
+3.4	                   17.09.0+
 ```
 If for some reason you have a previous version of Docker Engine or a previous version of Docker Compose and can't upgrade those, you can use 3.7 or 3.6 in docker-compose.yml
 
 
 ### Mapping volumes
-If you take a look of docker-compose.yml you will see you need some local folder that needs to be mapped, so before do docker-compose up, ensure folders (and config files) exist:
-- ./elasticsearch/data:/usr/share/elasticsearch/data
-- ./elasticsearch/logs:/usr/share/elasticsearch/logs
+If you take a look of docker-compose.yml you will see you need some local folder that needs to be mapped, so before do docker-compose up, ensure at least folders with config files exist:
 - ./cortex/application.conf:/etc/cortex/application.conf
 - ./thehive/application.conf:/etc/thehive/application.conf
-- ./data:/data
-- ./mysql:/var/lib/mysql
 
 Structure would look like:
 ```
 ├── docker-compose.yml
-├── elasticsearch
-│   └── data
-│   └── logs
+├── elasticsearch_data
+|── elasticsearch_logs
 ├── cortex
 │   └── application.conf 
-└── thehive
-   └── application.conf
-└── data
-└── mysql
+|── thehive
+|   └── application.conf
+|── data
+|── mysql
 ```
+If you run docker-compose with sudo, ensure you have created elasticsearch_data and elasticsearch_logs folders with non root user, otherwise elasticsearch container will not start.
 
 ### ElasticSearch
 ElasticSearch container likes big mmap count (https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html) so from shell you can change with
 ```sysctl -w vm.max_map_count=262144```
-Due you would run all on same system and maybe you have a limited amount of RAM, better to set some size, for ElasticSearch, in docker-compose.yml I added those:
+To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf. To verify after rebooting, run sysctl vm.max_map_count
+
+If you would run all containers on the same system - and maybe you have a limited amount of RAM - better to set some limit, for ElasticSearch, in docker-compose.yml I added those:
 
 ```- bootstrap.memory_lock=true```
 ```- "ES_JAVA_OPTS=-Xms256m -Xmx256m"```
 
 Adjust depending on your needs and your env. Without these settings in my environment ElasticSearch was using 1.5GB
 
+
 ### Cassandra
 Like for ElasticSearch maybe you would run all on same system and maybe you don't have a limited amount of RAM, better to set some size, here for Cassandra, in docker-compose.yml I added those:
 
@@ -68,7 +67,7 @@ Adjust depending on your needs and your env. Without these settings in my enviro
 
 ### Cortex-Analyzers
 - In order to use Analyzers in docker version, it is set  the online json url instead absolute path of analyzers in the application.conf of Cortex:
-  https://dl.bintray.com/thehive-project/cortexneurons/analyzers.json
+  https://download.thehive-project.org/analyzers.json
 - In order to use Analyzers in docker version it is set the application.conf thejob: ```
   job {
   runner = [docker]
@@ -142,3 +141,21 @@ curl -XPUT [email protected]:[email protected] -H 'Content-type: application
 ```
 - Now are able to play automation with The Hive, Cortex-Analyzers, MISP thanks to SHUFFLE!
 
+
+### Result
+In conclusion, after execute ```sudo docker-compose up``` you will have the following services running:
+
+
+| Service   |      Address      |  User |  Password |
+|----------|:-------------:|:------:|------:|
+| The Hive |  http://localhost:9000 | [email protected] | secret
+| Cortex |    http://localhost:9001  |    |
+| Elasticsearch | http://localhost:9200 |     |
+| Kibana |  http://localhost:5601 |  |
+| MISP |    https://localhost:443   |  [email protected] | admin
+| Shuffle | http://localhost:3001 |    |
+
+
+
+![image](https://user-images.githubusercontent.com/16938405/99674126-e8c99f80-2a75-11eb-9a8b-1603cf67d665.png)
+![image](https://user-images.githubusercontent.com/16938405/99674544-7c02d500-2a76-11eb-92a5-3fbb5c3c5cc5.png)
diff --git a/docker/cortex/application.conf b/docker/cortex/application.conf
@@ -179,7 +179,7 @@ analyzer {
   # - directory where analyzers are installed
   # - json file containing the list of analyzer descriptions
   urls = [
-    "https://dl.bintray.com/thehive-project/cortexneurons/analyzers.json"
+    "https://download.thehive-project.org/analyzers.json"
     #"/absolute/path/of/analyzers"
   ]
 
@@ -199,7 +199,7 @@ analyzer {
 responder {
   # responder location (same format as analyzer.urls)
   urls = [
-    "https://dl.bintray.com/thehive-project/cortexneurons/responders.json"
+    "https://download.thehive-project.org/responders.json"
     #"/absolute/path/of/responders"
   ]
 

diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
@@ -22,8 +22,8 @@ services:
         soft: 65536
         hard: 65536
     volumes:
-      - ./elasticsearch/data:/usr/share/elasticsearch/data
-      - ./elasticsearch/logs:/usr/share/elasticsearch/logs        
+      - ./elasticsearch_data:/usr/share/elasticsearch/data
+      - ./elasticsearch_logs:/usr/share/elasticsearch/logs        
   kibana:
     image: 'docker.elastic.co/kibana/kibana:7.9.3'
     container_name: kibana
@@ -67,7 +67,7 @@ services:
       - '0.0.0.0:9000:9000'
     volumes:
       - ./thehive/application.conf:/etc/thehive/application.conf
-      - ./data:/data
+      - ./data:/opt/data
     command: '--no-config --no-config-secret'
 
   redis:
@@ -77,6 +77,7 @@ services:
 
   db:
     image: mysql:latest
+    container_name: mysql    
     restart: unless-stopped     
     command: --default-authentication-plugin=mysql_native_password
     restart: always
@@ -98,12 +99,19 @@ services:
       - "80:80"
       - "443:443"
     environment:
-      - "HOSTNAME=http://misp"
+      - "HOSTNAME=https://localhost"
       - "REDIS_FQDN=redis"
       - "INIT=true"             # Initialze MISP, things includes, attempting to import SQL and the Files DIR
       - "CRON_USER_ID=1"        # The MISP user ID to run cron jobs as
       - "DISIPV6=true" # Disable IPV6 in nginx
-
+  misp-modules:
+    image: coolacid/misp-docker:modules-latest
+    container_name: misp-modules
+    environment:
+      - "REDIS_BACKEND=redis"
+    depends_on:
+      - redis
+      - db 
 
 #READY FOR AUTOMATION ?
   frontend:

diff --git a/docker/thehive/application.conf b/docker/thehive/application.conf
@@ -20,7 +20,7 @@ db {
 
 storage {
    provider: localfs
-   localfs.directory: /opt/data
+   localfs.location: /opt/data
 }
 
 play.modules.enabled += org.thp.thehive.connector.cortex.CortexModule

diff --git a/frontend/app/index.html b/frontend/app/index.html
@@ -43,12 +43,14 @@
     <link rel="stylesheet" href="styles/case.css"/>
     <link rel="stylesheet" href="styles/flow.css"/>
     <link rel="stylesheet" href="styles/label.css"/>
+    <link rel="stylesheet" href="styles/updatable.css"/>
     <link rel="stylesheet" href="styles/flex-table.css"/>
     <link rel="stylesheet" href="styles/search.css"/>
     <link rel="stylesheet" href="styles/filters.css"/>
     <link rel="stylesheet" href="styles/dashboard.css"/>
     <link rel="stylesheet" href="styles/case-item.css"/>
     <link rel="stylesheet" href="styles/case-template.css"/>
+    <link rel="stylesheet" href="styles/custom-fields.css"/>
     <link rel="stylesheet" href="styles/directives/page-sizer.css"/>
     <link rel="stylesheet" href="styles/directives/user.css"/>
     <!-- endbuild -->

diff --git a/frontend/app/scripts/app.js b/frontend/app/scripts/app.js
@@ -411,11 +411,7 @@ angular.module('thehive', [
                 controller: 'CaseAlertsCtrl',
                 resolve: {
                     alerts: function($stateParams, CaseSrv) {
-                        return CaseSrv.alerts({range: 'all'}, {
-                            query: {
-                              case: $stateParams.caseId
-                            }
-                        }).$promise;
+                        return CaseSrv.alerts($stateParams.caseId);
                     }
                 },
                 guard: {

diff --git a/frontend/app/scripts/controllers/SearchCtrl.js b/frontend/app/scripts/controllers/SearchCtrl.js
@@ -1,7 +1,7 @@
 (function() {
     'use strict';
     angular.module('theHiveControllers')
-        .controller('SearchCtrl', function($scope, $q, $stateParams, $uibModal, PSearchSrv, CaseTemplateSrv, CaseTaskSrv, NotificationSrv, EntitySrv, UserSrv, QueryBuilderSrv, GlobalSearchSrv, metadata) {
+        .controller('SearchCtrl', function($scope, $q, $stateParams, $uibModal, PSearchSrv, AlertingSrv, CaseTemplateSrv, CaseTaskSrv, NotificationSrv, EntitySrv, UserSrv, QueryBuilderSrv, GlobalSearchSrv, metadata) {
             $scope.metadata = metadata;
             $scope.toolbar = [
                 // {name: 'all', label: 'All', icon: 'glyphicon glyphicon-search'},
@@ -42,14 +42,24 @@
                     controllerAs: 'dialog',
                     size: 'max',
                     resolve: {
-                        event: event,
+                        event: function() {
+                            return AlertingSrv.get(event.id);
+                        },
                         templates: function() {
                             return CaseTemplateSrv.list();
                         },
                         readonly: true
                     }
-                }).result.then(function(/*response*/) {
+                })
+                .result
+                .then(function(/*response*/) {
                   $scope.searchResults.update();
+                })
+                .catch(function(err) {
+                    if(err && !_.isString(err)) {
+                        NotificationSrv.error('AlertPreview', err.data, err.status);
+                    }
+
                 });
             };
 

diff --git a/frontend/app/scripts/controllers/alert/AlertEventCtrl.js b/frontend/app/scripts/controllers/alert/AlertEventCtrl.js
@@ -39,11 +39,6 @@
                 AlertingSrv.get(eventId).then(function(data) {
                     self.event = data;
                     self.loading = false;
-
-                    self.dataTypes = _.countBy(self.event.artifacts, function(attr) {
-                        return attr.dataType;
-                    });
-
                 }, function(response) {
                   self.loading = false;
                   NotificationSrv.error('AlertEventCtrl', response.data, response.status);
@@ -185,6 +180,7 @@
 
             self.copyId = function(id) {
                 clipboard.copyText(id);
+                NotificationSrv.log('Alert ID has been copied to clipboard', 'success');
             };
 
             this.$onInit = function() {

diff --git a/frontend/app/scripts/controllers/case/CaseAlertsCtrl.js b/frontend/app/scripts/controllers/case/CaseAlertsCtrl.js
@@ -1,7 +1,7 @@
 (function() {
     'use strict';
     angular.module('theHiveControllers').controller('CaseAlertsCtrl',
-        function($scope, $state, $stateParams, $uibModal, $timeout, CaseTabsSrv, VersionSrv, alerts) {
+        function($scope, $state, $stateParams, $uibModal, $timeout, CaseTabsSrv, VersionSrv, NotificationSrv, alerts) {
             $scope.caseId = $stateParams.caseId;
             $scope.alerts = alerts;
             $scope.alertStats = [];
@@ -84,9 +84,16 @@
                         templates: function() {
                             //return CaseTemplateSrv.list();
                             return [];
-                        },                        
+                        },
                         readonly: true
                     }
+                })
+                .result
+                .catch(function(err) {
+                    if(err && !_.isString(err)) {
+                        NotificationSrv.error('AlertPreview', err.data, err.status);
+                    }
+
                 });
             };
 

diff --git a/frontend/app/scripts/controllers/case/CaseDetailsCtrl.js b/frontend/app/scripts/controllers/case/CaseDetailsCtrl.js
@@ -17,13 +17,14 @@
             version: 'v1',
             loadAll: false,
             filter: {
-                '_contains': 'attachment'
+                '_contains': 'attachment.id'
             },
             extraData: ['taskId'],
             pageSize: 100,
             operations: [
                 { '_name': 'getCase', 'idOrName': $scope.caseId },
                 { '_name': 'tasks' },
+                { '_name': 'filter', '_ne':{'_field': 'status', '_value': 'Cancel'}},
                 { '_name': 'logs' },
             ]
         });

diff --git a/frontend/app/scripts/directives/responder-actions.js b/frontend/app/scripts/directives/responder-actions.js
@@ -11,7 +11,7 @@
             templateUrl: 'views/directives/responder-actions.html',
             controller: function($scope, $uibModal) {
 
-                $scope.$watch('actions', function(list) {
+                $scope.$watchCollection('actions.values', function(list) {
                     if(!list) {
                         return;
                     }

diff --git a/frontend/app/scripts/directives/updatableBoolean.js b/frontend/app/scripts/directives/updatableBoolean.js
@@ -12,7 +12,8 @@
                     'active': '=?',
                     'placeholder': '@',
                     'trueText': '@?',
-                    'falseText': '@?'
+                    'falseText': '@?',
+                    'clearable': '<?'
                 }
             };
         });
+9 −0		core/src/main/scala/org/thp/scalligraph/controllers/Entrypoint.scala
+4 −12		core/src/main/scala/org/thp/scalligraph/macro/ModelMacro.scala
+7 −12		core/src/main/scala/org/thp/scalligraph/models/Mapping.scala
+5 −7		core/src/main/scala/org/thp/scalligraph/models/Model.scala
+15 −1		core/src/main/scala/org/thp/scalligraph/query/PublicProperty.scala
+59 −14		core/src/main/scala/org/thp/scalligraph/traversal/TraversalOps.scala
+1 −1		project/build.properties
+1 −1		project/plugins.sbt