Merge branch 'maintenance' into develop

CHANGELOG.md

@@ -63,6 +63,36 @@ CHANGELOG
 ### Known issues
 
 
+3.0.2 (2021-09-10)
+------------------
+
+### Core
+- `intelmq.lib.bot.CollectorBot`: Fixed an issue with within the `new_report` function, which re-loads the harmonization file after a new incoming dataset, which leads to CPU drain and decreased performance (PR#2106 by Sebastian Waldbauer, fixes #2098).
+- `intelmq.lib.bot.Bot`: Make private members `__is_multithreadable` and `__collector_empty_process` protected members `_is_multithreadable` and `_collector_empty_process` to make them easily modifiable by Bot classes (PR#2109 by Sebastian Wagner, fixes #2108).
+  Also affected and adapted bots by this change are:
+  - `intelmq.bots.collectors.api.collector_api`
+  - `intelmq.bots.collectors.stomp.collector`
+  - `intelmq.bots.experts.splunk_saved_search.expert`
+  - `intelmq.bots.experts.threshold.expert`
+  - `intelmq.bots.outputs.file.output`
+  - `intelmq.bots.outputs.misp.output_api`
+  - `intelmq.bots.outputs.misp.output_feed`
+  - `intelmq.bots.outputs.tcp.output`
+  - `intelmq.bots.outputs.udp.output`
+- `intelmq.lib.cache`: Do not create the Cache class if the host is null, allows deactivating the bot statistics (PR#2104 by Sebastian Waldbauer, fixes #2103).
+
+### Bots
+#### Experts
+- `intelmq.bots.experts.domain_suffix.expert`: Only print skipped database update message if verbose mode is active (PR#2107 by Sebastian Wagner, fixes #2016).
+
+### Documentation
+- Add configuration upgrade steps for 3.0 to NEWS (PR#2101 by Sebastian Wagner).
+
+### Known issues
+See [open bug reports](https://github.com/certtools/intelmq/issues?q=is%3Aissue+is%3Aopen+label%3Abug) for a more detailed list.
+- ParserBot: erroneous raw line recovery in error handling (#1850).
+
+
 3.0.1 (2021-09-02)
 ------------------
 

NEWS.md

@@ -29,6 +29,14 @@ The [Data format documentation](https://intelmq.readthedocs.io/en/maintenance/de
 ### Postgres databases
 
 
+3.0.2 Maintenance release (2021-09-10)
+--------------------------------------
+Two performance issues were fixed. One affected all collectors which processed high volumes of data and the other issue affected some bots which used threading.
+See the changelog for more details.
+
+The section on 3.0.0 in this file now contains more details for the upgrade to 3.0.0 in regards to the configuration.
+
+
 3.0.1 Maintenance release (2021-09-02)
 --------------------------------------
 
@@ -43,6 +51,28 @@ The postgresql output bot was removed. The bot was marked as deprecated in 2019
 ### Requirements
 IntelMQ now uses YAML for the runtime configuration and therefore needs the `ruamel.yaml` library.
 
+### Configuration
+The `defaults.conf` file was removed. Settings that should effect all the bots are not part of the runtime.conf file and are configured in the `global` section in that file.
+The `intelmqctl upgrade-config` command migrates the existing values from the `defaults.conf` file to the `runtime.conf` file under the `global` section and then deletes the `defaults.conf` file.
+The `pipeline.conf` file was removed. The source- and destination-queues of the bots are now configured in the bot configuration itself, thus in the `runtime.conf` file.
+The `intelmqctl upgrade-config` command migrates the existing configuration from the `pipeline.conf` file to the individual bot configurations in the `runtime.conf` configuration file.
+The `runtime.conf` file was replaced by a `runtime.yaml` file. IntelMQ moves the file for you if it does not find a runtime.conf but a runtime.yaml file. When IntelMQ changes the file, it now writes YAML syntax.
+
+#### When using the official deb/rpm-packages or the official Docker image
+Unfortunately, the automatic upgrade procedures has a flaw.
+The packages provide a default runtime configuration, but only for new installations if there is no previously existing installation.
+But as the runtime configuration was renamed from `/etc/intelmq/runtime.conf` to `/etc/intelmq/runtime.yaml`, this check comes to nothing, and the `/etc/intelmq/runtime.yaml` get installed.
+But only the new filename is considered by IntelMQ itself, so the configuration *appears* to be lost.
+To fix this:
+- remove the newly provided `runtime.yaml`
+- make sure that the `runtime.conf` is the correct file with your correct configuration
+- IntelMQ will rename and convert the configuration automatically, but we need to trigger the migration of the `pipeline.conf` and `defaults.conf`:
+  ```
+  sudo -u intelmq intelmqctl upgrade-config -f -u v300_pipeline_file_removal
+  sudo -u intelmq intelmqctl upgrade-config -f -u v300_defaults_file_removal
+  sudo -u intelmq intelmqctl upgrade-config -f -u v301_deprecations
+  ```
+
 ### Tools
 
 #### intelmqdump
@@ -119,13 +149,6 @@ Most of the usages were wrong anyway, and should have been infected-device, malw
 There is only one usage in IntelMQ, which can not be changed.
 And that one is really about malware itself (or: the hashes of samples). For this purpose, the new type "malware" under the taxonomy "other" was created, *slightly* deviating from the RSIT in this respect, but the "other" taxonomy can be freely extended.
 
-### Configuration
-
-The `defaults.conf` file was removed. Settings that should effect all the bots are not part of the runtime.conf file and are configured in the `global` section in that file.
-The `intelmqctl upgrade-config` command migrates the existing values from the `defaults.conf` file to the `runtime.conf` file under the `global` section and then deletes the `defaults.conf` file.
-The `pipeline.conf` file was removed. The source- and destination-queues of the bots are now configured in the bot configuration itself, thus in the `runtime.conf` file.
-The `intelmqctl upgrade-config` command migrates the existing configuration from the `pipeline.conf` file to the individual bot configurations in the `runtime.conf` configuration file.
-The `runtime.conf` file was replaced by a `runtime.yaml` file. IntelMQ moves the file for you if it does not find a runtime.conf but a runtime.yaml file. When IntelMQ changes the file, it now writes YAML syntax.
 
 #### Removal of deprecated bots and behaviour
 - The bot `intelmq.bots.experts.ripencc_abuse_contact.expert` has been removed. It was replaced by `intelmq.bots.experts.ripe.expert` and marked as deprecated in 2.0.0.beta1.

debian/changelog

@@ -4,6 +4,12 @@ intelmq (3.1.0~alpha1-1) UNRELEASED; urgency=medium
 
  -- Sebastian Wagner <[email protected]>  Fri, 13 Aug 2021 11:09:58 +0200
 
+intelmq (3.0.2-1) stable; urgency=medium
+
+  * Update to 3.0.2.
+
+ -- Sebastian Wagner <[email protected]>  Fri, 10 Sep 2021 08:53:47 +0200
+
 intelmq (3.0.1-1) stable; urgency=medium
 
   * Update to 3.0.1.

intelmq/bots/collectors/api/collector_api.py

@@ -37,9 +37,9 @@ class APICollectorBot(CollectorBot):
     """Collect data by exposing a HTTP API interface"""
     name: str = "API"
     port: int = 5000
-    __collector_empty_process: bool = True
+    _collector_empty_process: bool = True
     provider: str = "APICollector"
-    __is_multithreadable: bool = False
+    _is_multithreadable: bool = False
     use_socket = False
     socket_path = '/tmp/imq_api_default_socket'
     _server: Optional['HTTPServer'] = None

intelmq/bots/collectors/stomp/collector.py

@@ -69,7 +69,7 @@ class StompCollectorBot(CollectorBot):
     ssl_client_certificate_key: str = 'client.key'  # TODO pathlib.Path
     heartbeat: int = 6000
 
-    __collector_empty_process: bool = True
+    _collector_empty_process: bool = True
     __conn = False  # define here so shutdown method can check for it
 
     def init(self):

intelmq/bots/experts/domain_suffix/expert.py

@@ -91,7 +91,8 @@ def update_database(cls, verbose=False):
             sys.exit("Database update failed. Your configuration of {0} is missing key {1}.".format(bot, e))
 
         if not bots:
-            print("Database update skipped. No bots of type {0} present in runtime.conf.".format(__name__))
+            if verbose:
+                print("Database update skipped. No bots of type {0} present in runtime.conf.".format(__name__))
             sys.exit(0)
 
         # we only need to import now. If there are no asn_lookup bots, this dependency does not need to be installed

intelmq/bots/experts/splunk_saved_search/expert.py

@@ -81,7 +81,7 @@ class SplunkSavedSearchBot(ExpertBot):
     search_parameters = {"event field": "search parameter"}
     url: str = None
 
-    __is_multithreadable = False
+    _is_multithreadable = False
 
     def init(self):
         if requests is None:

intelmq/bots/experts/threshold/expert.py

@@ -68,7 +68,7 @@ class ThresholdExpertBot(ExpertBot, CacheMixin):
 
     _message_processed_verb = 'Forwarded'
 
-    __is_multithreadable = False
+    _is_multithreadable = False
     bypass = False
 
     def init(self):

intelmq/bots/outputs/file/output.py

@@ -22,7 +22,7 @@ class FileOutputBot(OutputBot):
     message_jsondict_as_string: bool = False
     message_with_type: bool = False
     single_key: bool = False
-    __is_multithreadable = False
+    _is_multithreadable = False
 
     def init(self):
         # needs to be done here, because in process() FileNotFoundError handling we call init(),

intelmq/bots/outputs/misp/output_api.py

@@ -95,7 +95,7 @@ class MISPAPIOutputBot(OutputBot):
     misp_url: str = None
     significant_fields: list = []
 
-    _Bot__is_multithreadable = False
+    _is_multithreadable = False
 
     def init(self):
         if pymisp is None and import_fail_reason == 'syntax':

intelmq/bots/outputs/misp/output_feed.py

@@ -35,7 +35,7 @@ class MISPFeedOutputBot(OutputBot):
     misp_org_name = None
     misp_org_uuid = None
     output_dir: str = "/opt/intelmq/var/lib/bots/mispfeed-output"  # TODO: should be path
-    __is_multithreadable: bool = False
+    _is_multithreadable: bool = False
 
     @staticmethod
     def check_output_dir(dirname):

intelmq/bots/outputs/tcp/output.py

@@ -25,7 +25,7 @@ class TCPOutputBot(OutputBot):
     port: int = None
     separator: str = None
 
-    __is_multithreadable = False
+    _is_multithreadable = False
 
     def init(self):
         self.to_intelmq = self.counterpart_is_intelmq

intelmq/bots/outputs/udp/output.py

@@ -19,7 +19,7 @@ class UDPOutputBot(OutputBot):
     udp_host: str = "localhost"
     udp_port: int = None
 
-    __is_multithreadable = False
+    _is_multithreadable = False
 
     def init(self):
         self.delimiter = self.field_delimiter

intelmq/lib/bot.py

@@ -109,9 +109,9 @@ class Bot(object):
     # True for (non-main) threads of a bot instance
     is_multithreaded: bool = False
     # True if the bot is thread-safe and it makes sense
-    __is_multithreadable: bool = True
+    _is_multithreadable: bool = True
     # Collectors with an empty process() should set this to true, prevents endless loops (#1364)
-    __collector_empty_process: bool = False
+    _collector_empty_process: bool = False
 
     _harmonization: dict = {}
 
@@ -165,11 +165,11 @@ def __init__(self, bot_id: str, start: bool = False, sighup_event=None,
 
             broker = self.source_pipeline_broker.title()
             if broker != 'Amqp':
-                self.__is_multithreadable = False
+                self._is_multithreadable = False
 
             """ Multithreading """
             if (self.instances_threads > 1 and not self.is_multithreaded and
-               self.__is_multithreadable and not disable_multithreading):
+               self._is_multithreadable and not disable_multithreading):
                 self.logger.handlers = []
                 num_instances = int(self.instances_threads)
                 instances = []
@@ -196,7 +196,7 @@ def handle_sighup_signal_threading(signum: int,
                     thread.join()
                 return
             elif (getattr(self, 'instances_threads', 1) > 1 and
-                  not self.__is_multithreadable):
+                  not self._is_multithreadable):
                 self.logger.error('Multithreading is configured, but is not '
                                   'available for this bot. Look at the FAQ '
                                   'for a list of reasons for this. '
@@ -446,7 +446,7 @@ def start(self, starting: bool = True, error_on_pipeline: bool = True,
                 if do_rate_limit:
                     if self.rate_limit and self.run_mode != 'scheduled':
                         self.__sleep()
-                    if self.__collector_empty_process and self.run_mode != 'scheduled':
+                    if self._collector_empty_process and self.run_mode != 'scheduled':
                         self.__sleep(1, log=False)
 
             self.__stats()
@@ -1163,7 +1163,7 @@ class CollectorBot(Bot):
     """
 
     bottype = BotType.COLLECTOR
-    __is_multithreadable: bool = False
+    _is_multithreadable: bool = False
     name: Optional[str] = None
     accuracy: int = 100
     code: Optional[str] = None
@@ -1221,7 +1221,7 @@ def send_message(self, *messages, path: str = "_default", auto_add: bool = True)
         super().send_message(*messages, path=path)
 
     def new_report(self):
-        return libmessage.Report()
+        return libmessage.Report(harmonization=self.harmonization)
 
 
 class ExpertBot(Bot):

intelmq/lib/upgrades.py

@@ -749,6 +749,7 @@ def v301_deprecations(configuration, harmonization, dry_run, **kwargs):
     ((2, 3, 3), (v233_feodotracker_browse, )),
     ((3, 0, 0), (v300_bots_file_removal, v300_defaults_file_removal, v300_pipeline_file_removal, )),
     ((3, 0, 1), (v301_deprecations, )),
+    ((3, 0, 2), ()),
     ((3, 1, 0), ()),
 ])