ENH: ctip parser: added parameter overwrite

intelmq.bots.parsers.microsoft.parser_ctip: New parameter `overwrite` (PR#2112 by Sebastian Wagner, fixes #2022).

for azure source: only affects feed.name
for interflow source: no change

fixes #2022

CHANGELOG.md

@@ -50,6 +50,7 @@ CHANGELOG
   - SMTP Data
   - Telnet Login
   - VNC/RFB Login
+- `intelmq.bots.parsers.microsoft.parser_ctip`: New parameter `overwrite` (PR#2112 by Sebastian Wagner, fixes #2022).
 
 #### Experts
 - `intelmq.bots.experts.domain_valid`: New bot for checking domain's validity (PR#1966 by Marius Karotkis).

docs/user/bots.rst

@@ -1464,6 +1464,12 @@ Microsoft CTIP Parser
 * `cache (redis db)`: none
 * `description`: Parses data from the Microsoft CTIP Feed
 
+ * `overwrite`: If an existing `feed.name` should be overwritten (only relevant for the azure data source).
+
+**Configuration Parameters**
+
+* ``overwrite``: Overwrite an existing field ``feed.name`` with ``DataFeed`` of the source.
+
 **Description**
 
 Can parse the JSON format provided by the Interflow interface (lists of dictionaries) as well as the format provided by the Azure interface (one dictionary per line).
@@ -1911,7 +1917,7 @@ Public documentation: https://www.team-cymru.com/IP-ASN-mapping.html#dns
 **Configuration Parameters**
 
 * **Cache parameters** (see in section :ref:`common-parameters`)
-* `overwrite`: Overwrite existing fields. Default: `True` if not given (for backwards compatibility, will change in version 3.0.0)
+* ``: Overwrite existing fields. Default: `True` if not given (for backwards compatibility, will change in version 3.0.0)
 
 
 .. _intelmq.bots.experts.domain_suffix.expert:

intelmq/bots/parsers/microsoft/parser_ctip.py

@@ -205,6 +205,7 @@
 
 class MicrosoftCTIPParserBot(ParserBot):
     """Parse JSON data from Microsoft's CTIP program"""
+    overwrite: bool = True  # overwrite existing fields
 
     def parse(self, report):
         raw_report = utils.base64_decode(report.get("raw"))
@@ -288,11 +289,13 @@ def parse_azure(self, line, report):
             elif key == 'Payload.Protocol':
                 payload_protocol = value[:value.find('/')]
                 if payload_protocol:
+                    # needs to overwrite a field previously parsed and written
                     event.add('protocol.application', payload_protocol, overwrite=True)  # "HTTP/1.1", save additionally
             elif not value:
                 continue
             if AZURE[key] != '__IGNORE__':
-                event.add(AZURE[key], value, overwrite=True)
+                # feed.accuracy is calculated newly and always needs to be overwritten
+                event.add(AZURE[key], value, overwrite=self.overwrite or AZURE[key] == "feed.accuracy")
         event.add('classification.type', 'infected-system')
         event.add('raw', raw)
         yield event

intelmq/tests/bots/parsers/microsoft/test_parser_ctip_azure.py

@@ -20,6 +20,7 @@
 
 EXAMPLE_REPORT = {
     "__type": "Report",
+    "feed.name": "ctip",
     "feed.accuracy": 100.0,
     "time.observation": "2016-06-15T09:25:26+00:00",
     "raw": base64_encode(EXAMPLE_DATA)
@@ -174,6 +175,14 @@ def test_event(self):
         for i in range(4):
             self.assertMessageEqual(i, EXAMPLE_EVENTS[i])
 
+    def test_not_overwrite(self):
+        """ Test with overwrite=False """
+        self.run_bot(parameters={'overwrite': False})
+        for i, event in enumerate(EXAMPLE_EVENTS):
+            tmp = event.copy()
+            tmp["feed.name"] = "ctip"
+            self.assertMessageEqual(i, tmp)
+
 
 if __name__ == '__main__':  # pragma: no cover
     unittest.main()