parent 7dc5b74

author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395284 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395281 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395278 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395264 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395260 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395256 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395141 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395131 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395127 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395122 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395058 +0000 Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192
certtools · Aug 1, 2022 · 67a52fe · 67a52fe
1 parent 7dc5b74
commit 67a52fe
Show file tree

Hide file tree

Showing 3 changed files with 35 additions and 53 deletions.
diff --git a/NEWS.md b/NEWS.md
@@ -9,13 +9,6 @@ NEWS
 This file lists all changes which have an affect on the administration of IntelMQ and contains steps that you need to be aware off for the upgrade.
 Please refer to the changelog for a full list of changes.
 
-<<<<<<< HEAD
-<<<<<<< HEAD
-=======
-TBD Shadowserver updates
-------------------------
-=======
->>>>>>> 90f08224... DOC: fix NEWS entry of PR#2143
 
 3.1.0 Feature release (unreleased)
 ----------------------------------
@@ -28,18 +21,40 @@ The misleading `country` parameter has been depreciated and a `reports` paramete
 The backwards-compatibility will be removed in IntelMQ version 4.0.0.
 See the [Shadowserver Reports API bot's documentation](https://intelmq.readthedocs.io/en/latest/user/bots.html#shadowserver-reports-api).
 
-<<<<<<< HEAD
-
-3.1.0 Feature release (unreleased)
-----------------------------------
-
-### Requirements
-
-### Bots
-#### ShadowServer Reports API collector
-The misleading `country` parameter has been depreciated and a `reports` parameter has been added.
-The backwards-compatibility will be removed in IntelMQ version 4.0.0.
-See the [Shadowserver Reports API bot's documentation](https://intelmq.readthedocs.io/en/latest/user/bots.html#shadowserver-reports-api).
+#### ShadowServer parser
+Previously, mappings used a mix of `extra.naics` and `extra.source.naics`. The parser has been updated to use the more specific term (`extra.source.naics`).
+
+A number of the _classification.identifier_ values have been updated to follow a common naming convention based on their canonical report name:
+
+| before IntelMQ 3.1.0 | in IntelMQ 3.1.0 and higher |
+| --- | --- |
+| accessible-adb | open-adb |
+| accessible-afp | open-afp |
+| accessible-amqp | open-amqp |
+| accessible-ard | open-ard |
+| accessible-cisco-smart-install | open-cisco-smart-install |
+| accessible-coap | open-coap |
+| accessible-ftp | open-ftp |
+| accessible-hadoop | open-hadoop |
+| accessible-http | open-http |
+| accessible-msrdpeudp | open-rdpeudp |
+| accessible-radmin | open-radmin |
+| accessible-rsync | open-rsync |
+| accessible-ubiquiti-discovery-service | open-ubiquiti |
+| amplification-ddos-victim | honeypot-ddos-amp |
+| blacklisted-ip | blocklist |
+| dns-open-resolver | open-dns |
+| honeypot-http-scan | honeypot-http-scan |
+| ics | honeypot-ics-scan |
+| ntp-monitor | open-ntpmonitor |
+| ntp-version | open-ntp |
+| open-db2-discovery-service | open-db2 |
+| open-ike | open-isakmp |
+| open-ldap | open-ldap-tcp |
+| open-natpmp | open-nat-pmp |
+| open-netbios-nameservice | open-netbios |
+| open-netis | open-netis-router |
+| sinkholedns | sinkhole-dns |
 
 ### Tools
 
@@ -148,15 +163,12 @@ UPDATE events
    SET "classification.identifier" = 'sinkhole-dns'
    WHERE "classification.identifier" = 'sinkholedns';
 ```
-<<<<<<< HEAD
 
 
 ### Bots
 
 #### Github Collector
 GitHub removed the basic `Username/Password` Authentication in favor of personal access tokens. So the GitHub Collector uses an Personal Access Token for authentication [Github Documentation: Generate a personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
-=======
->>>>>>> 90f08224... DOC: fix NEWS entry of PR#2143
 
 
 3.0.2 Maintenance release (2021-09-10)

diff --git a/intelmq/bots/parsers/shadowserver/parser.py b/intelmq/bots/parsers/shadowserver/parser.py
@@ -193,7 +193,7 @@ def parse_line(self, row, report):
         # Now add additional constant fields.
         event.update(conf.get('constant_fields', {}))
 
-        event.add('raw', self.recover_line(row))
+        event.add('raw', self.recover_line())
 
         # Add everything which could not be resolved to extra.
         for f in fields:

diff --git a/intelmq/lib/upgrades.py b/intelmq/lib/upgrades.py
@@ -757,36 +757,6 @@ def v310_shadowserver_feednames(configuration, harmonization, dry_run, **kwargs)
     return 'A discontinued feed has been found and must be removed %s' % ', '.join(names) if names else changed, configuration, harmonization
 
 
-def v310_shadowserver_feednames(configuration, harmonization, dry_run, **kwargs):
-    """
-    Remove legacy Shadowserver feednames
-    """
-    legacy = {
-        'Amplification-DDoS-Victim': 1,
-        'Blacklisted-IP': 1,
-        'CAIDA-IP-Spoofer': 1,
-        'Darknet': 1,
-        'Drone': 1,
-        'Drone-Brute-Force': 1,
-        'HTTP-Scanners': 1,
-        'ICS-Scanners': 1,
-        'IPv6-Sinkhole-HTTP-Drone': 1,
-        'Microsoft-Sinkhole': 1,
-        'Outdated-DNSSEC-Key': 1,
-        'Outdated-DNSSEC-Key-IPv6': 1,
-        'Sinkhole-HTTP-Drone': 1
-    }
-    changed = None
-    names = []
-    for bot_id, bot in configuration.items():
-        if bot_id == 'global':
-            continue
-        if bot["module"] == "intelmq.bots.parsers.shadowserver.parser":
-            if bot["parameters"]["feedname"] in legacy:
-                names.append(bot["parameters"]["feedname"])
-    return 'A discontinued feed has been found and must be removed %s' % ', '.join(names) if names else changed, configuration, harmonization
-
-
 def v310_feed_changes(configuration, harmonization, dry_run, **kwargs):
     """
     Migrates feeds' configuration for changed/fixed parameter