Skip to content

Commit

Permalink
Shadowserver feed updates
Browse files Browse the repository at this point in the history 
Microsoft Sinkhole Report has been replaced by two reports: Microsoft
Sinkhole Events Report and Microsoft Sinkhole HTTP Events Report.

Added the config, added the tests, csv files, license files and
mentioned the changes in the CHANGELOG.

Fixes: #2019
  • Loading branch information
Birger Schacht authored and Birger Schacht committed Aug 16, 2021
1 parent 103a584 commit 9d4baa8
Show file tree
Hide file tree
Showing 9 changed files with 343 additions and 0 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ CHANGELOG
#### Parsers
- `intelmq.bots.parsers.microsoft.parser_ctip`: Map `Payload.domain` to `destination.fqdn` instead of `extra.payload.domain` as it matches to `destination.ip` from `DestinationIp` (PR#2023 by Sebastian Wagner).
- Removed `intelmq.bots.parsers.malwaredomains` because the upstream data source (malwaredomains.com) does not exist anymore (PR#2026 by Birger Schacht, fixes #2024).
- `intelmq.bots.parsers.shadowserver._config`:
- Add support for the new feeds *Microsoft Sinkhole Events Report*, *Microsoft Sinkhole HTTP Events Report* (PR#2036 by Birger Schacht).

#### Experts
- `intelmq.bots.experts.truncate_by_delimiter.expert`: Cut string if its length is higher than a maximum length (PR#1967 by Marius Karotkis).
Expand Down
2 changes: 2 additions & 0 deletions docs/user/bots.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1627,6 +1627,8 @@ These are the supported feed name and their corresponding file name for automati
HTTP-Scanners `hp_http_scan`
ICS-Scanners `hp_ics_scan`
IP-Spoofer-Events `event4_ip_spoofer`
Microsoft-Sinkhole-Events IPv4 `event4_microsoft_sinkhole`
Microsoft-Sinkhole-Events-HTTP IPv4 `event4_microsoft_sinkhole_http`
NTP-Monitor `scan_ntpmonitor`
NTP-Version `scan_ntp`
Open-Chargen `scan_chargen`
Expand Down
2 changes: 2 additions & 0 deletions intelmq/bots/parsers/shadowserver/_config.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2849,6 +2849,8 @@ def scan_exchange_identifier(field):
('IPv6-Sinkhole-HTTP-Drone', 'sinkhole6_http', ipv6_sinkhole_http_drone), # legacy (replaced by event46_sinkhole_http)
('IP-Spoofer-Events', 'event4_ip_spoofer', event4_ip_spoofer),
('Microsoft-Sinkhole', 'microsoft_sinkhole', microsoft_sinkhole), # legacy (replaced by event46_sinkhole_http)
('Microsoft-Sinkhole-Events IPv4', 'event4_microsoft_sinkhole', event46_sinkhole),
('Microsoft-Sinkhole-Events-HTTP IPv4', 'event4_microsoft_sinkhole_http', event46_sinkhole_http),
('NTP-Monitor', 'scan_ntpmonitor', ntp_monitor),
('NTP-Version', 'scan_ntp', ntp_version),
('Open-Chargen', 'scan_chargen', open_chargen),
Expand Down
128 changes: 128 additions & 0 deletions intelmq/tests/bots/parsers/shadowserver/test_event4_microsoft_sinkhole.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
# SPDX-FileCopyrightText: 2021 Birger Schacht
#
# SPDX-License-Identifier: AGPL-3.0-or-later

# -*- coding: utf-8 -*-

import os
import unittest

import intelmq.lib.test as test
import intelmq.lib.utils as utils
from intelmq.bots.parsers.shadowserver.parser import ShadowserverParserBot

with open(os.path.join(os.path.dirname(__file__),
'testdata/event4_microsoft_sinkhole.csv')) as handle:
EXAMPLE_FILE = handle.read()
EXAMPLE_LINES = EXAMPLE_FILE.splitlines()

EXAMPLE_REPORT = {"feed.name": "ShadowServer Microsoft Sinkhole",
"raw": utils.base64_encode(EXAMPLE_FILE),
"__type": "Report",
"time.observation": "2015-01-01T00:00:00+00:00",
"extra.file_name": "2019-01-01-event4_microsoft_sinkhole.csv",
}
EVENTS = [{'__type': 'Event',
'classification.identifier': 'b68-zeroaccess-2-32bit',
'classification.taxonomy': 'malicious-code',
'classification.type': 'infected-system',
'destination.asn': 8075,
'destination.geolocation.cc': 'HK',
'destination.geolocation.city': 'HONG KONG',
'destination.geolocation.region': 'HONG KONG',
'destination.ip': '168.63.134.179',
'destination.port': 16464,
'extra.destination.naics': 334111,
'extra.destination.sector': 'Information',
'extra.public_source': 'MSDCU',
'extra.source.naics': 517311,
'feed.name': 'ShadowServer Microsoft Sinkhole',
'malware.name': 'b68-zeroaccess-2-32bit',
'protocol.transport': 'tcp',
'source.asn': 7303,
'source.geolocation.cc': 'AR',
'source.geolocation.city': 'CASEROS',
'source.geolocation.region': 'BUENOS AIRES',
'source.ip': '190.229.1.2',
'source.port': 52955,
'time.source': '2021-06-07T00:00:00+00:00',
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
EXAMPLE_LINES[1]])),
},
{'__type': 'Event',
'classification.identifier': 'b68-zeroaccess-2-32bit',
'classification.taxonomy': 'malicious-code',
'classification.type': 'infected-system',
'destination.asn': 8075,
'destination.geolocation.cc': 'IE',
'destination.geolocation.city': 'DUBLIN',
'destination.geolocation.region': 'DUBLIN',
'destination.ip': '52.169.3.4',
'destination.port': 16464,
'extra.destination.naics': 334111,
'extra.destination.sector': 'Information',
'extra.public_source': 'MSDCU',
'extra.source.naics': 517311,
'extra.source.sector': 'Communications, Service Provider, and Hosting Service',
'feed.name': 'ShadowServer Microsoft Sinkhole',
'malware.name': 'b68-zeroaccess-2-32bit',
'protocol.transport': 'tcp',
'source.asn': 5769,
'source.geolocation.cc': 'CA',
'source.geolocation.city': 'LAVAL',
'source.geolocation.region': 'QUEBEC',
'source.ip': '96.20.3.4',
'source.port': 16464,
'time.source': '2021-06-07T00:00:00+00:00',
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
EXAMPLE_LINES[2]])),
},
{'__type': 'Event',
'classification.identifier': 'b68-zeroaccess-2-32bit',
'classification.taxonomy': 'malicious-code',
'classification.type': 'infected-system',
'destination.asn': 8075,
'destination.geolocation.cc': 'HK',
'destination.geolocation.city': 'HONG KONG',
'destination.geolocation.region': 'HONG KONG',
'destination.ip': '168.63.134.179',
'destination.port': 16464,
'extra.destination.naics': 334111,
'extra.destination.sector': 'Information',
'extra.public_source': 'MSDCU',
'extra.source.naics': 517311,
'feed.name': 'ShadowServer Microsoft Sinkhole',
'malware.name': 'b68-zeroaccess-2-32bit',
'protocol.transport': 'tcp',
'source.asn': 8151,
'source.geolocation.cc': 'MX',
'source.geolocation.city': 'MEXICO CITY',
'source.geolocation.region': "CIUDAD DE MEXICO",
'source.ip': '187.222.5.6',
'source.port': 55049,
'time.source': '2021-06-07T00:00:00+00:00',
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
EXAMPLE_LINES[3]])),
},
]


class TestShadowserverParserBot(test.BotTestCase, unittest.TestCase):
"""
A TestCase for a ShadowserverParserBot.
"""

@classmethod
def set_bot(cls):
cls.bot_reference = ShadowserverParserBot
cls.default_input_message = EXAMPLE_REPORT

def test_event(self):
""" Test if correct Event has been produced. """
self.run_bot()
for i, EVENT in enumerate(EVENTS):
self.assertMessageEqual(i, EVENT)


if __name__ == '__main__': # pragma: no cover
unittest.main()
192 changes: 192 additions & 0 deletions intelmq/tests/bots/parsers/shadowserver/test_event4_microsoft_sinkhole_http.py
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
# SPDX-FileCopyrightText: 2021 Birger Schacht
#
# SPDX-License-Identifier: AGPL-3.0-or-later

# -*- coding: utf-8 -*-

import os
import unittest

import intelmq.lib.test as test
import intelmq.lib.utils as utils
from intelmq.bots.parsers.shadowserver.parser import ShadowserverParserBot

with open(os.path.join(os.path.dirname(__file__),
'testdata/event4_microsoft_sinkhole_http.csv')) as handle:
EXAMPLE_FILE = handle.read()
EXAMPLE_LINES = EXAMPLE_FILE.splitlines()

EXAMPLE_REPORT = {'feed.name': 'HTTP Microsoft Sinkhole IPv4',
"raw": utils.base64_encode(EXAMPLE_FILE),
"__type": "Report",
"time.observation": "2015-01-01T00:00:00+00:00",
"extra.file_name": "2019-01-01-event4_microsoft_sinkhole_http.csv",
}
EVENTS = [{'__type': 'Event',
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
'classification.identifier': 'necurs',
'classification.taxonomy': 'malicious-code',
'classification.type': 'infected-system',
'destination.asn': 8075,
'destination.geolocation.cc': 'US',
'destination.geolocation.city': 'ASHBURN',
'destination.geolocation.region': 'VIRGINIA',
'destination.ip': '40.121.206.97',
'destination.port': 80,
'destination.url': 'http://40.121.206.97/locator.php',
'extra.destination.naics': 334111,
'extra.destination.sector': 'Information',
'extra.public_source': 'MSDCU',
'malware.name': 'necurs',
'protocol.application': 'http',
'protocol.transport': 'tcp',
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
EXAMPLE_LINES[1]])),
'source.asn': 8386,
'source.geolocation.cc': 'TR',
'source.geolocation.city': 'KEPEZ',
'source.geolocation.region': 'ANTALYA',
'source.ip': '31.206.1.2',
'source.port': 49245,
'time.observation': '2015-01-01T00:00:00+00:00',
'time.source': '2021-06-07T00:00:00+00:00'},
{'__type': 'Event',
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
'classification.identifier': 'caphaw',
'classification.taxonomy': 'malicious-code',
'classification.type': 'infected-system',
'destination.asn': 8075,
'destination.fqdn': '3fo8jrthz3y.rgk.cc',
'destination.geolocation.cc': 'US',
'destination.geolocation.city': 'REDMOND',
'destination.geolocation.region': 'WASHINGTON',
'destination.ip': '204.95.99.204',
'destination.port': 443,
'destination.url': 'http://3fo8jrthz3y.rgk.cc/index.php',
'extra.destination.naics': 334111,
'extra.destination.sector': 'Information',
'extra.http_agent': 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.6103)',
'extra.http_referer': 'null',
'extra.public_source': 'MSDCU',
'extra.source.naics': 517312,
'malware.name': 'caphaw',
'protocol.application': 'http',
'protocol.transport': 'tcp',
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
EXAMPLE_LINES[2]])),
'source.asn': 28573,
'source.geolocation.cc': 'BR',
'source.geolocation.city': 'SAO PAULO',
'source.geolocation.region': 'SAO PAULO',
'source.ip': '177.140.3.4',
'source.port': 35919,
'time.observation': '2015-01-01T00:00:00+00:00',
'time.source': '2021-06-07T00:00:00+00:00'},
{'__type': 'Event',
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
'classification.identifier': 'necurs',
'classification.taxonomy': 'malicious-code',
'classification.type': 'infected-system',
'destination.asn': 8075,
'destination.geolocation.cc': 'US',
'destination.geolocation.city': 'ASHBURN',
'destination.geolocation.region': 'VIRGINIA',
'destination.ip': '40.121.206.97',
'destination.port': 80,
'destination.url': 'http://40.121.206.97/locator.php',
'extra.destination.naics': 334111,
'extra.destination.sector': 'Information',
'extra.public_source': 'MSDCU',
'extra.source.naics': 517311,
'malware.name': 'necurs',
'protocol.application': 'http',
'protocol.transport': 'tcp',
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
EXAMPLE_LINES[3]])),
'source.asn': 132199,
'source.geolocation.cc': 'PH',
'source.geolocation.city': 'MANDAUE',
'source.geolocation.region': 'CEBU',
'source.ip': '180.190.5.6',
'source.port': 49264,
'time.observation': '2015-01-01T00:00:00+00:00',
'time.source': '2021-06-07T00:00:01+00:00'},
{'__type': 'Event',
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
'classification.identifier': 'necurs',
'classification.taxonomy': 'malicious-code',
'classification.type': 'infected-system',
'destination.asn': 8075,
'destination.geolocation.cc': 'US',
'destination.ip': '40.121.206.97',
'destination.geolocation.city': 'ASHBURN',
'destination.geolocation.region': 'VIRGINIA',
'destination.port': 80,
'destination.url': 'http://40.121.206.97/news/stream.php',
'extra.destination.naics': 334111,
'extra.destination.sector': 'Information',
'extra.public_source': 'MSDCU',
'malware.name': 'necurs',
'protocol.application': 'http',
'protocol.transport': 'tcp',
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
EXAMPLE_LINES[4]])),
'source.asn': 37129,
'source.geolocation.cc': 'KE',
'source.geolocation.city': 'NAIROBI',
'source.geolocation.region': 'NAIROBI CITY',
'source.ip': '197.157.7.8',
'source.port': 55307,
'time.observation': '2015-01-01T00:00:00+00:00',
'time.source': '2021-06-07T00:00:01+00:00'},
{'__type': 'Event',
'feed.name': 'HTTP Microsoft Sinkhole IPv4',
'classification.identifier': 'necurs',
'classification.taxonomy': 'malicious-code',
'classification.type': 'infected-system',
'destination.asn': 8075,
'destination.geolocation.cc': 'US',
'destination.geolocation.city': 'ASHBURN',
'destination.geolocation.region': 'VIRGINIA',
'destination.ip': '40.121.206.97',
'destination.port': 80,
'destination.url': 'http://40.121.206.97/locator.php',
'extra.destination.naics': 334111,
'extra.destination.sector': 'Information',
'extra.public_source': 'MSDCU',
'extra.source.naics': 517311,
'extra.source.sector': 'Communications, Service Provider, and Hosting Service',
'malware.name': 'necurs',
'protocol.application': 'http',
'protocol.transport': 'tcp',
'raw': utils.base64_encode('\n'.join([EXAMPLE_LINES[0],
EXAMPLE_LINES[5]])),
'source.asn': 812,
'source.geolocation.cc': 'CA',
'source.geolocation.city': 'OTTAWA',
'source.geolocation.region': 'ONTARIO',
'source.ip': '174.114.9.10',
'source.port': 59000,
'time.observation': '2015-01-01T00:00:00+00:00',
'time.source': '2021-06-07T00:00:01+00:00'}]


class TestShadowserverParserBot(test.BotTestCase, unittest.TestCase):
"""
A TestCase for a ShadowserverParserBot.
"""

@classmethod
def set_bot(cls):
cls.bot_reference = ShadowserverParserBot
cls.default_input_message = EXAMPLE_REPORT

def test_event(self):
""" Test if correct Event has been produced. """
self.run_bot()
for i, EVENT in enumerate(EVENTS):
self.assertMessageEqual(i, EVENT)


if __name__ == '__main__': # pragma: no cover
unittest.main()
7 changes: 7 additions & 0 deletions intelmq/tests/bots/parsers/shadowserver/testdata/event4_microsoft_sinkhole.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
"timestamp","protocol","src_ip","src_port","src_asn","src_geo","src_region","src_city","src_hostname","src_naics","src_sector","device_vendor","device_type","device_model","dst_ip","dst_port","dst_asn","dst_geo","dst_region","dst_city","dst_hostname","dst_naics","dst_sector","public_source","infection","family","tag","application","version","event_id"
"2021-06-07 00:00:00","tcp","190.229.1.2",52955,7303,"AR","BUENOS AIRES","CASEROS",,517311,,,,,"168.63.134.179",16464,8075,"HK","HONG KONG","HONG KONG",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
"2021-06-07 00:00:00","tcp","96.20.3.4",16464,5769,"CA","QUEBEC","LAVAL",,517311,"Communications, Service Provider, and Hosting Service",,,,"52.169.3.4",16464,8075,"IE","DUBLIN","DUBLIN",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
"2021-06-07 00:00:00","tcp","187.222.5.6",55049,8151,"MX","CIUDAD DE MEXICO","MEXICO CITY",,517311,,,,,"168.63.134.179",16464,8075,"HK","HONG KONG","HONG KONG",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
"2021-06-07 00:00:00","tcp","75.84.7.8",64190,20001,"US","CALIFORNIA","NORTH HOLLYWOOD",,517311,"Communications, Service Provider, and Hosting Service",,,,"52.169.7.8",16464,8075,"IE","DUBLIN","DUBLIN",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
"2021-06-07 00:00:00","tcp","24.15.9.10",60373,7922,"US","ILLINOIS","HOMER GLEN",,517311,"Communications, Service Provider, and Hosting Service",,,,"104.40.6.5",16464,8075,"US","CALIFORNIA","SAN FRANCISCO",,334111,"Information","MSDCU","b68-zeroaccess-2-32bit",,"b68-zeroaccess-2-32bit",,,
"2021-06-07 00:00:00","tcp","124.101.11.12",50386,4713,"JP","FUKUOKA","FUKUOKA",,517311,"Communications, Service Provider, and Hosting Service",,,,"23.99.101.165",16465,8075,"HK","HONG KONG","HONG KONG",,334111,"Information","MSDCU","b68-zeroaccess-2-64bit",,"b68-zeroaccess-2-64bit",,
2 changes: 2 additions & 0 deletions intelmq/tests/bots/parsers/shadowserver/testdata/event4_microsoft_sinkhole.csv.license
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
SPDX-FileCopyrightText: 2021 Birger Schacht
SPDX-License-Identifier: AGPL-3.0-or-later
6 changes: 6 additions & 0 deletions intelmq/tests/bots/parsers/shadowserver/testdata/event4_microsoft_sinkhole_http.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"timestamp","protocol","src_ip","src_port","src_asn","src_geo","src_region","src_city","src_hostname","src_naics","src_sector","device_vendor","device_type","device_model","dst_ip","dst_port","dst_asn","dst_geo","dst_region","dst_city","dst_hostname","dst_naics","dst_sector","public_source","infection","family","tag","application","version","event_id","http_url","http_host","http_agent","forwarded_by","ssl_cipher","http_referer"
"2021-06-07 00:00:00","tcp","31.206.1.2",49245,8386,"TR","ANTALYA","KEPEZ",,,,,,,"40.121.206.97",80,8075,"US","VIRGINIA","ASHBURN",,334111,"Information","MSDCU","necurs",,"necurs",,,,"/locator.php","40.121.206.97",,,,
"2021-06-07 00:00:00","tcp","177.140.3.4",35919,28573,"BR","SAO PAULO","SAO PAULO",,517312,,,,,"204.95.99.204",443,8075,"US","WASHINGTON","REDMOND",,334111,"Information","MSDCU","caphaw",,"caphaw",,,,"/index.php","3fo8jrthz3y.rgk.cc","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.6103)",,,"null"
"2021-06-07 00:00:01","tcp","180.190.5.6",49264,132199,"PH","CEBU","MANDAUE",,517311,,,,,"40.121.206.97",80,8075,"US","VIRGINIA","ASHBURN",,334111,"Information","MSDCU","necurs",,"necurs",,,,"/locator.php","40.121.206.97",,,,
"2021-06-07 00:00:01","tcp","197.157.7.8",55307,37129,"KE","NAIROBI CITY","NAIROBI",,,,,,,"40.121.206.97",80,8075,"US","VIRGINIA","ASHBURN",,334111,"Information","MSDCU","necurs",,"necurs",,,,"/news/stream.php","40.121.206.97",,,,
"2021-06-07 00:00:01","tcp","174.114.9.10",59000,812,"CA","ONTARIO","OTTAWA",,517311,"Communications, Service Provider, and Hosting Service",,,,"40.121.206.97",80,8075,"US","VIRGINIA","ASHBURN",,334111,"Information","MSDCU","necurs",,"necurs",,,,"/locator.php","40.121.206.97",,,,
2 changes: 2 additions & 0 deletions intelmq/tests/bots/parsers/shadowserver/testdata/event4_microsoft_sinkhole_http.csv.license
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
SPDX-FileCopyrightText: 2021 Birger Schacht
SPDX-License-Identifier: AGPL-3.0-or-later

0 comments on commit 9d4baa8

Please sign in to comment.