Shadowserver parser: determine feed by report's extra fields

[ghost]

HTTP, RT and Mail collectors provide the file name in a field (extra.file_name). This can be used to determine the feedname and thus would obsolete having lots of different shadowserver parsers with different feedname parameters.

[grodriguezl]

I'm working on this. I will assume the following:

1. report's field extra.file_name is not modified (the same as if I download the file manually), so it could be something like 2019-09-05-botnet_drone-$suffix.csv. The important value here is botnet_drone which is after the date and before any other '-'

Parse function will determine which config has to be used to process that report.

Any comments or ideas are appreciated

[e3rd]

Thanks!

These are the file name → feed name mapping I've been able to guess. I wonder if there is a complete list somewhere.
(list updated thanks to @grodriguezl )

feeds = [
"Blacklisted-IP",
"Microsoft-Sinkhole",
"Open-Elasticsearch",
"Open-Memcached",
"Accessible-HTTP",
"Accessible-ADB",
"Accessible-AFP",
"Accessible-Cisco-Smart-Install",
"Accessible-CWMP",
"Accessible-FTP",
"Accessible-RDP",
"Accessible-Rsync",
"Accessible-SMB",
"Accessible-Telnet",
"Accessible-Ubiquiti-Discovery-Service",
"Accessible-VNC",
"Amplification-DDoS-Victim",
"DNS-Open-Resolvers",
"Drone-Brute-Force",
"Drone",
"NTP-Monitor",
"NTP-Version",
"Open-Chargen",
"Open-IPMI",
"Open-LDAP",
"Open-mDNS",
"Open-MongoDB",
"Open-MSSQL",
"Open-NATPMP",
"Open-NetBIOS-Nameservice",
"Open-Portmapper",
"Open-QOTD",
"Open-Redis",
"Open-SNMP",
"Open-SSDP",
"Open-TFTP",
"Sandbox-URL",
"Sinkhole-HTTP-Drone",
"SSL-FREAK-Vulnerable-Servers",
"SSL-POODLE-Vulnerable-Servers",
"Vulnerable-ISAKMP"]

files= [
"blacklist",
"microsoft_sinkhole",
"scan_elasticsearch",
"scan_memcached",
"scan_http",
"scan_adb",
"scan_afp",
"cisco_smart_install",
"scan_cwmp",
"scan_ftp",
"scan_rdp",
"scan_rsync",
"scan_smb",
"scan_telnet",
"scan_ubiquiti",
"scan_vnc",
"ddos_amplification",
"scan_dns",
"drone_brute_force",
"botnet_drone",
"scan_ntpmonitor",
"scan_ntp",
"scan_chargen",
"scan_ipmi",
"scan_ldap_tcp",
"scan_mdns",
"scan_mongodb",
"scan_mssql",
"scan_nat_pmp",
"scan_netbios",
"scan_portmapper",
"scan_qotd",
"scan_redis",
"scan_snmp",
"scan_ssdp",
"scan_tftp",
"cwsandbox_url",
"sinkhole_http_drone",
"scan_ssl_freak",
"scan_ssl_poodle",
"scan_isakmp"]

@CSIRTMalta

[grodriguezl]

@e3rd add these to that list :)

Blacklisted-IP: blacklist
Accessible-HTTP: scan_http
Microsoft-Sinkhole: microsoft_sinkhole
Open-Elasticsearch: scan_elasticsearch
Open-Memcached: scan_memcached

These are still missing (for me)

"Open-DB2-Discovery-Service"
"Accessible-Hadoop"
"Compromised-Website"
"HTTP-Scanners"
"ICS-Scanners"
"IPv6-Sinkhole-HTTP-Drone"
"Darknet"
"Open-Netis"
"Open-XDMCP"
"Outdated-DNSSEC-Key"
"Outdated-DNSSEC-Key-IPv6"
"Spam-URL"

[ghost]

For reports are too big to get attached, they need to be downloaded. Currently the file name field does not exist in these cases, so I'll adapt the collectors to extract the file name from the HTTP Response Headers.

[ghost]

It's btw ok if the mapping is not complete, we can always make adaptions and extensions to it as soon as we know more file name patterns.

[e3rd]

I updated the list in the top. For future reference I publish here a gist that I generated collectors and parsers to runtime.conf and pipeline.conf with:

https://gist.github.com/e3rd/faf219c50136246616bf76df18750957

(It would take a thousand of clicks in the IntelMQ-Manager otherwise.)

[e3rd]

(Isn't it so that file collector puts the filename to feed.url? Don't we end up in having filename in both extra.file_name and feed.url?)

[ghost]

(Isn't it so that file collector puts the filename to feed.url? Don't we end up in having filename in both extra.file_name and feed.url?)

Maybe in some cases they are equally useful, but they can differ: the extra.file_name is the name of the extracted file in case of archives. So you can have multiple reports with the same feed.url and different extra.file_names.

[ghost]

Concerning the feed name - file name mapping the last one missing is Open-Netis, I added the mapping for all the others. I will add Open-Netis later, I asked a friendly organziation for the file name