Shadowserver 202207 #2227
Merged
Shadowserver 202207 #2227
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sebix
reviewed
Aug 1, 2022
| This module is maintained by [The Shadowserver Foundation](https://www.shadowserver.org/). | ||
|
|
||
| Please contact [email protected] with any issues or concerns. | ||
|
|
Codecov Report
@@ Coverage Diff @@
## develop #2227 +/- ##
===========================================
+ Coverage 76.29% 76.50% +0.21%
===========================================
Files 439 454 +15
Lines 23670 23993 +323
Branches 3751 3787 +36
===========================================
+ Hits 18058 18355 +297
- Misses 4872 4888 +16
- Partials 740 750 +10
|
|
I squashed the changes into a small number of commits. |
sebix
reviewed
Aug 2, 2022
sebix
reviewed
Aug 2, 2022
This PR has 95 commits? |
I did a The git log shows them as follows: |
|
GitHub says 95 commits |
|
What would you suggest? |
then squash various commit together and |
author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659452139 +0000 parent ba7d24b author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659452135 +0000 parent ba7d24b author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659452132 +0000 parent ba7d24b author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659452116 +0000 parent ba7d24b author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659452111 +0000 parent ba7d24b author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659452106 +0000 parent ba7d24b author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659451910 +0000 parent ba7d24b author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659451905 +0000 parent ba7d24b author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659451900 +0000 parent ba7d24b author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659451775 +0000 Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192 parent 6194014 author elsif2 <[email protected]> 1659389509 +0000 committer elsif2 <[email protected]> 1659392927 +0000 parent 6194014 author elsif2 <[email protected]> 1659389509 +0000 committer elsif2 <[email protected]> 1659392915 +0000 parent 6194014 author elsif2 <[email protected]> 1659389509 +0000 committer elsif2 <[email protected]> 1659392880 +0000 parent 6194014 author elsif2 <[email protected]> 1659389509 +0000 committer elsif2 <[email protected]> 1659392820 +0000 parent 6194014 author elsif2 <[email protected]> 1659389509 +0000 committer elsif2 <[email protected]> 1659392766 +0000 parent 6194014 author elsif2 <[email protected]> 1659389509 +0000 committer elsif2 <[email protected]> 1659392468 +0000 parent 6194014 author elsif2 <[email protected]> 1659389509 +0000 committer elsif2 <[email protected]> 1659392457 +0000 parent 6194014 author elsif2 <[email protected]> 1659389509 +0000 committer elsif2 <[email protected]> 1659392039 +0000 Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192 Replaced misleading _country_ parameter with _reports_. Update parser to support all available reports. Update to existing test cases to match current report types. pycodestyle fixes add testdata licenses pycodestyle fix Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Realign columns Remove duplicates Changed malware.name to extra.infection DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Fix for recover_line method as commited in #2192 Revert: Fix for recover_line method as commited in #2192 Test case for scan_couchdb Restored feed names and classification.identifiers to minimize upgrade impact. Added function to notify if any unsupported feeds are configured. pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Update intelmq/bots/collectors/shadowserver/collector_reports_api.py Co-authored-by: Sebastian <[email protected]> Update intelmq/bots/collectors/shadowserver/README.md Co-authored-by: Sebastian <[email protected]> Update intelmq/tests/bots/parsers/shadowserver/README.md Co-authored-by: Sebastian <[email protected]> resync to develop parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395284 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395281 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395278 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395264 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395260 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395256 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395141 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395131 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395127 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395122 +0000 parent 7dc5b74 author elsif2 <[email protected]> 1643216571 +0000 committer elsif2 <[email protected]> 1659395058 +0000 Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192 resolve conflict Replaced misleading _country_ parameter with _reports_. Update parser to support all available reports. Update to existing test cases to match current report types. New tests for added report types. pycodestyle fixes add testdata licenses pycodestyle fix Added reports parameter Suggested changes to the parser Proposed details for the release Test script updates for suggested changes Test input updates Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Update scan_snmp.csv Realign columns Remove duplicates Changed malware.name to extra.infection Updated SPDX-FileCopyrightText shadowserver api: document and warn on old parameter document the old parameter `country` and its status warn if used adapt the test DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright and raw field updates Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Addeed event4_honeypot_ddos, event4_honeypot_ddos_target, scan_dvr_dhcpdiscover, and scan_socks. Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file update:scan_mdns, scan_smb, and special; add:scan_ddos_middle_box cleanup renamed license files updated scan_mdns test files updated scan_smb test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test updated schema Updated scan_smb tests Updated scan_ntp tests Updated scan_snmp tests New scan_docker test New scan_kubernetes test New scan_mysql test Updated report schema for June 2022 Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Revert: Fix for recover_line method as commited in #2192 Added scan_couchdb Test case for scan_couchdb Added scan6_rpd Added/updated README with maintainer details Restored feed names and classification.identifiers to minimize upgrade impact. Merge repair pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Added additional IPv6 aliases Fix for recover_line method as commited in #2192 Update to existing test cases to match current report types. add testdata licenses pycodestyle fix Proposed details for the release Test script updates for suggested changes Realign columns Update compromised_website.csv Update scan_adb.csv Update scan_ftp.csv Update scan_ipp.csv Realign columns Remove duplicates DOC: fix NEWS entry of PR#2143 Added the sector field to scan_amqp, scan_cwmp, and scan_vnc. Copyright updates Added phish_url and scan_modbus reports. Update source.url and source.fqdn for phish_url and malware_url reports. Update classification.taxonomy and classification.type for scan_modbus report. * additional field type validation changes * added count, bytes, duration, avg_pps, and max_pps fields to event_honeypot_ddos_amp * added 'protocol.application': 'https' to scan_ssl, scan_ssl_freak, and scan_ssl_poodle * added 'extra.tag' to scan_* and device_id Replaced scan_modbus with scan_ics Tests for event4_honeypot_ddos. Tests for event4_honeypot_ddos_target. Tests for scan_dvr_dhcpdiscover. Tests for scan_socks. Rename file Rename file cleanup renamed license files updated scan_mdns test files updated special test files add scan_ddos_middlebox test files add scan_ddos_middlebox test New scan_docker test New scan_kubernetes test New scan_mysql test Added scan_epmd test Revert "Added scan_epmd test" This reverts commit 01edea1. Fix for recover_line method as commited in #2192 Revert: Fix for recover_line method as commited in #2192 Test case for scan_couchdb Restored feed names and classification.identifiers to minimize upgrade impact. Added function to notify if any unsupported feeds are configured. pycodestyle repairs codespell fixes license compliance fixes pycodestyle fixes Feed configuration updates for compatibility with the original. Added scan_postgres test Update intelmq/bots/collectors/shadowserver/collector_reports_api.py Co-authored-by: Sebastian <[email protected]> Update intelmq/bots/collectors/shadowserver/README.md Co-authored-by: Sebastian <[email protected]> Update intelmq/tests/bots/parsers/shadowserver/README.md Co-authored-by: Sebastian <[email protected]> resync to develop
elsif2
force-pushed
the
shadowserver-202207
branch
from
dad59a7 to
1e4a16c
Compare
Done:
|
|
Cool, looks tidy now :) |
|
Agreement is that I merge it in, if things break, @elsif2 will fix it quickly. Thanks @elsif2 for your enormous heavy lifting and improvements! |
wagner-intevation
added a commit
to wagner-intevation/intelmq
that referenced
this pull request
Sep 19, 2022
Add an example configuration for the modify bot. It reverts the changes of classification.identifier values in the ShadowServer parser bot effective in IntelMQ 3.1. see also certtools#2227
wagner-intevation
added a commit
to wagner-intevation/intelmq
that referenced
this pull request
Sep 20, 2022
Add an example configuration for the modify bot. It reverts the changes of classification.identifier values in the ShadowServer parser bot effective in IntelMQ 3.1. see also certtools#2227
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Large set of changes previously submitted as 2143.