ENH: Shodan parser: handle invalid hostnames, move _common_keys out of class, adjust apply_mapping #2117
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes some changes to the Shodan parser, namely:
extra.ftp.<something>.parameters
, FTP parameters are collected together intoextra.ftp.features
as a list of said features, reducing field countrsync.modules
is collectedNoValueException
with a string argument to signify that the conversion would not succeed, such as in the case of a single IP address being given inhostnames
, which would then be passed intosource.reverse_dns
and fail to validate as a FQDN_common_keys
is moved out of the class_dict_dict_to_obj_list
is introduced, for converting a string-to-dict mapping into a list of dicts with the previous key as an attribute of the dict; this can be useful for preventing issues where, when feeding the data into aggregating tools, you'd end up with many more fields than necessary, e.gvulns.CVE-2010-0001.cvss
,CVE-2010-0002.cvss
etc._get_first
to get the first item from a list, withNoValueException
raised on empty lists_get_first_hostname
to handle the first valid FQDN from a list of hostnames forhostnames
in the Shodan banner, if there is one, and givesNoValueException
otherwisessl.cert.serial
andssl.dhparams.generator
, which may return both integers and strings, are converted to stringsapply_mapping
, such as reducing needless loop iterations, removing a big try-except, and adding theNoValueException
handling described aboveFalse
, 0) besidesNone
from being filtered out