Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IDF: Fix and enforce field name restrictions #2059

Merged
6 commits merged into from
Aug 23, 2021
Merged

IDF: Fix and enforce field name restrictions #2059

6 commits merged into from
Aug 23, 2021

Conversation

ghost
Copy link

@ghost ghost commented Aug 20, 2021

lib/message: fix and pre-compile data format key regex
the regular expression, which is used to check the validity of
harmonization key names, was never effective as the dot wasn't escaped.
Additionally, the characters of the first element were not allowed to
repeat.

add github action for regexploit
Pre-compile the regular expression on module loading instead of every instantiation.
check also extra field names
update two affected bots
plus documentation

@ghost ghost added this to the 3.0.1 milestone Aug 20, 2021
@ghost ghost self-requested a review August 20, 2021 10:22
@ghost ghost force-pushed the wagner/fix-1807 branch from 9f8cb7a to dd6f9dd Compare August 20, 2021 10:27
@codecov-commenter
Copy link

codecov-commenter commented Aug 20, 2021

Codecov Report

Merging #2059 (e927fad) into develop (b14d9c6) will increase coverage by 0.00%.
The diff coverage is 84.61%.

@@           Coverage Diff            @@
##           develop    #2059   +/-   ##
========================================
  Coverage    76.03%   76.04%           
========================================
  Files          428      428           
  Lines        23040    23048    +8     
  Branches      3083     3084    +1     
========================================
+ Hits         17519    17527    +8     
  Misses        4813     4813           
  Partials       708      708           
Impacted Files Coverage Δ
...q/tests/bots/parsers/anubisnetworks/test_parser.py 100.00% <ø> (ø)
intelmq/bots/parsers/alienvault/parser_otx.py 80.28% <50.00%> (ø)
intelmq/lib/message.py 95.34% <80.00%> (+0.04%) ⬆️
intelmq/bots/parsers/anubisnetworks/parser.py 82.79% <100.00%> (+0.09%) ⬆️
...ts/parsers/html_table/test_parser_ignore_values.py 100.00% <100.00%> (ø)
intelmq/tests/lib/test_message.py 100.00% <100.00%> (ø)

@ghost ghost force-pushed the wagner/fix-1807 branch 3 times, most recently from f0e0538 to f7208e9 Compare August 20, 2021 14:15
@ghost ghost modified the milestones: 3.0.1, 3.1.0 Aug 20, 2021
@ghost ghost force-pushed the wagner/fix-1807 branch from f7208e9 to e79db25 Compare August 20, 2021 14:51
@ghost ghost force-pushed the wagner/fix-1807 branch from e79db25 to 3001f45 Compare August 23, 2021 12:26
Sebastian Wagner added 6 commits August 23, 2021 14:29
the regular expression, which is used to check the validity of
harmonization key names, was never effective as the dot wasn't escaped.
Additionally, the characters of the first element were not allowed to
repeat.

Pre-compile the regular expression on module loading instead of every
instanciation.
if they match the regular expression for key names

fixes #1807
fix some used field names, as the field name restrictions are now
enforced
a used field name did not match the field name restrictions causing a
test error
the current version of this feed does not have field anymore, so no
futher action is required here
@ghost ghost force-pushed the wagner/fix-1807 branch from 3001f45 to e927fad Compare August 23, 2021 12:30
@ghost ghost merged commit f3d3df0 into develop Aug 23, 2021
@ghost ghost deleted the wagner/fix-1807 branch August 23, 2021 12:55
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant