Sieve Filter expert #1083

digihash · 2017-08-25T11:20:51Z

This is for the Sievebot which was written in collaboration with @antoinet and @helderfernandes1279.

The sieve bot is used to filter and/or modify events based on a set of rules. The rules are specified in an external configuration file and with a syntax similar to the Sieve language used for mail filtering.

Each rule defines a set of matching conditions on received events. Events can be matched based on keys and values in the event. If the processed event matches a rule's conditions, the corresponding actions are performed. Actions can specify whether the event should be kept or dropped in the pipeline (filtering actions) or if keys and values should be changed (modification actions).

For more information, see README.md under "intelmq/bots/experts/sieve/README.md"

…nd forwards message.

See: https://travis-ci.org/certtools/intelmq/jobs/268325253#L3166

antoinet · 2017-08-28T08:26:31Z

I fixed the codestyle checks in https://github.com/antoinet/intelmq/commit/17c0d4d1d70903ff08f00a33ab6cb1a3276dc220
@digihash: can you add this commit to the PR?

ghost

Issues pointed out by the tests:

don't use tabs in BOTS
fix the codestyle:
- intelmq/bots/experts/sieve/expert.py:222:1: E305 expected 2 blank lines after class or function definition, found 1
- intelmq/bots/experts/sieve/validator.py:46:1: E305 expected 2 blank lines after class or function definition, found 1
- intelmq/bots/experts/sieve/validator.py:54:36: W292 no newline at end of file
do not "rely" on the textx package during the module loading, have a look at e.g. the asn lookup: https://github.com/certtools/intelmq/blob/master/intelmq/bots/experts/asn_lookup/expert.py#L6 and https://github.com/certtools/intelmq/blob/master/intelmq/bots/experts/asn_lookup/expert.py#L15
add a test.skip_exotic() decorator, so the textx won't be required for a test run if not explicitly requested like here: https://github.com/certtools/intelmq/blob/master/intelmq/tests/bots/experts/asn_lookup/test_expert.py#L42
ipaddress is already in cpython >= 3.3, we do not support anything lower, so remove this requirement

Will look at the code once these issues are resolvend.

ghost · 2017-08-28T08:29:04Z

@antoinet Thanks. If you push to antoinet:develop, the commits are part of this PR

See: #1083 (review) * removed tabs in BOTS * safely import textx package * added `skip_exotic` decorator to sievebot test class * removed package `ipaddress` from REQUIREMENTS.txt Other changes: * removed `print` statement from expert.py

* Fixed BOTS format * replaced re.fullsearch with re.search (method unavailable in python < 3.4) * removed Enum definition (unavailable in python < 3.4)

ghost

failing tests for cymru can be ignored

Looks good overall.

I guess there is some room for optimization as some rules are evaulated every time.

See also https://github.com/antoinet/intelmq/pull/15 fixing some of the issues.

I am not very happy about just another configuration language in intelmq but I don't have better solution currently.

ghost · 2017-08-28T09:17:19Z

intelmq/bots/experts/sieve/expert.py

+Parameters:
+    file: string
+"""
+from __future__ import unicode_literals


We support python3 only, this is obsolete. Also, you may find isort useful.

ghost · 2017-08-28T11:30:59Z

intelmq/bots/BOTS

@@ -572,6 +572,13 @@
            "module": "intelmq.bots.experts.taxonomy.expert",
            "parameters": {}
        },
+        "Sievebot": {
+            "description": "Sievebot is the bot responsible to filter and modify intelMQ events.",


Please be more verbose, say that the sieve language is used. E.g. 'This bot filters and modifies events based on a sieve-based language.'

ghost · 2017-08-28T12:08:49Z

intelmq/bots/experts/sieve/expert.py

+from intelmq.lib.bot import Bot
+
+try:
+    import textx


imported but unused. You can shorten it by doing:

try: from textx.metamodel import metamodel_from_file from textx.exceptions import TextXError, TextXSemanticError except ImportError: metamodel_from_file = None

and then

if metamodel_from_file is None: raise ...

ghost · 2017-08-28T12:28:47Z

intelmq/bots/experts/sieve/expert.py

+        except TextXError as e:
+            self.logger.error('Could not process sieve grammar file. Error in (%d, %d).', e.line, e.col)
+            self.logger.error(str(e))
+            self.stop()


Instead of these three lines simply do raise ValueError('...'), everything else is handled by the Bot-class.

ghost · 2017-08-28T13:26:18Z

intelmq/bots/experts/sieve/.gitignore

@@ -0,0 +1,2 @@
+*.dot


because they might have used graphviz :) Anyway, yes. Not needed here.

ghost · 2017-08-28T13:50:40Z

intelmq/bots/experts/sieve/expert.py

+            return self.process_ip_range_match(match.key, match.range, event)
+        elif match.__class__.__name__ == 'Expression':
+            return self.match_expression(match, event)
+        pass


unnecessary

ghost · 2017-08-28T13:53:52Z

intelmq/bots/experts/sieve/expert.py

+
+        try:
+            addr = ipaddress.ip_address(event[key])
+        except ValueError:


Did this happen in your tests? It should never, actually

Since the event type is not enforced (any key can be specified as the left hand side of the << operator), the value is validated here. We did discuss whether or not to restrict this operator to keys with a type corresponding to an IP address (https://github.com/antoinet/intelmq/issues/11), and I just realized I didn't implement it the suggested way.

ghost · 2017-08-28T14:37:53Z

intelmq/bots/BOTS

@@ -572,6 +572,13 @@
            "module": "intelmq.bots.experts.taxonomy.expert",
            "parameters": {}
        },
+        "Sievebot": {


The bot is called sieve everywhere else.

ghost · 2017-08-28T15:11:25Z

intelmq/tests/bots/experts/sieve/test_sieve_files/test_add.sieve

@@ -0,0 +1,3 @@
+if comment == 'add field' {
+    add destination.ip="150.50.50.10"
+}


missing newlines everywhere

ghost · 2017-08-28T15:13:53Z

intelmq/bots/experts/sieve/expert.py

+            return False
+
+    def process_numeric_operator(self, lhs, op, rhs):
+        return eval(str(lhs) + op + str(rhs))  # TODO graceful error handling


Is it possible without eval? At least a strict checking of the arguments should be done.

Fix some issues in sieve expert

aaronkaplan

I had quite a few comments after more careful review. but overall this is a fantastic addition!

Most welcome enhancement would be to be able to specify the output pipeline with this sieve filter.

Then you could do a type of switch-case statement where the data should be flowing to.

aaronkaplan · 2017-08-30T14:22:20Z

intelmq/bots/experts/sieve/.gitignore

@@ -0,0 +1,2 @@
+*.dot


because they might have used graphviz :) Anyway, yes. Not needed here.

aaronkaplan · 2017-08-30T14:23:31Z

intelmq/bots/experts/sieve/README.md

+rule's conditions, the corresponding actions are performed. Actions can specify
+whether the event should be kept or dropped in the pipeline (filtering actions)
+or if keys and values should be changed (modification actions).
+


Would it be possible to specify a specific outgoing pipeline as an action?

Example:

if $condition then send-to "pipeline-A"; fi

This is not supported by the core.

In our instance, we are already (ab)using a newly introduced key called keep (boolean), which can be set anywhere in the pipeline. Downstream, events with keep==False are dropped.

This can be generalized by introducing a new key, e.g. route, with values that describe a specific destination. But this requires to put filter bots upstream of every destination such that only the corresponding events are kept.

If a routing model was provided in the core, we would certainly have use-cases for it.

I wonder why no issue existed for this which and created #1088 for it

aaronkaplan · 2017-08-30T14:25:08Z

intelmq/bots/experts/sieve/README.md

+## Parameters
+
+The sieve bot takes only one parameter:
+ * `file` - filesystem path the the sieve file


grammar: file system path of the sieve file

aaronkaplan · 2017-08-30T14:26:03Z

intelmq/bots/experts/sieve/README.md

+  drop  // aborts processing of subsequent rules and drops the event.
+}
+
+if source.ip << 192.0.0.0/24 {


according to the docs below, IP addresses must be quoted.

aaronkaplan · 2017-08-30T14:27:16Z

intelmq/bots/experts/sieve/README.md

+
+   ```if feed.name != 'acme-security' || feed.accuracy == 100 { ... }```
+
+ * `:contains` matches on substrings.


wish list: if we have contains, out of reasons of symmetry , may we have :notcontains ?

aaronkaplan · 2017-08-30T14:32:29Z

intelmq/bots/experts/sieve/README.md

+
+ * `keep` marks the event to be forwarded to the next bot in the pipeline
+ (same as the default behaviour), but in addition the sieve file processing is
+ interrupted upon reaching this action.


So implicitly here you are saying that the default behaviour without keep is to contineue with the matching ?

Yes, the keep and drop actions explicitly interrupt the processing of the rules.

Without a mention of keep or drop, processing of subsequent rules continues. Afaik this is the same behavior as mail sieve.

aaronkaplan · 2017-08-30T14:33:01Z

intelmq/bots/experts/sieve/README.md

+
+### Comments
+
+Comments may be used in the sieve file: all characters after `//` and until the end of the line will be ignored.


wish-list: // or # as a comment character.

aaronkaplan · 2017-08-30T14:34:03Z

intelmq/bots/experts/sieve/README.md

+
+Use the following command to validate your sieve files:
+```
+$ intelmq.bots.experts.sieve.validator


this should be called implicitly when the intelmqctl configtest gets called.

It is intelmqctl check

aaronkaplan · 2017-08-30T14:34:30Z

intelmq/bots/experts/sieve/expert.py

+            return False
+
+    def process_numeric_operator(self, lhs, op, rhs):
+        return eval(str(lhs) + op + str(rhs))  # TODO graceful error handling


aaronkaplan · 2017-08-30T14:35:46Z

intelmq/tests/bots/experts/sieve/test_sieve_files/test_ip_range_list_match.sieve

@@ -0,0 +1,3 @@
+if source.ip << ['192.0.0.0/24', '169.254.0.0/16'] {


this does not seem to be documented in the docu. I mean that the << operator also works on lists.

Actually, every operator works on lists. I will point that out.

aaronkaplan · 2017-08-30T14:37:30Z

I move that we merge this soon!

- Remove .gitignore for .dot files - Typo and erroneous syntax in README.md - codestyle in validator.py

codecov-io · 2017-08-30T23:18:09Z

Codecov Report

Merging #1083 into develop will increase coverage by 2.38%.
The diff coverage is 90.13%.

@@             Coverage Diff             @@
##           develop    #1083      +/-   ##
===========================================
+ Coverage    73.93%   76.31%   +2.38%     
===========================================
  Files          217      220       +3     
  Lines         9417    10338     +921     
  Branches      1283     1363      +80     
===========================================
+ Hits          6962     7889     +927     
+ Misses        2176     2142      -34     
- Partials       279      307      +28

Impacted Files	Coverage Δ
...elmq/tests/bots/experts/cymru_whois/test_expert.py	`100% <ø> (+40%)`	⬆️
intelmq/bots/experts/sieve/validator.py	`0% <0%> (ø)`
intelmq/bin/intelmqctl.py	`11.05% <0%> (-0.07%)`	⬇️
intelmq/tests/bots/experts/sieve/test_expert.py	`100% <100%> (ø)`
intelmq/bots/experts/asn_lookup/expert.py	`62.5% <25%> (ø)`	⬆️
intelmq/bots/outputs/file/output.py	`74.07% <28.57%> (ø)`	⬆️
intelmq/lib/bot.py	`62.69% <75%> (ø)`	⬆️
intelmq/bots/experts/sieve/expert.py	`84.82% <84.82%> (ø)`
intelmq/bots/experts/abusix/expert.py	`39.28% <0%> (-46.43%)`	⬇️
intelmq/tests/bots/experts/abusix/test_expert.py	`56.66% <0%> (-43.34%)`	⬇️
... and 21 more

…arily unavailable).

fixes #1090

…q#16

antoinet · 2017-10-16T08:48:33Z

@wagner-certat and @aaronkaplan: I think I addressed all review points so far up to the :in operator, used to check if an ip address matches a list of values, e.g.:
source.ip :in ['127.0.0.1', '192.168.1.1', '10.0.0.1']
As mentioned above, most of the operators support a list in the right-hand-side by default, e.g.:
source.ip != ['127.0.0.1', '192.168.1.1', ...]
So it is a design question:

we can introduce the :in operator (and by symmetry :notin) and drop list-support for all other operators (makes it simpler)
:in is just an alias for == (and :notin an alias for !=), and we keep list-support for all other operators.
What do you all think?

ghost · 2017-11-30T12:52:39Z

Concerning the :in and == and lists...

If we one day support multiple source IP addresses or similar (with the IDEA format or whatever) everything blows up anyway... So I am in favour of just using the currently implemented behaviour with ==.

@aaronkaplan What do you think?
And @dmth maybe you have the leisure to comment too? :D

SYNchroACK · 2017-12-14T11:04:25Z

Concerning the :in and == and lists...

If we one day support multiple source IP addresses or similar (with the IDEA format or whatever) everything blows up anyway... So I am in favour of just using the currently implemented behaviour with ==.

I understand, its well pointed @wagner-certat . However, I think we are far way from that day and also, the day we change to multi source IP addresses we need to change multiple experts bots. IMHO, we should keep with our principle of using one key = value, therefore, the presented code is fine.
Makes sense?

Antoine Neuenschwander and others added 20 commits August 24, 2017 11:21

Added initial sieve metamodel

24ab6ca

Refined sieve metamodel

6e6465d

Added first draft of sieve/expert.py with unittests (not functional)

0e43dd8

Implemented parsing the sieve file (partially)

a187b3f

Sieve bot added to BOTS registration file

9eb33c6

Add comment regex in model

54c05d0

Remove debugging prints

e3eaa46

Fix antoinet/intelmq#5: added error and debug logging messages

d06b197

Implemented antoinet/intelmq#6: added keep action: stops processing a…

873a9fb

…nd forwards message.

Add else and elsif clauses in rules, fixes antoinet/intelmq#4

8446988

Add new test to test ADD action

fda9cad

Document usage/features in README.md, antoinet/intelmq#9

99b6cc2

Implementation of Numeric match value list and String match value list

6b2759f

Add tests for REMOVE action

706819f

Added section about comments, antoinet/intelmq#9

f5cc29c

Added command to validate sieve files, antoinet/intelmq#10

3f48d8f

Add 'ip in netblock' match, antoinet/intelmq#11

40dda0d

Update README.md

76bc0dd

Added test missing test cases antoinet/intelmq#8

b73c48f

Added REQUIREMENTS.txt with depedencies, antoinet/intelmq#13

8f1881c

ghost changed the title ~~Develop~~ Sieve Filter expert Aug 28, 2017

ghost added this to the 1.1.0 milestone Aug 28, 2017

ghost added component: bots feature Indicates new feature requests or new features labels Aug 28, 2017

Fixed codestyle errors.

17c0d4d

See: https://travis-ci.org/certtools/intelmq/jobs/268325253#L3166

ghost suggested changes Aug 28, 2017

View reviewed changes

Antoine Neuenschwander added 2 commits August 28, 2017 10:55

Fix failing tests, antoinet/intelmq#14

cda7c7d

* Fixed BOTS format * replaced re.fullsearch with re.search (method unavailable in python < 3.4) * removed Enum definition (unavailable in python < 3.4)

Sebastian Wagner added 4 commits August 28, 2017 17:02

BUG: sieve filter expert error handling

ea03ee2

MAINT: sieve expert bot: shorten some code

8276b5e

TST: sieve filter expert: add missing newlines

6d50608

Merge branch 'develop' into antoinet-develop

1b148d5

ghost suggested changes Aug 28, 2017

View reviewed changes

ghost self-assigned this Aug 28, 2017

ghost requested a review from aaronkaplan August 28, 2017 15:20

Merge pull request #15 from wagner-certat/antoinet-develop

3701bda

Fix some issues in sieve expert

aaronkaplan requested changes Aug 30, 2017

View reviewed changes

Review quick fixes, antoinet/intelmq#14

f590a95

- Remove .gitignore for .dot files - Typo and erroneous syntax in README.md - codestyle in validator.py

Antoine Neuenschwander and others added 4 commits September 7, 2017 11:43

antoinet/intelmq#18: rename action modify to update.

b46b642

Re-enable cymru-whois tests (were disabled because service was tempor…

e385bfa

…arily unavailable).

ENH+DOC: check function for bots

2d5ac01

fixes #1090

ENH: check method for asn_lookup and file output

764015b

ghost mentioned this pull request Sep 11, 2017

Call bot check #1091

Merged

Antoine Neuenschwander added 7 commits September 18, 2017 10:47

Merge branch 'call-bot-check' into develop

7ddce57

check method for sieve filter, antoinet/intelmq#20

e878cc9

Use '//' or '#' for line comments, antoinet/intelmq#19

e2f6a8c

implemented :notcontains operator, antoinet/intelmq#17

342b483

Strict checking for numeric match while using eval(), antoinet/intelm…

7ee859b

…q#16

Codestyle fixes

1419da5

Validate IP addresses according to harmonization, antoinet/intelmq#11

c8f6134

ghost approved these changes Dec 18, 2017

View reviewed changes

aaronkaplan approved these changes Dec 18, 2017

View reviewed changes

ghost merged commit c8f6134 into certtools:develop Dec 18, 2017

This pull request was closed.


		```if feed.name != 'acme-security' \|\| feed.accuracy == 100 { ... }```

		* `:contains` matches on substrings.


		### Comments

		Comments may be used in the sieve file: all characters after `//` and until the end of the line will be ignored.

		@@ -0,0 +1,3 @@
		if source.ip << ['192.0.0.0/24', '169.254.0.0/16'] {

Sieve Filter expert #1083

Sieve Filter expert #1083

Conversation

digihash commented Aug 25, 2017

antoinet commented Aug 28, 2017

ghost left a comment

Choose a reason for hiding this comment

ghost commented Aug 28, 2017 • edited by ghost Loading

ghost left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronkaplan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronkaplan commented Aug 30, 2017

codecov-io commented Aug 30, 2017 • edited Loading

Codecov Report

antoinet commented Oct 16, 2017

ghost commented Nov 30, 2017 • edited by ghost Loading

SYNchroACK commented Dec 14, 2017

ghost commented Aug 28, 2017 •

edited by ghost

Loading

codecov-io commented Aug 30, 2017 •

edited

Loading

ghost commented Nov 30, 2017 •

edited by ghost

Loading