Alerting fails if there's an empty field in the alert

[nickbabkin]

Currently, if one of the specified fields is not present in elastalert {match}, TheHive alerter fails with KeyError.

Example config sample:

hive_alert_config:
  description: '{match[clientip]}'

Test run:
elastalert_error - {'message': "Uncaught exception running rule --: 'clientip'", 'traceback': ['Traceback (most recent call last):', ' File "/usr/local/lib/python3.5/dist-packages/elastalert-0.2.1-py3.5.egg/elastalert/elastalert.py", line 1444, in alert', ' return self.send_alert(matches, rule, alert_time=alert_time, retried=retried)', ' File "/usr/local/lib/python3.5/dist-packages/elastalert-0.2.1-py3.5.egg/elastalert/elastalert.py", line 1533, in send_alert', ' alert.alert(matches)', ' File "/usr/local/lib/python3.5/dist-packages/elastalert-0.2.1-py3.5.egg/elastalert/alerts.py", line 2154, in alert', ' alert_config[alert_config_field] = alert_config_value.format(**context)', "KeyError: 'clientip'"], 'data': {'rule': '--'}}

[danielbrowne]

It appears from your traceback that you are using the version of the alerter included in Elastalert. If you use the standalone one from this repo, that issue has been resolved.

[nickbabkin]

@danielbrowne indeed I do.
Can you point me at the fix commit please?

[danielbrowne]

It looks like 0e9a7c8, however I know this code has been refactored once or twice since the original version was put into Elastalert, so you might struggle a little with the merging if you want to get it into the Elastalert repo.

[nickbabkin]

@danielbrowne thanks.
How do I use the current version from repo instead of the one embedded in elastalert? Just replace the folder and restart?

[danielbrowne]

You can check out this repo and install it as a python package using setup.py as usual, then update your Elastalert to point to this package instead of the internal one as detailed in the documentation.

https://github.com/Nclose-ZA/elastalert_hive_alerter/blob/master/README.rst

[webhead404]

@danielbrowne I am using the version that is here and this is still occurring for me. within the description field and observable fields both. @nickbabkin were you able to figure this out?

Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]: ERROR:root:Traceback (most recent call last):
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]:   File "/usr/local/lib/python3.6/dist-packages/elastalert-0.2.1-py3.6.egg/elastalert/elastalert.py", line 1450, in alert
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]:     return self.send_alert(matches, rule, alert_time=alert_time, retried=retried)
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]:   File "/usr/local/lib/python3.6/dist-packages/elastalert-0.2.1-py3.6.egg/elastalert/elastalert.py", line 1544, in send_alert
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]:     alert.alert(matches)
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]:   File "/usr/local/lib/python3.6/dist-packages/elastalert_hive_alerter-1.0.0rc2-py3.6.egg/elastalert_hive_alerter/hive_alerter.py", line 85, in alert
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]:     alert_config = self.create_alert_config(match)
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]:   File "/usr/local/lib/python3.6/dist-packages/elastalert_hive_alerter-1.0.0rc2-py3.6.egg/elastalert_hive_alerter/hive_alerter.py", line 56, in create_alert_config
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]:     alert_config[alert_config_field] = alert_config_value.format(**context)
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]: KeyError: 'source'
Mar 26 09:53:34 elkhunter-virtual-machine elastalert[7157]: ERROR:root:Uncaught exception running rule New-High-Severity-in-SIEM: 'source'

alert: elastalert_hive_alerter.hive_alerter.HiveAlerter

hive_connection:
  hive_host: http://$IP
  hive_port: $PORT
  hive_apikey: $API

[danielbrowne]

@webhead404 I will have to have a look at this when I get a moment, can you confirm that you don't expect "source" to be in all your matches?

[webhead404]

This is a blanket rule with different observables. So if there is an alert without that observable it doesn't alert at all. I figured I would have to create multiple elastalert rules for the specific event types is that correct?

[danielbrowne]

I guess as a workaround you could create more specific rules that target data which will always have the values you are looking for. But I would like to fix the bug so that you don't have to in future.

[webhead404]

That would be awesome! I would be willing to help however I can.

[JoshuaSmeda]

Keyerror appears to be thrown on description only. Observable keyerror is ignored when using latest branch on Python 3.6, elastalert release v0.2.1