Error Installing on ubuntu 20.04

[fopson]

Hi all, I have unsuccessfully been trying to get this module working on ubuntu 20.04. I Used to use the latest elastalert v0.2.4 with python3.8 which worked great exept for the KeyError: when any entry in the hive_observable_data_mapping doesn't find a match which causes it not to send the alert as previously discribed in issue #10. From issue #10, I read that this module solves this problem when coupled with elastalert release v0.2.1 and phython 3.6 so thought it was going to save my life but I have been struggling to get it to work. Here is my install process:

1. Install phython 3.6 and python virtual env

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.6
sudo apt-get install python3-virtualenv

2. Download and extract release v0.2.1 of elastalert from Yelp
wget https://github.com/Yelp/elastalert/archive/v0.2.1.tar.gz -O - | sudo tar -xz -C /opt/
3. Cd to project dir and create a virutal env called "elastalert-venv"
cd /opt/elastalert-0.2.1 && virtualenv -p /usr/bin/python3.6 elastalert-venv
4. Activate the virtual environment
source ./elastalert-venv/bin/activate
5. Install dependency (without it, I get the error: warning: no files found matching 'blist.rst' on step 6)
sudo apt install libpython3.6-dev
6. Install the Yelp Python package
python setup.py install
7. Clone the Nclose Hive Alerter master branch
git clone https://github.com/Nclose-ZA/elastalert_hive_alerter.git
8. Install the Nclose Hive Alerter python package
python elastalert_hive_alerter/setup.py install

I then used one of my previous rules but changed the alert module to alert: "elastalert_hive_alerter.hive_alerter.HiveAlerter" as per the documentation on this site. Now, when I ran elastalert, I got this error:

Based on this output:

I changed the alert module to alert: "elastalert_hive_alerter.elastalert_hive_alerter.hive_alerter" and now when I run elastert, I get the following error:

Note that if I change the alert back to alert: hivealerter, I don't get the above error but then I am back to square one where I get the KeyError: when a field is empty in a match.

I am not sure what I am doing wrong at this stage. Has anybody gotten this working lately on ubuntu 20.04? I have been googling but so far I am coming up empty. Any help will be appreciated!

[danielbrowne]

It looks like you had the right idea as it seems the module was installed differently in your environment, but you seem to have left the class name off the end when you changed it.

Maybe try "elastalert_hive_alerter.elastalert_hive_alerter.hive_alerter.HiveAlerter"?

I imagine that will get you past the "issubclass" error at least.

[fopson]

Thanks @danielbrowne, changing the alert to "elastalert_hive_alerter.elastalert_hive_alerter.hive_alerter.HiveAlerter" got me further but now I am getting an error a bit similar to the KeyError: I get with release v0.2.4 from Yelp. It seem not to like it when something doesn't exist.

To put thing more into context, what I am essentially trying to acheive here is grap signals from elastic SIEM, and send them to thehive while attaching the observables if and when they exist. I works great if I leave out the attaching the observables part but then what I have to do is, every time I get alerts in thehive, I have to go back in kibana, find the signal and manually copy the observables. It would be such a time savior if those observables could be attached automatically by elastalert.

Because all signals don't have the same set of fields, I am trying to match the posible values to grab them when they exist.

Here is an example attempt to grap related ip addresses if and when they exist in the signal:

hive_observable_data_mapping:
  - ip: "{match[source][ip]}"
  - ip: "{match[destination][ip]}"
  - ip: "{match[host][ip]}"
  - ip: "{match[client][ip]}"
  - ip: "{match[server][ip]}"

Every time any of these fields is missing from the signal, the alert never makes it to thehive and I get the following error:

Or could this be acheived in a different way?

[danielbrowne]

@fopson It looks like you have the latest version of the code which is just printing an warning when trying to create an artifact for which data does not exist. That should not stop the alert from being sent without that specific artifact. That KeyError would have caused the problem, but that was fixed and the warning is now printed instead. My next suggestion would be to print some debug lines in the HiveAlerter code to ensure that send_to_thehive is being called.

[fopson]

@danielbrowne the error above is definately different than the KeyError which I guess is a good thing. Here is a tail of the above error.

I see a few of these. There seem to be an error while trying to create the alert.

and here is a snippet of thehive's log

It consistantly goes like this for a few pages and doesn't show any connections from elastalert.

I am trying to produce a pdb trace of the module running but passing modules directly to pdb was not support in python3.6 so am trying to find a way around that.

[fopson]

The problem may not be with the hive_observable_data_mapping section. I commented out the entire hive_observable_data_mapping section and I get this.

[fopson]

I have just build a new Ubuntu 20.04 server VM and reinstalled from the blank server but I get the same error as my previous comment.

[danielbrowne]

@fopson I see that there is a commit where jsonify is being monkey patched at the top of hive_alerter.py, but I am not sure why. Something to do with python2 vs python3. I suspect that maybe maybe allowing that monkey patch to accept *args and **kwargs might solve the problem you are seeing.

[fopson]

@danielbrowne Thanks for pointing me to the right direction. For testing, I commented out the monkey patch and can now receive elarts in some conditions with observables attached to the alert.

This in itself is already a win as it will save me soo much time. Thanks for this module!

After getting these results, I decided to test the same setup but with latest elastalert v0.2.4 and tadahh I also receive alerts with mapped observables with the latest version version of elastalert. So for anybody who wants to try this in the future, this setup also works with v0.2.4 of elastalert.

I seem to have lost a feature though. Whether the setup is with v0.2.1 or v0.2.4, I seem to have lost the ability use lists either in the desciption:

  description: |
    '{match[signal][rule][description]}'

    * **Tactique Regerence:** '{match[signal][rule][threat][tactic][reference]}'
    * **Technique Reference:** '{match[signal][rule][threat][technique][reference]}'
    * **False Positive:** 'Unknown'

or in the tags:
tags: ['{match[signal][rule][threat][tactic][id]}', '{match[signal][rule][threat][technique][id]}', '{match[signal][rule][threat][tactic][name]}', 'ElastAlert']

In either cases I get the following error:

and the alerts are not sent to thehive. It seems to want the values to be integers. I am not sure if I now need to format it in a different way to get it to work. Single entries workd fine though. like this:
description: '{match[signal][rule][description]}'
Any suggestions on the way to pass lists? or multiline entries?

[danielbrowne]

@fopson The error indicates that one of the properties in the match context path is a list, but that the subsequent element in the path treats it as a dictionary.

For example, the following format would generate the error if "rule" was actually a list in the context:

In [10]: '{match[signal][rule][description]}'.format(**{'match': {'signal': {'rule': [{'description': 'something'}]}}})                                                                                            
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-89b62d749f3f> in <module>
----> 1 '{match[signal][rule][description]}'.format(**{'match': {'signal': {'rule': [{'description': 'something'}]}}})

TypeError: list indices must be integers or slices, not str

You can still use a list as the leaf value in the format:

In [11]: '{match[signal][rule]}'.format(**{'match': {'signal': {'rule': [{'description': 'something'}]}}})                                                                                                         
Out[11]: "[{'description': 'something'}]"

Or if you know which element you want you can refer to it specifically:

In [12]: '{match[signal][rule][0]}'.format(**{'match': {'signal': {'rule': [{'description': 'something'}]}}})                                                                                                      
Out[12]: "{'description': 'something'}"

In [14]: '{match[signal][rule][0][description]}'.format(**{'match': {'signal': {'rule': [{'description': 'something'}]}}})                                                                                         
Out[14]: 'something'

Does that help?

[fopson]

@danielbrowne Yes, your explaination is of great help thanks. It helped me understand what I was doing wrong with the tags. The tags where fairely easy to resolve one I understood the right way to express the path with a list for the match. I am still having a bit of a hard time sorting the multiline description. I am sure I am screwing up the syntax somehow. When I try to retrieve just the the desciption in one line:

hive_alert_config:
  ...
  description: '{match[signal][rule][description]}'
  severity: 2
  ....

It works, which makes me believe it dosn't have to do with the path of the match. Now as soon as I try to use one of the multiline operators, it complains about list indices:

hive_alert_config:
  ...
  description: |
    '{match[signal][rule][description]}'
  severity: 2
  ....

My final goal is to be able to use multilines formated like this:

hive_alert_config:
  ...
  description: |
    '{match[signal][rule][description]}'

    * **UserName:** '{match[related][user]}'
    * **Technique Reference:** '{match[signal][rule][threat][technique][reference]}'
    * **False Positive:** '{match[signal][rule][false_positives][0]}'
  severity: 2
  ...

Do the multiline operators (> or |) turn the following lines into a list? How could I then get around that to achieve my goal?

[fopson]

@danielbrowne It's all working like a charm now. I was messing up the json syntax in the description. Thanks again for your help and guidance!

[danielbrowne]

@fopson no problem, I'm glad you found this useful.