[OSCD Initiative] add Gmail responder

[strassi]

Implementation of the Gmail Responder for the OSCD Initiative. Can only be used with a valid Gsuite Account.

This fixes Issue #859

[yugoslavskiy]

Hello @strassi!

Sorry for the delay, it was a quite busy time.

I've tested your responder and got a few comments:

1. Please replace urllib with urllib3 in requrements.txt, as it doesn't work the way it is
2. It seems that if a non-company email (sender) was blocked, it's not getting a proper tag, which is required for further unblocking. So we can block a sender, but cannot unblock. Could you please fix it? I would be able to proceed to test your Responder after that

[strassi]

Hi @yugoslavskiy!

1. not a problem
2. I just had a quick look at the code but can't spot issues that would match your problem description. The observables being blocked (sender) does not get a tag. The tags are set on the gmail addresses of the case. This is done because filters are set in the context of the gmail user.

Expected behavior (that worked in my tests, in November):

1. Add [email protected] observable to case.
2. Add [email protected] observable to case.
3. Start Gmail_BlockSender Responder on [email protected].
4. [email protected] gets tag gmail_filter:[email protected]:<gmail_filter_id>
5. Start Gmail_UnblockSender Responder on [email protected].
6. Tag on [email protected] should get removed.

Can you provide a some sort of screenshots or output along the way of the described test run?

[yugoslavskiy]

Hi @strassi!

Here is how it was:

1. Add [email protected] (it utilizes Gmail) observable to a case.
2. Add [email protected] observable to a case.
3. Start Gmail_BlockSender Responder on [email protected].
4. [email protected] doesn't gets tag gmail_filter:[email protected]:<gmail_filter_id>.
5. Start Gmail_UnblockSender Responder on [email protected].
6. It doesn't start with an error: No valid filter tag found on observable. Tags: ['<random tag that I had to create while adding the observable>']

Here is the screenshots:

[strassi]

I took the TheHive 3.4.0 Training VM for debugging the problem. Additionally I disabled the Gmail API calls in block_messages and unblock_messages to test the tagging mechanism for the thehive. The tagging seems to work:

To get this work in the training VM I had to upgrade thehive4py pip3 packages to 1.8.1. The responder uses the thehive4py.query API. Without the upgrade the responder fails because EndsWith is not defined.

@yugoslavskiy can you check if the installed version of your thehive4py is >= 1.8.1. If not please upgrade the thehive4py version and repeat your tests.

I'm going to set the requirements.txt to include the minimum version for the library to exclude future problems of this kind.

[yugoslavskiy]

Hello @strassi!

Hope you are doing well (:

I had thehive4py version 1.8.1 at my previous tests:

$ python3 -m pip show thehive4py | grep 'Version'
Version: 1.8.1

So I decided to update all the packages to the versions you fixed in your last commit, and here is what I got:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
docker-compose 1.24.1 requires jsonschema<3,>=2.5.1, but you have jsonschema 3.2.0 which is incompatible.
docker-compose 1.24.1 requires requests!=2.11.0,!=2.12.2,!=2.18.0,<2.21,>=2.6.1, but you have requests 2.25.0 which is incompatible.
botocore 1.14.17 requires urllib3<1.26,>=1.20; python_version != "3.4", but you have urllib3 1.26.3 which is incompatible.

It seems that it's time to move to the docker, but I have not idea how to make it work there, even tho I see the Dockerfile you've created.
OK let's hope that problem was somewhere on my side, or in my workflow.
I will repeat the steps anyway and let you know how it went.

[yugoslavskiy]

Here is how it was:

1. Create a new case "Gmail Responder Test Case"
2. Add [email protected] (it utilizes Gmail) observable to a case.
3. Add [email protected] observable to a case.

Here is how it looked at that step:

4. Execute Gmail_BlockSender Responder on [email protected]. Here is the result:

Here is the info about successful execution:

Here is the list of observables (no tags created):

So it seems that the result is the same, [email protected] doesn't gets tag gmail_filter:[email protected]:<gmail_filter_id>.

Could you please give me a hint on the debugging? It seems that you somehow spot the error with the old version of thehive4py. Maybe I will see a different error in my environment using your approach.

[strassi]

Hi @yugoslavskiy !

I'm doing fine besides wondering about this bug 🤔

So far we see that

• the responder results in a success state (so no exceptions)
• the blocked observable gets a tag (so tagging works fine)
• your thehive4py version is correct (so EndsWith() from thehive4py.query is available)

I'm wondering if the responder gets any gmail obervables from the case. You could change line 145

145        self.report({'message': "Added filters"})

to

145        self.report({'message': "Added filters to {}".format(gmail_observables)})

this should print the list of gmail observables to the responders output. Furthermore you could check if the filters are set in the Gmail settings of the respective gmail address (e.g. [email protected]).

[yugoslavskiy]

Yeah, the filters are set, that's for sure. I had to delete the previous one manually.
Also, removing filters via the Gmail_UnblockSender responder works fine if creating the proper tag manually.

The only issue with the Gmail_UnblockSender is that it breaks if there are two fiter tags present. The first one — old, that has been removed, the second one — an active filter:

So it breaks on first and didn't try to proceed with the second one. Maybe the main issue here is that it doesn't remove the filter tag from the observable after deleting the real filter in the Gmail (after successful execution), which (IMHO) it should do.

I've updated the line you've suggested, and here is what I got:

{
  "message": "Added filters to [{'updatedAt': 1614729931036, 'tlp': 0, 'tags': ['test', 'gmail_filter:[email protected]:ANe1BmiYy6Kn74pGfhlfV1nDOUZmAyfjASxB3Q'], 'data': '[email protected]', 'ioc': False, 'reports': {}, 'status': 'Ok', 'sighted': False, 'createdAt': 1614712731513, 'createdBy': 'admin', 'message': 'test', 'dataType': 'mail', 'updatedBy': 'admin', 'startDate': 1614712731514, '_type': 'case_artifact', '_routing': 'sZxf9HcBfFDkyb-x_nuK', '_parent': 'sZxf9HcBfFDkyb-x_nuK', '_id': 'bfc80b63e6be12a61ddbb40b997163ee', '_seqNo': 121, '_primaryTerm': 2, 'id': 'bfc80b63e6be12a61ddbb40b997163ee'}]"
}

It seems that it wants to add a new tag or consider that it has been added, but it haven't:

Also, if we consider this one as a UI bug, and try to execute Gmail_UnblockSender, we will see the same error:

So the problem is that Cortex for some reason cannot add the tag, and does it silently. At the same time it somehow added the "gmail:handled" tag.

So I was about to ask what permissions did you provide cortex user account with, but decided to add write permissions to it and check if it will start working (it had only read permission before).

So it did. Now it creates tags, removes tags, does its job, works fine.
Tomorrow will test the rest of the responders from your PR.

Thank you for your work and for your patience, man!

[strassi]

So I was about to ask what permissions did you provide cortex user account with, but decided to add write permissions to it and check if it will start working (it had only read permission before).

I see the problem. For my tests I used the API key of the thehive admin user in the training VM. This user does have write permissions.

I just realized, that the API provides a get_current_user function, which returns a user object. Maybe the user object does have a property roles. This could be checked for read,write. The responder could fail if it does not have sufficient user rights. This might fix these kind of problems in the future.

Nice analysis @yugoslavskiy; would not have thought of the api key permissions

[yugoslavskiy]

Hello @strassi!

I've tested the other functions and everything works smoothly.
I believe we can summon maintainers and ask them for merge (:

@nadouani @jeromeleonard guys, is there anything else required from our side to proceed with the merging?

[yugoslavskiy]

Hello @nadouani @jeromeleonard!

Is there anything required from our side to proceed with merging? (:

[yugoslavskiy]

Hello @strassi! Could you please change the base (target branch) to the feature/oscd?
We would like to have 1 final PR with the entire OSCD sprint work.