[OSCD Initiative] Develop Responder for Gmail

[yugoslavskiy]

Feature description

Responder for Gmail that would be able to execute the following Response Actions:

• RA4201: Delete email message
• RA3201: Block domain on email
• RA3202: Block sender on email
• RA5201: Unblock domain on email
• RA5202: Unblock sender on email

Describe the solution you'd like

It could be done via Gmail API using the following methods:

• Delete email message — https://developers.google.com/gmail/api/v1/reference/users/messages/delete
• Block domain on email — https://developers.google.com/gmail/api/guides/filter_settings (automatic redirection to trash)
• Block sender on email — https://developers.google.com/gmail/api/guides/filter_settings (automatic redirection to trash)
• Unblock domain on email — https://developers.google.com/gmail/api/guides/filter_settings (disabling filter)
• Unblock sender on email — https://developers.google.com/gmail/api/guides/filter_settings (disabling filter)

[strassi]

I would implement this responder. @yugoslavskiy do you have some hints if the API is the same for G Suite and private GMail?

[strassi]

OK. If I understood correctly everything is based on the OAuth 2.0 Authentication / Authorization. So the API is the same for GSuite/Google Workplace and a private Gmail Account.

According to

• https://support.google.com/a/answer/7281227?hl=en&ref_topic=10021546
• https://support.google.com/a/answer/162106?hl=en&ref_topic=10021546

Apps can be given a service account which authenticates via OAuth 2.0. This service account is able to connect to G Suite / Google Workspace data without individual user consent. Otherwise users must authorize the app access to their data.

OAuth 2.0 authorizes actions via scopes. The valid Gmail scopes can be under https://developers.google.com/identity/protocols/oauth2/scopes#gmail. For the feature we might need the following scopes:

Action	Scope
Delete	https://mail.google.com/
Block	https://www.googleapis.com/auth/gmail.settings.basic
Unblock	https://www.googleapis.com/auth/gmail.settings.basic

The API endpoint https://developers.google.com/gmail/api/reference/rest/v1/users can then be used to manipulate Gmail data for a given user. The userId is the email address of a user mailbox.

It shall be mentioned, that Gmail also has usage limites for the Gmail API: https://developers.google.com/gmail/api/reference/quota

Daily usage | 1,000,000,000 quota units per day for your application.
Per user rate limit | 250 quota units per user per second, moving average (allows short bursts).

The responders per-method cost is as follows:

Method	Unit Cost
messages.delete	10
settings.filters.create	5
settings.filters.delete	5

By the nature of this responder I do not think any rate limit issues will arise.

[strassi]

Just found out, that interaction with the Gmail API in this case needs a service account. A normal OAuth 2.0 Flow is not going to work.

Here is a link to the documentation how to create a service account : https://support.google.com/a/answer/7378726?hl=en

[strassi]

I'm currently stuck on this issue googleapis/google-api-nodejs-client#2322. Since I do not have a G Suite account I cannot delegate the service account. Working as a real user might be quite inconvenient.

Maybe the respondere could return a OAuth 2.0 URL for the analyst but I'm not sure if this is helping a lot.

[yugoslavskiy]

Hello @strassi ! Thank you for your contribution!
I am sorry for not replying. I will join the discussion tomorrow. Today I've spent all time reviewing Sigma rules.

[yugoslavskiy]

Hello @strassi ! Sorry for the long answers. Lot's of Sigma rules were contributed since Monday.
I have access to the GSuite account, and I can collab on that one with you.
Can you share your current progression? Maybe some code snippets/functions, that I could check?
If I will confirm that these functions are fully operational, we can proceed to work on other functions of the Responder.

[strassi]

I basically need OAuth2 to work. I created this gist https://gist.github.com/strassi/326edd3193e0cf3ea8db3198b09131cc with the authentication method in the responder.

If you could get it to work, this would already help a lot.

[strassi]

@yugoslavskiy if we talk about deleting an email, can I assume an analyst does have the message id of the email in question?

Otherwise I need to search the email based on some specified parameters. This could be from, subject, body strings etc.

[yugoslavskiy]

Hello @strassi! I am sorry for the delays, too many tasks....
I will do it my first priority work out your questions and return back with the oauth solution tomorrow during the day.

[yugoslavskiy]

Here are the updates:

1. service account created
2. credentials.json downloaded
3. tried to use your code snippet, as well as the others from official documentation, stackoverflow, blogs etc

no results. still have the same:

>>> credentials.valid
False

I think it could be related to the authorization timeout:

So let's see how it will go tomorrow.

[alejandroortuno]

@strassi I can also help with this one and give it a go to your code snipped.

can I assume an analyst does have the message id of the email in question?

I think it makes sense to assume the analyst will have the message id though.

[strassi]

@alejandroortuno Yeah I also think an analyst does have a message id, but it has the disadvantage of not being able to delete phishing messages via a bulk operation.

Proposal

• Block Domain/Block Sender/Unblock Domain/Unblock Sender work on Case Level
• Delete Message works on artifact level (Artifact Type: message_id or other)

Filtering messages

1. Responder gets thehive:case
2. Responder looks up observables of given case (they are not provided with case data)
3. mail artifacts with *@gmail.com will be used as subjects (mailboxes that need filtering)
4. mail/domain artifacts with IoC marking will be used for blocking (depending on the selected responder flavor)
5. the created filter Gmail IDs get added to the case as custom field (used for deleting the filter in case of unblock)

Deleting Message

1. Responder gets thehive:artifact of type message_id or other
2. Responder tags message with TRASH label

A disadvantage of working with a single message_id is, that you can't perform a bulk delete operation for many *@gmail.com addresses.

We could also operate like this:

1. Responder gets thehive:artifact of type gmail_query
2. Responder gets all messages matching the gmail_query
3. responder tags all matching messages with the TRASH label

This responder might be used on phishing cases, so I imagined it might me useful if operations could always be carried out in bulk.

@alejandroortuno @yugoslavskiy any thoughts or additional ideas on this?

[yugoslavskiy]

Hello guys!

Today I've tried to authorize on Gmail API for 6 hours and didn't succeed.
I performed all the steps from the official guideline, trying some other options/approaches, double-checking the configuration, and repeating the steps from scratch.

It seems that there is something that I am missing.

Need some help with it. Does anyone else have access to Gsuite?

[alejandroortuno]

Code snipped seems fine. @yugoslavskiy did you perform the steps for the https://github.com/googleapis/google-api-python-client/blob/master/docs/oauth-server.md#delegating-domain-wide-authority-to-the-service-account as the app will be accessing the user inboxes so this is a required steps. This requires a GSuite enterprise account which I am not in possession for personal purposes.

An idea is we can get in contact with GSuite to get into their enterprise solutions with some of their non-profit programs https://edu.google.com/products/gsuite-for-education/enterprise/ to get us unlocked.

@strassi your proposal looks good!

[yugoslavskiy]

@yugoslavskiy did you perform the steps for the https://github.com/googleapis/google-api-python-client/blob/master/docs/oauth-server.md#delegating-domain-wide-authority-to-the-service-account as the app will be accessing the user inboxes so this is a required steps.

Hello @alejandroortuno! Yep, I performed them.

[strassi]

I now pushed a first implementation into my forked repository.

Service files need to be updated and the authentication is still missing. Maybe I can get a trail of Gsuite and get a service account working.

Sry for breaking the sprint timeframe, but I had multiple things get in my way these two weeks.

[yugoslavskiy]

Hello @strassi! No worries, you didn't break anything. We will finalize existing PRs and everything which is WIP. Technically the sprint ended yesterday, but that's totally fine (: Last year we've been finalizing results for 4 months (: So it's fine, we have plenty of time to find somebody with Gsuit account and properly test the responder.

Thank you very much for your contribution, @strassi! And sorry that I didn't participate much. Lots of work at the moment...

[tuckner]

I've used this in the past to get the audit logs, I imagine it should be similar at least from an auth perspective:

from googleapiclient.discovery import build
from google.oauth2 import service_account
import json

SCOPES = ['https://www.googleapis.com/auth/admin.reports.audit.readonly']
SERVICE_ACCOUNT_FILE = 'credentials.json'

def main():
    credentials = service_account.Credentials.from_service_account_file(
        SERVICE_ACCOUNT_FILE, subject='[email protected]', scopes=SCOPES)
    service = build('admin', 'reports_v1', credentials=credentials)
    results = service.activities().list(userKey='all', applicationName='admin', startTime='2020-01-01T00:00:00Z', endTime='2020-08-08T23:59:59Z', eventName='CHANGE_GMAIL_SETTING', maxResults=30).execute()
    print(json.dumps(results, indent=2))

if __name__ == '__main__':
    main()

The key difference here is that you aren't including an account to impersonate. A way to think about it is that the service account is only performing authentication, while the impersonation account has the authorization IIRC.

edit - oh didn't see your google-test.py file which looks to be what I was suggesting

[strassi]

Yes, this code looks pretty much like my implementations. To manipulate the mailbox of a user one must use the service account credentials and a domain wide delegation.

credentials = service_account.Credentials.from_service_account_file(
        SERVICE_ACCOUNT_FILE, subject='[email protected]', scopes=SCOPES)

# get delegation for each gmail account
credentials.with_subject("[email protected]")

gmail_service = build("gmail", "v1", credentials=credentials)
result = gmail_service.users().messages().trash(userId=observable, id=message_id).execute()

I would love to test this and get a working Gsuite environment to test this responder. Since I'm not a professional developer I need my short and fast trail and error feedback loops 😄

[strassi]

I created a Gsuite trail account. I followed the guideline and it did not work. you basically need to

1. enable a service account via GCP
2. enable Gmail API
3. get service account client_id (oauth approval screens + domain-wide delegation needed)
4. change to Gsuite Admin panel
5. add third party app (security->API controls) with client_id
6. add domain-wide delegation with client_id

The authentication works now. Maybe it didn't work for @yugoslavskiy because of an error in my snippet:

credentials = service_account.Credentials.from_service_account_file(
        service_account_file,
        scopes=scopes,
       subject=subject
    )
if (credentials.valid) and (credentials.has_scopes(scopes)):
        return credentials
    else:
        return None

The credentials are not valid until they are refreshed. The refresh gets you an oauth token which sets the credentials.valid. I removed the credential validity check code in my testing snippet and it worked. credentials.valid is true after you tried to execute the first API call:

print(credentials.valid) # is false
my_emails = gmail_service.users().messages().list(userId=subject).execute()
print(credentials.valid) # is true

[yugoslavskiy]

Thank you, @tuckner, @strassi! You guys ROCK!

Finally, credentials.valid is True. Here is the 100% operational snippet I've created out of your examples:

from google.oauth2 import service_account
from googleapiclient.discovery import build

SUBJECT = "[email protected]"
SCOPES = [ "https://mail.google.com/", "https://www.googleapis.com/auth/gmail.settings.basic"]
SERVICE_ACCOUNT_FILE = "/path/to/credentials.json"

credentials = service_account.Credentials.from_service_account_file(
        SERVICE_ACCOUNT_FILE, subject=SUBJECT, scopes=SCOPES)

gmail_service = build("gmail", "v1", credentials=credentials)
my_emails = gmail_service.users().messages().list(userId=SUBJECT).execute()
print(credentials.valid)

And this way I moved the messages from [email protected] from SUBJECT's mailbox (thanks to @strassi) to trash:

query = "from:[email protected]"
response = gmail_service.users().messages().list(userId=SUBJECT,q=query).execute()

for message in response['messages']:
    gmail_service.users().messages().trash(userId=SUBJECT, id=message['id']).execute()

And permanently deleted them using the following:

for message in response['messages']:
    gmail_service.users().messages().delete(userId=SUBJECT, id=message['id']).execute()

I believe that deleting is our case, but we could develop the responder with both options.

Now we need to figure out how to remove specific email from all users' mailboxes.
@strassi proposed a few ideas here. I think that the stuff with gmail search operators could fly.

[strassi]

I pushed some new code with updated authentication into my fork At the moment I'm fixing bugs on thehive observable updates (like tagging gmail adresses with their filter ids).

The ground work on the blocking, unblocking and deleting of messages is done. Support for custom gmail domains has also been implemented.

[strassi]

Alright I think it is working (at least in my tests xD). I implemented the responder with bulk operations in mind.

The responder behaves as follows:

• You can block mail and domain observables
• Operations are carried out against all gmail addresses in the case (custom domain can be set in the responder config)
• The message ID of deleted E-mails get added as tag to the respective gmail address
• The filter ID of a blocked domain or mail gets added as tag to respective gmail address
• All observables that get blocked/unblocked get a gmail:handled tag
• Messages can only be deleted via Gmail query syntax; this enables one to bulk delete a lot of messages

My fork should be ready to run. Maybe some of you guys can run some test cases and report bugs here if you find one.

[yugoslavskiy]

Great job, @strassi!
I will be able to test your responder next Monday and will provide you the feedback here (:
I've also dropped a few comments in the #891.
Well done!