Details
| ID | Gmail Single User |
|---|---|
| Provider | |
| Category | |
| From Version | 5.0.0 |
| Docker Image | demisto/google-api-py3:1.0.0.10182333 |
| Supported Modules | Agentix Cloud Runtime Security Cloud Posture Security XSIAM EDR Cortex Cloud |
README
Use the Gmail Single User integration to send emails and fetch emails as incidents to Cortex XSOAR.
Note: Use this integration if you only want to fetch and send emails from a single user’s mailbox. If you need to access multiple user mailboxes, use the GMail Integration.
Configure Gmail Single User on Cortex XSOAR
- Navigate to Settings > Integrations > Servers & Services.
- Search for Gmail Single User.
- Click Add instance to create and configure a new integration instance.
| Parameter Name | Description | Required |
|---|---|---|
| Gmail of the user | True | |
| send_as | Allows you to specify the e-mail address from which to send e-mails from | False |
| code | Auth Code (run the !gmail-auth-link command to start the auth flow - see the Application Authorization Flow section) | False |
| client_id | Client ID (see Application Authorization Flow section) | True |
| client_secret | Client Secret (see Application Authorization Flow section) | True |
| redirect_uri | Auth Redirect URI (see Application Authorization Flow section) | True |
| isFetch | Fetch incidents | False |
| fetch_time | First fetch timestamp, in days | False |
| query | Events query (e.g., “from:example@demisto.com”) | False |
| fetch_limit | Maximum number of emails to pull per fetch. Default is 50. The maximum number is 200 emails per fetch (even if a higher number is configured). |
False |
| insecure | Trust any certificate (not secure) | False |
| proxy | Use system proxy settings | False |
Application Authorization Flow
- To allow Cortex XSOAR to access Gmail, you will need create your own OAuth2 app.
- Once you have the app, follow the steps in Authorization Flow In Cortex XSOAR to configure the OAuth 2.0 authorization.
Create Your OAuth2 App
To use the GMail Single User integration, you will need to create your own OAuth2 Google app.
- Go to the developers credentials page (you may need to set up a new project if you haven’t already).
- If needed, configure the Consent Screen. Fill in the Consent Screen information you want to display to your users.
- Make sure in the consent screen that you publish the app by clicking
Publish Appand confirming.

- In the credentials page choose:
Create Credentials->OAuth client ID.

- When creating the OAuth client ID, select Web application as the application type.
- Name the credential. For example:
Cortex XSOAR GMail Integration. - Add an Authorized redirect URIs:
https://oproxy.demisto.ninja/authcode. Theoproxyurl is a client side only web page which provides an easy interface to copy the obtained auth code from the authorization respone to the integration configuration in the authorization flow steps. Optionally: if you don’t want to use theoproxyurl, you may use alocalhosturl on a port which is not used locally on your machine. For example:http://localhost:9004. You will then need to copy the code from the url address bar in the response (see Authorization Flow In Cortex XSOAR).

- Make sure to enable the Gmail API if you haven’t already.
- After you create the app, copy the client id and client secret of the app that you created to the integration configuration. If you want to use a
localhostredirect URI, copy the chosen auth redirect URI to the integration configuration. - Proceed to Authorization Flow In Cortex XSOAR to configure OAuth 2.0 authorization in Cortex XSOAR.
Deprecated: Use the Demisto App for GSuite Admins (OOB Authorization)
Note: This is an old method of authentication which was provided for GSuite Admins via the use of the Demisto App. This method uses the OOB authentication flow which has been deprecated by Google. This method of authentication will stop working on Jan 31, 2023. Instructions are provided here for backwards compatibility until the official end of life of OOB authentication. All customers must migrate to the web app authorization flow using their own Google App as documented at Create Your OAuth2 App. For further details See:
- https://developers.google.com/identity/protocols/oauth2/resources/oob-migration
- https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html
To configure the use of the Demisto App for GSuite Admins there is no need to create your own app (you will be using the predefined Demisto App). Thus, there is no need to fill in the Client ID and Client Secret configuration fields.
GSuite Admins can choose to trust the Demisto App so users can configure the app:
- Go to App Access Control.
- Choose:
Configure new app->OAuth App Name Or Client ID.

- Enter the following Client ID:
391797357217-pa6jda1554dbmlt3hbji2bivphl0j616.apps.googleusercontent.com
You see theDemisto Appin the results page.

- Select the app and grant the app access as
Trusted. - Add the Demisto app client ID
391797357217-pa6jda1554dbmlt3hbji2bivphl0j616.apps.googleusercontent.comto the integration configuration. - Proceed to ‘Authorization Flow In Cortex XSOAR’ to configure OAuth 2.0 authorization in Cortex XSOAR.
Authorization Flow In Cortex XSOAR
- Create and save an integration instance of the Gmail Single User integration. Do not fill in the Auth Code field, this is obtained in the next steps.
- To obtain the Auth Code, run the following command in the playground: !gmail-auth-link. Access the link you receive to authenticate your Gmail account.
- If you get a message from Google saying that it cannot verify the application, click proceed and click enter for the app name to give the app you created permissions to your Google account. Then click proceed.
- Complete the authentication process.
- If using the
oproxyredirect URI, you will be redirected to a page which will provide an option to copy the auth code. Note that this page processes the code fully in the browser (client side) and displays it to be copied. The code is not transmitted to the oproxy web server. Example screenshot:

If you are using thelocalhostredirect URI, you will be redirected to an unavailable page. Copy the code, including thecode=prefix up to the first&from the URL address bar. Example screenshot:

- Copy the received code to the Auth Code configuration parameter of the integration instance.
- Save the instance.
- To verify that authentication was configured correctly, run the !gmail-auth-test.
Fetched Incidents Data
- Incident Name
- Occurred
- Owner
- Type
- Severity
- Email From
- Email Message ID
- Email Subject
- Email To
- Attachment Extension
- Attachment Name
- Email Body
- Email Body Format
Commands
You can execute these commands from the Cortex XSOAR CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.
send-mail
Sends an email using Gmail.
Base Command
send-mail
Input
| Argument Name | Description | Required |
|---|---|---|
| to | The email addresses of the receiver. | Required |
| body | The content (body) of the email to be sent in plain text. | Optional |
| subject | The subject of the email to be sent. | Required |
| attachIDs | A comma-separated list of IDs of War Room entries that contain files, which need be attached to the email. | Optional |
| cc | The CC additional recipient email address. | Optional |
| bcc | The BCC additional recipient email address. | Optional |
| htmlBody | The content (body) of the email to be sent in HTML format. | Optional |
| replyTo | The email address used to reply to the message. | Optional |
| attachNames | A comma-separated list of new names for attachments, according to the order they were attached to the email. For example, to rename the first and third file: attachNames=new_fileName1,,new_fileName3 To rename the second and fifth files: attachNames=,new_fileName2,,,new_fileNam |
Optional |
| attachCIDs | A comma-separated list of CID images to embed attachments in the email. | Optional |
| transientFile | The text name for an attached file. Multiple files are supported as a comma-separated list. For example, transientFile=”t1.txt,temp.txt,t3.txt” transientFileContent=”test 2,temporary file content,third file content” transientFileCID=”t1.txt@xxx.yyy,t2.txt@xxx.zzz”. | Optional |
| transientFileContent | The content for the attached file. Multiple files are supported as a comma-separated list. For example, transientFile=”t1.txt,temp.txt,t3.txt” transientFileContent=”test 2,temporary file content,third file content” transientFileCID=”t1.txt@xxx.yyy,t2.txt@xxx.zzz”. | Optional |
| transientFileCID | The CID image for an attached file to include within the email body. Multiple files are supported as comma-separated list. For example, transientFile=”t1.txt,temp.txt,t3.txt” transientFileContent=”test 2,temporary file content,third file content” transientFileCID=”t1.txt@xxx.yyy,t2.txt@xxx.zzz”. | Optional |
| additionalHeader | A comma-separated list of additional headers in the format headerName=headerValue. For example, “headerName1=headerValue1,headerName2=headerValue2”. | Optional |
| templateParams | ‘Replaces {varname} variables with values from this parameter. Expected values are in the form of a JSON document. For example, {“varname” :{“value” “some value”, “key”: “context key”}}. Each var name can either be provided with the value or a context key to retrieve the value. Note that only context data is accessible for this argument, while incident fields are not.’ | Optional |
Context Output
| Path | Type | Description |
|---|---|---|
| Gmail.SentMail.ID | String | The immutable ID of the message. |
| Gmail.SentMail.Labels | String | A list of label IDs applied to this message. |
| Gmail.SentMail.ThreadId | String | The ID of the thread in which the message belongs. |
| Gmail.SentMail.To | String | The email recipient. |
| Gmail.SentMail.From | String | The email sender. |
| Gmail.SentMail.Cc | String | The additional CC recipient email address. |
| Gmail.SentMail.Bcc | String | The additional BCC recipient email address. |
| Gmail.SentMail.Subject | String | The email subject. |
| Gmail.SentMail.Body | String | The plain-text version of the email. |
| Gmail.SentMail.MailBox | String | The mailbox from which the mail was sent. |
| Gmail.SentMail.BodyHTML | String | The HTML version of the email. |
Command Example
!send-mail subject="this is the subject" to=test@demistodev.com body="this is the body"
Context Example
{
"Gmail.SentMail": [
{
"Body": "this is the body",
"From": "example@demisto.com",
"Cc": null,
"Labels": [
"SENT"
],
"Bcc": null,
"To": "test@demistodev.com",
"ThreadId": "16f662789d3a2972",
"Mailbox": "test@demistodev.com",
"Type": "Gmail",
"ID": "16f662789d3a2972",
"Subject": "this is the subject"
}
]
}
Human Readable Output
Email sent
Type ID To From Subject Body Labels ThreadId Gmail 16f662789d3a2972 test@demistodev.com example@demisto.com this is the subject this is the body SENT 16f662789d3a2972
gmail-auth-link
Starts the OAuth2 process. Gets a link to use to authenticate to Gmail.
Base Command
gmail-auth-link
Input
There is no input for this command.
Context Output
There is no context output for this command.
Command Example
#### Human Readable Output
## Gmail Auth Link
Please follow the following **link**.
After Completing the authentication process, copy the received code
to the **Auth Code** configuration parameter of the integration instance.
Save the integration instance and then run *!gmail-auth-test* to test that
the authentication is properly set.
### gmail-auth-test
***
Tests that Gmail auth is configured properly. Use this command after completing the OAuth2 authentication process.
#### Base Command
`gmail-auth-test`
#### Input
There is no input for this command.
#### Context Output
There is no context output for this command.
#### Command Example
```!gmail-auth-test```
#### Human Readable Output
Authentication test completed successfully.
### reply-mail
***
Replies to a mail using Gmail.
#### Limitations
The *subject* argument should be the same as the subject of the email you are replying to in order for the reply to be a part of the same conversation.
#### Base Command
`reply-mail`
#### Input
| **Argument Name** | **Description** | **Required** |
| --- | --- | --- |
| to | Email addresses of the recipients. | Required |
| from | Email address of the sender. | Optional |
| body | The contents (body) of the email to be sent in plain text. | Optional |
| subject | Subject of the email to be sent. <br/> Should be the same as the subject of the email you are replying to in order for the reply to be a part of the same conversation. | Required |
| inReplyTo | A comma-separated list of message IDs to reply to. | Required |
| references | A comma-separated list of message IDs to refer to. | Optional |
| attachIDs | A comma-separated list of IDs of War Room entries that contain the files that need be attached to the email. | Optional |
| cc | Additional recipient email addresses (CC). | Optional |
| bcc | Additional recipient email addresses (BCC). | Optional |
| htmlBody | The contents (body) of the email to be sent in HTML format. | Optional |
| replyTo | Address that needs to be used to reply to the message. | Optional |
| attachNames | A comma-separated list of new names to rename attachments corresponding to the order in which they were attached to the email.<br/> Examples - To rename first and third file attachNames=new_fileName1,,new_fileName3<br/> To rename second and fifth files attachNames=,new_fileName2,,,new_fileName5 | Optional |
| attachCIDs | A comma-separated list of CID images to embed attachments inside the email. | Optional |
| transientFile | Textual name for an attached file. Multiple files are supported as a<br/> comma-separated list. For example, transientFile="t1.txt,temp.txt,t3.txt" transientFileContent="test<br/> 2,temporary file content,third file content" transientFileCID="t1.txt@xxx.yyy,t2.txt@xxx.zzz" | Optional |
| transientFileContent | Content for the attached file. Multiple files are supported as a comma-separated<br/> list. For example, transientFile="t1.txt,temp.txt,t3.txt" transientFileContent="test<br/> 2,temporary file content,third file content" transientFileCID="t1.txt@xxx.yyy,t2.txt@xxx.zzz" | Optional |
| transientFileCID | CID image for an attached file to include within the email body. Multiple files are<br/> supported as comma-separated list. For example, transientFile="t1.txt,temp.txt,t3.txt"<br/> transientFileContent="test 2,temporary file content,third file content" transientFileCID="t1.txt@xxx.yyy,t2.txt@xxx.zzz" | Optional |
| additionalHeader | A comma-separated list of additional headers in the format: headerName=headerValue. For example: "headerName1=headerValue1,headerName2=headerValue2". | Optional |
| templateParams | 'Replaces {varname} variables with values from this parameter. Expected<br/> values are in the form of a JSON document. For example, {"varname" :{"value" "some<br/> value", "key": "context key"}}. Each var name can either be provided with<br/> the value or a context key to retrieve the value.<br/> Note that only context data is accessible for this argument, while incident fields are not.' | Optional |
#### Context Output
| **Path** | **Type** | **Description** |
| --- | --- | --- |
| Gmail.SentMail.ID | String | The immutable ID of the message. |
| Gmail.SentMail.Labels | String | List of IDs of the labels applied to this message. |
| Gmail.SentMail.ThreadId | String | The ID of the thread in which the message belongs. |
| Gmail.SentMail.To | String | The recipients of the email. |
| Gmail.SentMail.From | Unknown | The sender of the email. |
| Gmail.SentMail.Cc | String | Additional recipient email addresses \(CC\). |
| Gmail.SentMail.Bcc | String | Additional recipient email addresses \(BCC\). |
| Gmail.SentMail.Subject | String | The subject of the email. |
| Gmail.SentMail.Body | Unknown | The plain-text version of the email. |
| Gmail.SentMail.MailBox | String | The mailbox from which the mail was sent. |
#### Command Example
```!reply-mail subject="this is the subject" to=test@demistodev.com replyTo=test@demistodev.com body="this is the body" inReplyTo=<CAEvnzx+zEeFJ1U5g4FOfHKeWe-H3hU7kGiKaK7q0F0A@mail.gmail.com> references=<CAEvnzx+zEeFJ1U5g4FOfHKeWe-H3hU7kGiKaK7q0F0A@mail.gmail.com>```
#### Context Example
{
“Gmail.SentMail”: [
{
“Bcc”: null,
“Body”: “this is the body”,
“Cc”: null,
“From”: “admin@demistodev.com”,
“ID”: “16d43287fc29b71a”,
“Labels”: [
“SENT”
],
“Mailbox”: “test@demistodev.com”,
“Subject”: “this is the subject”,
“ThreadId”: “16d43287fc29b71a”,
“To”: “test@demistodev.com”,
“Type”: “Gmail”
}
]
}
```
Human Readable Output
Email sent
Type ID To From Subject Body Labels ThreadId Gmail 16d43287fc29b71a test@demistodev.com admin@demistodev.com this is the subject this is the body SENT 16d43287fc29b71a
gmail-get-attachments
Retrieves attachments from a sent Gmail message.
Base Command
gmail-get-attachments
Input
| Argument Name | Description | Required |
|---|---|---|
| message-id | The ID of the message to retrieve. | Required |
| user-id | The user’s email address. The “me” special value can be used to indicate the authenticated user. | Required |
Command Example
!gmail-get-attachments message-id=16d4316a25a332e4 user-id=admin@demistodev.com
gmail-get-incidents
Retrieves a list of emails from Gmail within a specific date range and formats them as incidents.
Base Command
gmail-get-incidents
Input
| Argument Name | Description | Required |
|---|---|---|
| max_results | Maximum number of emails to retrieve. Default is 10. | Optional |
| labels | List of label IDs to filter emails. | Optional |
| query | Additional Gmail search query string. | Optional |
| subject | Filter emails by subject. | Optional |
| from | Filter emails sent from a specific sender. | Optional |
| to | Filter emails sent to a specific recipient. | Optional |
| filename | Filter emails by attached filename. | Optional |
| in | Specify the email folder to search (e.g., ‘inbox’, ‘sent’). | Optional |
| has_attachments | Whether to filter emails that contain attachments. Possible values are: true, false. Default is false. | Optional |
| before | Filter emails before this date. | Optional |
| after | Filter emails after this date. | Optional |
Context Output
There is no context output for this command.
Command Example
!gmail-get-incidents from="admin@dummy.com"
Configuration parameters
email— Email of user (required)send_as— Send ascredentials— Client IDclient_id— Client IDclient_secret— Client Secretredirect_uri— Auth Redirect URIauth_code_creds—code— Auth Code (run the !gmail-auth-link command to start the auth flow - see Help section)incidentType— Incident typeincidentFetchInterval— Incidents Fetch IntervalisFetch— Fetch incidentsfetch_time— First fetch timestamp, in days.query— Events query (e.g., "from:example@demisto.com")fetch_limit— Maximum number of emails to pull per fetchinsecure— Trust any certificate (not secure)proxy— Use system proxy settingslegacy_name— Use legacy attachment name
Commands (6)
-
gmail-auth-linkStarts the OAuth2 process. Get a link to use to authenticate to Gmail.
-
gmail-auth-testTests that Gmail auth is configured properly. Use this command after completing the OAuth2 authentication process.
-
gmail-get-attachmentsRetrieves attachments from a sent Gmail message.
-
gmail-get-incidentsRetrieves a list of emails from Gmail within a specific date range and formats them as incidents.
-
reply-mailReplies to a mail using Gmail.
-
send-mailSends an email using Gmail.
import uuid from CommonServerPython import * """ IMPORTS """ import re import json import base64 from datetime import datetime, timedelta, UTC from email.utils import parsedate_to_datetime, format_datetime import httplib2 from httplib2 import socks import sys from html.parser import HTMLParser from html.entities import name2codepoint from email.mime.audio import MIMEAudio from email.mime.base import MIMEBase from email.mime.image import MIMEImage from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText from email.mime.application import MIMEApplication from email.header import Header import mimetypes import random import string from apiclient import discovery from oauth2client.client import AccessTokenCredentials from googleapiclient.discovery_cache.base import Cache import itertools as it import urllib.parse import secrets import hashlib from googleapiclient.errors import HttpError """ GLOBAL VARS """ params = demisto.params() EMAIL = params.get("email", "") SEND_AS = params.get("send_as") PROXIES = handle_proxy() DISABLE_SSL = params.get("insecure", False) FETCH_TIME = params.get("fetch_time", "1 days") MAX_FETCH = int(params.get("fetch_limit") or 50) AUTH_CODE = params.get("auth_code_creds", {}).get("password") or params.get("code") AUTH_CODE_UNQUOTE_PREFIX = "code=" LEGACY_NAME = argToBoolean(params.get("legacy_name", False)) LOOK_BACK_IN_MINUTES = 1 OOB_CLIENT_ID = "391797357217-pa6jda1554dbmlt3hbji2bivphl0j616.apps.googleusercontent.com" # guardrails-disable-line CLIENT_ID = params.get("credentials", {}).get("identifier") or params.get("client_id") or OOB_CLIENT_ID CLIENT_SECRET = params.get("credentials", {}).get("password") or params.get("client_secret") REDIRECT_URI = params.get("redirect_uri") TOKEN_URL = "https://www.googleapis.com/oauth2/v4/token" TOKEN_FORM_HEADERS = { "Content-Type": "application/x-www-form-urlencoded", "Accept": "application/json", } EXECUTION_METRICS = ExecutionMetrics() """ HELPER FUNCTIONS """ def execute_gmail_action(service, action: str, action_kwargs: dict) -> dict: """Executes a specified action on the Gmail API, while collecting execution metrics. This function dynamically executes an action such as sending an email, retrieving an email, getting attachments, or listing emails based on the specified action and its arguments. Args: service: The Gmail API service instance. action (str): The action to perform. Supported actions are "send", "get", "get_attachments", and "list". action_kwargs (dict): The keyword arguments required for the specified action. Returns: dict: The result from the Gmail API call. Raises: HttpError: If an error occurs from the Gmail API request. Exception: If a general error occurs during the function execution. Note: This function updates execution metrics counters based on the outcome of the API call. """ try: match action: case "send": result = service.users().messages().send(**action_kwargs).execute() case "get": result = service.users().messages().get(**action_kwargs).execute() case "get_attachments": result = service.users().messages().attachments().get(**action_kwargs).execute() case "list": result = service.users().messages().list(**action_kwargs).execute() case _: raise ValueError(f"Unsupported action: {action}") except HttpError as error: if error.status_code == 429: EXECUTION_METRICS.quota_error += 1 elif error.reason == "Unauthorized": EXECUTION_METRICS.auth_error += 1 else: EXECUTION_METRICS.general_error += 1 raise except Exception: EXECUTION_METRICS.general_error += 1 raise EXECUTION_METRICS.success += 1 return result def return_metrics(): if EXECUTION_METRICS.metrics is not None and ExecutionMetrics.is_supported(): return_results(EXECUTION_METRICS.metrics) else: demisto.debug("Not returning metrics. Either metrics are not supported in this XSOAR version, or none were collected") # See: https://github.com/googleapis/google-api-python-client/issues/325#issuecomment-274349841 class MemoryCache(Cache): _CACHE: dict = {} def get(self, url): return MemoryCache._CACHE.get(url) def set(self, url, content): MemoryCache._CACHE[url] = content class TextExtractHtmlParser(HTMLParser): def __init__(self): HTMLParser.__init__(self) self._texts: list = [] self._ignore = False def handle_starttag(self, tag, attrs): if tag in ("p", "br") and not self._ignore: self._texts.append("\n") elif tag in ("script", "style"): self._ignore = True def handle_startendtag(self, tag, attrs): if tag in ("br", "tr") and not self._ignore: self._texts.append("\n") def handle_endtag(self, tag): if tag in ("p", "tr"): self._texts.append("\n") elif tag in ("script", "style"): self._ignore = False def handle_data(self, data): if data and not self._ignore: stripped = data.strip() if stripped: self._texts.append(re.sub(r"\s+", " ", stripped)) def handle_entityref(self, name): if not self._ignore and name in name2codepoint: self._texts.append(chr(name2codepoint[name])) def handle_charref(self, name): if not self._ignore: if name.startswith("x"): c = chr(int(name[1:], 16)) else: c = chr(int(name)) self._texts.append(c) def get_text(self): return "".join(self._texts) class Client: def html_to_text(self, html): parser = TextExtractHtmlParser() try: parser.feed(html) parser.close() except html.parser.HTMLParseError: pass return parser.get_text() def get_http_client_with_proxy(self): https_proxy = PROXIES.get("https") proxy_info = None if https_proxy: if not https_proxy.startswith("https") and not https_proxy.startswith("http"): https_proxy = "https://" + https_proxy parsed_proxy = urllib.parse.urlparse(https_proxy) proxy_info = httplib2.ProxyInfo( # disable-secrets-detection proxy_type=socks.PROXY_TYPE_HTTP, # disable-secrets-detection proxy_host=parsed_proxy.hostname, proxy_port=parsed_proxy.port, proxy_user=parsed_proxy.username, proxy_pass=parsed_proxy.password, ) return httplib2.Http(proxy_info=proxy_info, disable_ssl_certificate_validation=DISABLE_SSL) def get_service(self, serviceName, version): token = self.get_access_token() credentials = AccessTokenCredentials(token, "Demisto Gmail integration") if PROXIES or DISABLE_SSL: http_client = credentials.authorize(self.get_http_client_with_proxy()) return discovery.build(serviceName, version, http=http_client, cache=MemoryCache()) return discovery.build(serviceName, version, credentials=credentials, cache=MemoryCache()) def get_refresh_token(self, integration_context): # use cached refresh_token only if auth code hasn't changed. If it has we will try to obtain a new # refresh token if integration_context.get("refresh_token") and integration_context.get("code") == AUTH_CODE: return integration_context.get("refresh_token") if not AUTH_CODE: raise ValueError("Auth code not set. Make sure to follow the auth flow. Start by running !gmail-auth-link.") refresh_prefix = "refresh_token:" if AUTH_CODE.startswith(refresh_prefix): # for testing we allow setting the refresh token directly demisto.debug("Using refresh token set as auth_code") return AUTH_CODE[len(refresh_prefix) :] demisto.info(f"Going to obtain refresh token from google's oauth service. For client id: {CLIENT_ID}") code = AUTH_CODE if AUTH_CODE.lower().startswith(AUTH_CODE_UNQUOTE_PREFIX): code = urllib.parse.unquote(AUTH_CODE[len(AUTH_CODE_UNQUOTE_PREFIX) :]) body = { "client_id": CLIENT_ID, "grant_type": "authorization_code", "code": code, } if not CLIENT_SECRET: # old OOB authentication verifier = integration_context.get("verifier") if not verifier: raise ValueError("Missing verifier. Make sure to follow the auth flow. Start by running !gmail-auth-link.") body["code_verifier"] = verifier body["redirect_uri"] = "urn:ietf:wg:oauth:2.0:oob" else: body["redirect_uri"] = REDIRECT_URI body["client_secret"] = CLIENT_SECRET h = self.get_http_client_with_proxy() resp, content = h.request(TOKEN_URL, "POST", urllib.parse.urlencode(body), TOKEN_FORM_HEADERS) if resp.status not in {200, 201}: raise ValueError( f"Error obtaining refresh token. Make sure to follow auth flow. {resp.status} {resp.reason} {content}" ) resp_json = json.loads(content) if not resp_json.get("refresh_token"): raise ValueError(f"Error obtaining refresh token. Missing refresh token in response: {content}") return resp_json.get("refresh_token") def get_access_token(self): integration_context = demisto.getIntegrationContext() or {} access_token = integration_context.get("access_token") valid_until = integration_context.get("valid_until") if access_token and valid_until and integration_context.get("code") == AUTH_CODE and self.epoch_seconds() < valid_until: demisto.debug("Using access token from integration context") return access_token refresh_token = self.get_refresh_token(integration_context) demisto.debug(f"Going to obtain access token for client id: {CLIENT_ID}") body = { "refresh_token": refresh_token, "client_id": CLIENT_ID, "grant_type": "refresh_token", } if CLIENT_SECRET: body["client_secret"] = CLIENT_SECRET h = self.get_http_client_with_proxy() resp, content = h.request(TOKEN_URL, "POST", urllib.parse.urlencode(body), TOKEN_FORM_HEADERS) if resp.status not in {200, 201}: msg = "Error obtaining access token. Try checking the credentials you entered." try: demisto.info(f"Authentication failure from server: {resp.status} {resp.reason} {content}") msg += f" Server message: {content}" except Exception as ex: demisto.error(f"Failed parsing error response - Exception: {ex}") raise Exception(msg) parsed_response = json.loads(content) access_token = parsed_response.get("access_token") expires_in = parsed_response.get("expires_in", 3595) time_now = self.epoch_seconds() time_buffer = 5 # seconds by which to shorten the validity period if expires_in - time_buffer > 0: # err on the side of caution with a slightly shorter access token validity period expires_in = expires_in - time_buffer integration_context["access_token"] = access_token integration_context["valid_until"] = time_now + expires_in integration_context["refresh_token"] = refresh_token integration_context["code"] = AUTH_CODE demisto.setIntegrationContext(integration_context) demisto.debug(f"Done obtaining access token for client id: {CLIENT_ID}. Expires in: {expires_in}") return access_token def parse_mail_parts(self, parts: list[dict]): body = "" html = "" attachments: list = [] for part in parts: if "multipart" in part["mimeType"] and part.get("parts"): part_body, part_html, part_attachments = self.parse_mail_parts(part["parts"]) body += part_body html += part_html attachments.extend(part_attachments) elif len(part["filename"]) == 0: text = str(base64.urlsafe_b64decode(part["body"].get("data", "").encode("ascii")), "utf-8") if "text/html" in part["mimeType"]: html += text else: body += text else: if part["body"].get("attachmentId") is not None: content_id = "" is_inline = False attachmentName = part["filename"] for header in part.get("headers", []): if header.get("name") == "Content-ID": content_id = header.get("value").strip("<>") if header.get("name") == "Content-Disposition": is_inline = "inline" in header.get("value").strip("<>") if is_inline and content_id and content_id != "None" and not LEGACY_NAME: attachmentName = f"{content_id}-attachmentName-{attachmentName}" attachments.append( { "ID": part["body"]["attachmentId"], "Name": attachmentName, } ) return body, html, attachments def get_attachments(self, user_id, _id): mail_args = { "userId": user_id, "id": _id, "format": "full", } service = self.get_service("gmail", "v1") result = execute_gmail_action(service, "get", mail_args) result = self.get_email_context(result, user_id)[0] command_args = { "userId": user_id, "messageId": _id, } files = [] for attachment in result.get("Attachments", []): command_args["id"] = attachment["ID"] result = execute_gmail_action(service, "get_attachments", command_args) file_data = base64.urlsafe_b64decode(result["data"].encode("ascii")) files.append((attachment["Name"], file_data)) return files @staticmethod def get_date_from_email_header(header: str) -> datetime | None: """Parse an email header such as Date or Received. The format is either just the date or name value pairs followed by ; and the date specification. For example: by 2002:a17:90a:77cb:0:0:0:0 with SMTP id e11csp4670216pjs; Mon, 21 Dec 2020 12:11:57 -0800 (PST) Args: header (str): header value to parse Returns: Optional[datetime]: parsed datetime """ if not header: return None try: date_part = header.split(";")[-1].strip() res = parsedate_to_datetime(date_part) if res.tzinfo is None: # some headers may contain a non TZ date so we assume utc res = res.replace(tzinfo=UTC) return res except Exception as ex: demisto.debug(f"Failed parsing date from header value: [{header}]. Err: {ex}. Will ignore and continue.") return None @staticmethod def get_occurred_date(email_data: dict) -> tuple[datetime, bool]: """Get the occurred date of an email. The date gmail uses is actually the X-Received or the top Received dates in the header. If fails finding these dates will fall back to internal date. Args: email_data (dict): email to extract from Returns: Tuple[datetime, bool]: occurred datetime, can be used for incrementing search date """ headers = demisto.get(email_data, "payload.headers") if not headers or not isinstance(headers, list): demisto.error(f"couldn't get headers for msg (shouldn't happen): {email_data}") else: # use x-received or received. We want to use x-received first and fallback to received. for name in [ "x-received", "received", ]: header = next(filter(lambda ht: ht.get("name", "").lower() == name, headers), None) if header: val = header.get("value") if val: res = Client.get_date_from_email_header(val) if res: demisto.debug(f"Using occurred date: {res} from header: {name} value: {val}") return res, True internalDate = email_data.get("internalDate") demisto.info(f"couldn't extract occurred date from headers trying internalDate: {internalDate}") if internalDate and internalDate != "0": # internalDate timestamp has 13 digits, but epoch-timestamp counts the seconds since Jan 1st 1970 # (which is currently less than 13 digits) thus a need to cut the timestamp down to size. timestamp_len = len(str(int(time.time()))) if len(str(internalDate)) > timestamp_len: internalDate = str(internalDate)[:timestamp_len] return datetime.fromtimestamp(int(internalDate), tz=UTC), True # we didn't get a date from anywhere demisto.info("Failed finding date from internal or headers. Using 'datetime.now()'") return datetime.now(tz=UTC), False def get_email_context(self, email_data, mailbox) -> tuple[dict, dict, dict, datetime, bool]: """Get the email context from email data Args: email_data (dict): the email data received from the gmail api mailbox (str): mail box name Returns: (context_gmail, headers, context_email, received_date, is_valid_received): note that if received date is not resolved properly is_valid_received will be false """ occurred, occurred_is_valid = Client.get_occurred_date(email_data) context_headers = email_data.get("payload", {}).get("headers", []) context_headers = [{"Name": v["name"], "Value": v["value"]} for v in context_headers] headers = {h["Name"].lower(): h["Value"] for h in context_headers} body = demisto.get(email_data, "payload.body.data") body = body.encode("ascii") if body is not None else "" parsed_body = base64.urlsafe_b64decode(body) base_time = headers.get("date", "") if not base_time or not Client.get_date_from_email_header(base_time): # we have an invalid date. use the occurred in rfc 2822 demisto.debug(f"Using Date base time from occurred: {occurred} instead of date header: [{base_time}]") base_time = format_datetime(occurred) context_gmail = { "Type": "Gmail", "Mailbox": EMAIL if mailbox == "me" else mailbox, "ID": email_data.get("id"), "ThreadId": email_data.get("threadId"), "Labels": ", ".join(email_data.get("labelIds", [])), "Headers": context_headers, "Attachments": email_data.get("payload", {}).get("filename", ""), # only for format 'raw' "RawData": email_data.get("raw"), # only for format 'full' and 'metadata' "Format": headers.get("content-type", "").split(";")[0], "Subject": headers.get("subject"), "From": headers.get("from"), "To": headers.get("to"), # only for format 'full' "Body": str(parsed_body, "utf-8"), # only for incident "Cc": headers.get("cc", []), "Bcc": headers.get("bcc", []), "Date": base_time, "Html": None, } context_email = { "ID": email_data.get("id"), "Headers": context_headers, "Attachments": {"entryID": email_data.get("payload", {}).get("filename", "")}, # only for format 'raw' "RawData": email_data.get("raw"), # only for format 'full' and 'metadata' "Format": headers.get("content-type", "").split(";")[0], "Subject": headers.get("subject"), "From": headers.get("from"), "To": headers.get("to"), # only for format 'full' "Body/Text": str(parsed_body, "utf-8"), "CC": headers.get("cc", []), "BCC": headers.get("bcc", []), "Date": base_time, "Body/HTML": None, } if "text/html" in context_gmail["Format"]: # type: ignore context_gmail["Html"] = context_gmail["Body"] context_gmail["Body"] = self.html_to_text(context_gmail["Body"]) context_email["Body/HTML"] = context_gmail["Html"] context_email["Body/Text"] = context_gmail["Body"] if "multipart" in context_gmail["Format"]: # type: ignore context_gmail["Body"], context_gmail["Html"], context_gmail["Attachments"] = self.parse_mail_parts( email_data.get("payload", {}).get("parts", []) ) context_gmail["Attachment Names"] = ", ".join([attachment["Name"] for attachment in context_gmail["Attachments"]]) # type: ignore context_email["Body/Text"], context_email["Body/HTML"], context_email["Attachments"] = self.parse_mail_parts( email_data.get("payload", {}).get("parts", []) ) context_email["Attachment Names"] = ", ".join([attachment["Name"] for attachment in context_email["Attachments"]]) # type: ignore return context_gmail, headers, context_email, occurred, occurred_is_valid def create_incident_labels(self, parsed_msg, headers): labels = [ {"type": "Email/ID", "value": parsed_msg["ID"]}, {"type": "Email/subject", "value": parsed_msg["Subject"]}, {"type": "Email/text", "value": parsed_msg["Body"]}, {"type": "Email/from", "value": parsed_msg["From"]}, {"type": "Email/html", "value": parsed_msg["Html"]}, ] labels.extend([{"type": "Email/to", "value": to} for to in headers.get("To", "").split(",")]) labels.extend([{"type": "Email/cc", "value": cc} for cc in headers.get("Cc", "").split(",")]) labels.extend([{"type": "Email/bcc", "value": bcc} for bcc in headers.get("Bcc", "").split(",")]) for key, val in headers.items(): labels.append({"type": "Email/Header/" + key, "value": val}) return labels @staticmethod def get_date_isoformat_server(dt: datetime) -> str: """Get the datetime str in the format a server can parse. UTC based with Z at the end Args: dt (datetime): datetime Returns: str: string representation """ return datetime.fromtimestamp(dt.timestamp()).isoformat(timespec="seconds") + "Z" @staticmethod def parse_date_isoformat_server(dt: str) -> datetime: """Get the datetime by parsing the format passed to the server. UTC basded with Z at the end Args: dt (str): datetime as string Returns: datetime: datetime representation """ return datetime.strptime(dt, "%Y-%m-%dT%H:%M:%SZ").replace(tzinfo=UTC) def mail_to_incident(self, msg, service, user_key) -> tuple[dict, datetime, bool]: """Parse an email message Args: msg service user_key Raises: Exception: when problem getting attachments Returns: Tuple[dict, datetime, bool]: incident object, occurred datetime, boolean indicating if date is valid or not """ parsed_msg, headers, _, occurred, occurred_is_valid = self.get_email_context(msg, user_key) # convert occurred to gmt and then isoformat + Z occurred_str = Client.get_date_isoformat_server(occurred) file_names = [] command_args = { "messageId": parsed_msg["ID"], "userId": user_key, } for attachment in parsed_msg["Attachments"]: command_args["id"] = attachment["ID"] result = execute_gmail_action(service, "get_attachments", command_args) file_data = base64.urlsafe_b64decode(result["data"].encode("ascii")) # save the attachment file_result = fileResult(attachment["Name"], file_data) # check for error if file_result["Type"] == entryTypes["error"]: demisto.error(file_result["Contents"]) raise Exception(file_result["Contents"]) file_names.append( { "path": file_result["FileID"], "name": attachment["Name"], } ) incident = { "type": "Gmail", "name": parsed_msg["Subject"], "details": parsed_msg["Body"], "labels": self.create_incident_labels(parsed_msg, headers), "occurred": occurred_str, "attachment": file_names, "rawJSON": json.dumps(parsed_msg), } return incident, occurred, occurred_is_valid def sent_mail_to_entry(self, title, response, to, emailfrom, cc, bcc, bodyHtml, body, subject): gmail_context = [] for mail_results_data in response: gmail_context.append( { "Type": "Gmail", "ID": mail_results_data.get("id"), "Labels": mail_results_data.get("labelIds", []), "ThreadId": mail_results_data.get("threadId"), "To": to, "From": emailfrom, "Cc": cc, "Bcc": bcc, "Subject": subject, "Body": body, "Mailbox": to, "BodyHTML": bodyHtml, } ) headers = ["Type", "ID", "To", "From", "Cc", "Bcc", "Subject", "Body", "Labels", "ThreadId"] return { "ContentsFormat": formats["json"], "Type": entryTypes["note"], "Contents": response, "ReadableContentsFormat": formats["markdown"], "HumanReadable": tableToMarkdown(title, gmail_context, headers, removeNull=True), "EntryContext": {"Gmail.SentMail(val.ID && val.Type && val.ID == obj.ID && val.Type == obj.Type)": gmail_context}, } def epoch_seconds(self, d=None): """ Return the number of seconds for given date. If no date, return current. Args: d (datetime): timestamp Returns: int: timestamp in epoch """ if not d: d = datetime.utcnow() return int((d - datetime.utcfromtimestamp(0)).total_seconds()) def search( self, user_id, subject="", _from="", to="", before="", after="", filename="", _in="", query="", fields=None, label_ids=None, max_results=100, page_token=None, include_spam_trash=False, has_attachments=None, ): query_values = { "subject": subject, "from": _from, "to": to, "before": before, "after": after, "filename": filename, "in": _in, "has": "attachment" if has_attachments else "", } q = " ".join(f"{name}:{value} " for name, value in query_values.items() if value != "") q = (f"{q} {query}").strip() command_args = { "userId": user_id, "q": q, "maxResults": max_results, "fields": fields, "labelIds": label_ids, "pageToken": page_token, "includeSpamTrash": include_spam_trash, } service = self.get_service("gmail", "v1") result = execute_gmail_action(service, "list", command_args) demisto.info( f"Retrieved the following email IDs from the list API call: {[mail.get('id') for mail in result.get('messages', [])]}" ) return [self.get_mail(user_id, mail["id"], "full") for mail in result.get("messages", [])], q def get_mail(self, user_id, _id, _format): command_args = { "userId": user_id, "id": _id, "format": _format, } service = self.get_service("gmail", "v1") return execute_gmail_action(service, "get", command_args) """MAIL SENDER FUNCTIONS""" def randomword(self, length): """ Generate a random string of given length """ letters = string.ascii_lowercase return "".join(random.choice(letters) for i in range(length)) def header(self, s): if not s: return None s_no_newlines = " ".join(s.splitlines()) return Header(s_no_newlines, "utf-8") def template_params(self, paramsStr): """ Translate the template params if they exist from the context """ actualParams = {} if paramsStr: try: params = json.loads(paramsStr) except ValueError as e: return_error(f"Unable to parse templateParams: {str(e)}") # Build a simple key/value for p in params: if params[p].get("value"): actualParams[p] = params[p]["value"] elif params[p].get("key"): actualParams[p] = demisto.dt(demisto.context(), params[p]["key"]) return actualParams else: return None def transient_attachments(self, transientFile, transientFileContent, transientFileCID): if transientFile is None or len(transientFile) == 0: return [] if transientFileContent is None: transientFileContent = [] if transientFileCID is None: transientFileCID = [] attachments = [] for file_name, file_data, file_cid in it.zip_longest(transientFile, transientFileContent, transientFileCID): if file_name is None: break content_type, encoding = mimetypes.guess_type(file_name) if content_type is None or encoding is not None: content_type = "application/octet-stream" main_type, sub_type = content_type.split("/", 1) attachments.append( {"name": file_name, "maintype": main_type, "subtype": sub_type, "data": file_data, "cid": file_cid} ) return attachments def handle_html(self, htmlBody): """ Extract all data-url content from within the html and return as separate attachments. Due to security implications, we support only images here We might not have Beautiful Soup so just do regex search """ attachments = [] cleanBody = "" lastIndex = 0 for i, m in enumerate( re.finditer(r"<img.+?src=\"(data:(image\/.+?);base64,([a-zA-Z0-9+/=\r\n]+?))\"", htmlBody, re.I | re.S) ): maintype, subtype = m.group(2).split("/", 1) name = f"image{i}.{subtype}" cid = f"{name}@{str(uuid.uuid4())[:8]}_{str(uuid.uuid4())[:8]}" attachment = { "maintype": maintype, "subtype": subtype, "data": b64_decode(m.group(3)), "name": name, "cid": cid, "ID": cid, } attachments.append(attachment) cleanBody += htmlBody[lastIndex : m.start(1)] + "cid:" + attachment["cid"] lastIndex = m.end() - 1 cleanBody += htmlBody[lastIndex:] return cleanBody, attachments def collect_inline_attachments(self, attach_cids): """ collects all attachments which are inline - only used in html bodied emails """ inline_attachment = [] if attach_cids is not None and len(attach_cids) > 0: for cid in attach_cids: file = demisto.getFilePath(cid) file_path = file["path"] content_type, encoding = mimetypes.guess_type(file_path) if content_type is None or encoding is not None: content_type = "application/octet-stream" main_type, sub_type = content_type.split("/", 1) fp = open(file_path, "rb") data = fp.read() fp.close() inline_attachment.append( {"ID": cid, "name": file["name"], "maintype": main_type, "subtype": sub_type, "data": data, "cid": cid} ) return inline_attachment return None def collect_manual_attachments(self): attachments = [] for attachment in demisto.getArg("manualAttachObj") or []: res = demisto.getFilePath(os.path.basename(attachment["RealFileName"])) path = res.get("path", "") content_type, encoding = mimetypes.guess_type(path) if content_type is None or encoding is not None: content_type = "application/octet-stream" maintype, subtype = content_type.split("/", 1) if maintype == "text": with open(path) as fp: data = fp.read() else: with open(path, "rb") as fp: # type: ignore data = fp.read() # type: ignore attachments.append( {"name": attachment["FileName"], "maintype": maintype, "subtype": subtype, "data": data, "cid": None} ) return attachments def collect_attachments(self, entry_ids, file_names): """ Creates a dictionary containing all the info about all attachments """ attachments = [] entry_number = 0 if entry_ids is not None and len(entry_ids) > 0: for entry_id in entry_ids: file = demisto.getFilePath(entry_id) file_path = file["path"] if file_names is not None and len(file_names) > entry_number and file_names[entry_number] is not None: file_name = file_names[entry_number] else: file_name = file["name"] content_type, encoding = mimetypes.guess_type(file_name) if content_type is None or encoding is not None: content_type = "application/octet-stream" main_type, sub_type = content_type.split("/", 1) fp = open(file_path, "rb") data = fp.read() fp.close() attachments.append( {"ID": entry_id, "name": file_name, "maintype": main_type, "subtype": sub_type, "data": data, "cid": None} ) entry_number += 1 return attachments def attachment_handler(self, message, attachments): """ Adds the attachments to the email message """ for att in attachments: if att["maintype"] == "text": msg_txt = MIMEText(att["data"], att["subtype"], "utf-8") if att["cid"] is not None: msg_txt.add_header("Content-Disposition", "inline", filename=att["name"]) msg_txt.add_header("Content-ID", "<" + att["cid"] + ">") else: msg_txt.add_header("Content-Disposition", "attachment", filename=att["name"]) message.attach(msg_txt) elif att["maintype"] == "image": msg_img = MIMEImage(att["data"], att["subtype"]) if att["cid"] is not None: msg_img.add_header("Content-Disposition", "inline", filename=att["name"]) msg_img.add_header("Content-ID", "<" + att["cid"] + ">") if att.get("ID"): msg_img.add_header("X-Attachment-Id", att["ID"]) else: msg_img.add_header("Content-Disposition", "attachment", filename=att["name"]) message.attach(msg_img) elif att["maintype"] == "audio": msg_aud = MIMEAudio(att["data"], att["subtype"]) if att["cid"] is not None: msg_aud.add_header("Content-Disposition", "inline", filename=att["name"]) msg_aud.add_header("Content-ID", "<" + att["cid"] + ">") else: msg_aud.add_header("Content-Disposition", "attachment", filename=att["name"]) message.attach(msg_aud) elif att["maintype"] == "application": msg_app = MIMEApplication(att["data"], att["subtype"]) if att["cid"] is not None: msg_app.add_header("Content-Disposition", "inline", filename=att["name"]) msg_app.add_header("Content-ID", "<" + att["cid"] + ">") else: msg_app.add_header("Content-Disposition", "attachment", filename=att["name"]) message.attach(msg_app) else: msg_base = MIMEBase(att["maintype"], att["subtype"]) msg_base.set_payload(att["data"]) if att["cid"] is not None: msg_base.add_header("Content-Disposition", "inline", filename=att["name"]) msg_base.add_header("Content-ID", "<" + att["cid"] + ">") else: msg_base.add_header("Content-Disposition", "attachment", filename=att["name"]) message.attach(msg_base) def send_mail( self, emailto, emailfrom, send_as, cc, bcc, subject, body, htmlBody, entry_ids, replyTo, file_names, attach_cid, manualAttachObj, transientFile, transientFileContent, transientFileCID, additional_headers, templateParams, inReplyTo=None, references=None, ): templateParams = self.template_params(templateParams) if templateParams is not None: if body is not None: body = body.format(**templateParams) if htmlBody is not None: htmlBody = htmlBody.format(**templateParams) attach_body_to = None if htmlBody and not any([entry_ids, file_names, attach_cid, manualAttachObj, body]): # if there is only htmlbody and no attachments to the mail , we would like to send it without attaching the body message = MIMEText(htmlBody, "html") # type: ignore elif body and not any([entry_ids, file_names, attach_cid, manualAttachObj, htmlBody]): # if there is only body and no attachments to the mail , we would like to send it without attaching every part message = MIMEText(body, "plain", "utf-8") # type: ignore elif htmlBody and body and any([entry_ids, file_names, attach_cid, manualAttachObj]): # if all these exist - htmlBody, body and one of the attachment's items, the message object will be: # a MimeMultipart object of type 'mixed' which contains # a MIMEMultipart object of type `alternative` which contains # the 2 MIMEText objects for each body part and the relevant Mime<type> object for the attachments. message = MIMEMultipart("mixed") # type: ignore alt = MIMEMultipart("alternative") message.attach(alt) attach_body_to = alt else: message = MIMEMultipart() # type: ignore if not attach_body_to: attach_body_to = message # type: ignore message["to"] = emailto message["cc"] = cc message["bcc"] = bcc message["from"] = send_as or emailfrom message["subject"] = self.header(subject) message["reply-to"] = replyTo # # The following headers are being used for the reply-mail command. if inReplyTo: message["In-Reply-To"] = self.header(" ".join(inReplyTo.split())) if references: message["References"] = self.header(" ".join(references)) # if there are any attachments to the mail or both body and htmlBody were given if entry_ids or file_names or attach_cid or manualAttachObj or (body and htmlBody): htmlAttachments = [] # type: list inlineAttachments = [] # type: list if htmlBody: htmlBody, htmlAttachments = self.handle_html(htmlBody) msg = MIMEText(htmlBody, "html", "utf-8") attach_body_to.attach(msg) # type: ignore if attach_cid: inlineAttachments = self.collect_inline_attachments(attach_cid) else: # if not html body, cannot attach cids in message transientFileCID = None msg = MIMEText(body, "plain", "utf-8") attach_body_to.attach(msg) # type: ignore attachments = self.collect_attachments(entry_ids, file_names) manual_attachments = self.collect_manual_attachments() transientAttachments = self.transient_attachments(transientFile, transientFileContent, transientFileCID) attachments = attachments + htmlAttachments + transientAttachments + inlineAttachments + manual_attachments self.attachment_handler(message, attachments) if additional_headers is not None and len(additional_headers) > 0: for h in additional_headers: header_name, header_value = h.split("=", 1) message[header_name] = self.header(header_value) encoded_message = base64.urlsafe_b64encode(message.as_bytes()) command_args = {"raw": encoded_message.decode()} emailfrom = emailfrom or EMAIL return self.send_email_request(email_from=emailfrom, body=command_args) def send_email_request(self, email_from: str, body: dict) -> dict: """ Request to sends a mail through Gmail. Args: email_from (str): the email to send an email from. body (dict): email body. Returns: dict: the email send response. """ command_args = {"userId": email_from, "body": body} service = self.get_service("gmail", "v1") return execute_gmail_action(service, "send", command_args) def generate_auth_link(self): """Generate an auth2 link. Returns: str -- Return the link """ url_params = { "scope": "https://www.googleapis.com/auth/gmail.compose https://www.googleapis.com/auth/gmail.send https://www.googleapis.com/auth/gmail.readonly", # noqa: E501 "client_id": CLIENT_ID, "response_type": "code", } if EMAIL: url_params["login_hint"] = EMAIL if not CLIENT_SECRET: # old OOB authentication verifier = secrets.token_hex(64) sha = hashlib.sha256() sha.update(bytes(verifier, "us-ascii")) challenge = str(base64.urlsafe_b64encode(sha.digest()), "us-ascii").rstrip("=") url_params["redirect_uri"] = "urn:ietf:wg:oauth:2.0:oob" url_params["code_challenge"] = challenge url_params["code_challenge_method"] = "S256" integration_context = demisto.getIntegrationContext() or {} integration_context["verifier"] = verifier demisto.setIntegrationContext(integration_context) else: if not REDIRECT_URI.lower().startswith("http"): raise DemistoException(f'Invalid redirect URI: "{REDIRECT_URI}". The URI should start with "http".') if CLIENT_ID == OOB_CLIENT_ID: raise DemistoException( "Client ID is not set. Make sure to set the Client ID param of the integration configuration." ) url_params["redirect_uri"] = REDIRECT_URI url_params["access_type"] = "offline" url_params["prompt"] = "consent" params = urllib.parse.urlencode(url_params) link = f"https://accounts.google.com/o/oauth2/v2/auth?{params}" # noqa: E501 return link def test_module(client): demisto.results("Test is not supported. Please use the following command: !gmail-auth-test.") def mail_command(client: Client, args: dict, email_from, send_as, subject_prefix="", in_reply_to=None, references=None): email_to = args.get("to") body = args.get("body") subject = f"{subject_prefix}{args.get('subject')}" entry_ids = argToList(args.get("attachIDs")) cc = args.get("cc") bcc = args.get("bcc") html_body = args.get("htmlBody") reply_to = args.get("replyTo") attach_names = argToList(args.get("attachNames")) attach_cids = argToList(args.get("attachCIDs")) transient_file = argToList(args.get("transientFile")) transient_file_content = argToList(args.get("transientFileContent")) transient_file_cid = argToList(args.get("transientFileCID")) manual_attach_obj = argToList(args.get("manualAttachObj")) # when send-mail called from within XSOAR (like reports) additional_headers = argToList(args.get("additionalHeader")) template_param = args.get("templateParams") render_body = argToBoolean(args.get("renderBody", False)) body_type = args.get("bodyType", "Text").lower() rendering_body = html_body if body_type == "html" else body result = client.send_mail( email_to, email_from, send_as, cc, bcc, subject, body, html_body, entry_ids, reply_to, attach_names, attach_cids, manual_attach_obj, transient_file, transient_file_content, transient_file_cid, additional_headers, template_param, in_reply_to, references, ) send_mail_result = client.sent_mail_to_entry( "Email sent:", [result], email_to, email_from, cc, bcc, html_body, rendering_body, subject ) if render_body: html_result = {"Type": entryTypes["note"], "ContentsFormat": formats["html"], "Contents": html_body} return [send_mail_result, html_result] return send_mail_result def send_mail_command(client: Client): args = demisto.args() return mail_command(client, args, EMAIL, SEND_AS or EMAIL) def reply_mail_command(client: Client): args = demisto.args() email_from = args.get("from") send_as = args.get("send_as") in_reply_to = args.get("inReplyTo") references = argToList(args.get("references")) return mail_command(client, args, email_from, send_as, "Re: ", in_reply_to, references) def get_attachments_command(client: Client): args = demisto.args() _id = args.get("message-id") attachments = client.get_attachments("me", _id) return [fileResult(name, data) for name, data in attachments] """FETCH INCIDENTS""" def fetch_incidents(client: Client): user_key = "me" query = "" if params.get("query") is None else params.get("query") last_run = demisto.getLastRun() demisto.debug(f"last run: {last_run}") last_fetch = last_run.get("gmt_time") next_last_fetch = last_run.get("next_gmt_time") page_token = last_run.get("page_token") ignore_ids: list[str] = last_run.get("ignore_ids") or [] ignore_list_used = last_run.get("ignore_list_used") or False # can we reset the ignore list if we haven't used it lookback_ids_and_dates: list[tuple[str, str]] = last_run.get("lookback_msg", []) # handle first time fetch - gets current GMT time -1 day if not last_fetch: last_fetch, _ = parse_date_range(date_range=FETCH_TIME, utc=True, to_timestamp=False) last_fetch = str(last_fetch.isoformat(timespec="seconds")) + "Z" # use replace(tzinfo) to make the datetime aware of the timezone as all other dates we use are aware last_fetch = client.parse_date_isoformat_server(last_fetch) if next_last_fetch: next_last_fetch = client.parse_date_isoformat_server(next_last_fetch) else: next_last_fetch = last_fetch + timedelta(seconds=1) service = client.get_service("gmail", "v1") # use seconds for the filter (note that it is inclusive) # see: https://developers.google.com/gmail/api/guides/filtering query += f" after:{int(last_fetch.timestamp())}" max_results = MAX_FETCH if MAX_FETCH > 200: max_results = 200 demisto.debug( f"GMAIL: fetch parameters: user: {user_key} {query=}" f" fetch time: {last_fetch} page_token: {page_token} max results: {max_results}" ) list_command_args = {"userId": user_key, "maxResults": max_results, "pageToken": page_token, "q": query} result = execute_gmail_action(service, "list", list_command_args) incidents = [] emails_ids_and_dates: list[tuple] = [] # so far, so good demisto.debug(f"GMAIL: possible new incidents are {result}") lookback_ids = [msg_id for msg_id, _ in lookback_ids_and_dates] for msg in result.get("messages", []): msg_id = msg["id"] if msg_id in ignore_ids: demisto.info(f"Ignoring msg id: {msg_id} as it is in the ignore list") ignore_list_used = True continue if msg_id in lookback_ids: continue command_kwargs = {"userId": user_key, "id": msg_id} msg_result = execute_gmail_action(service, "get", command_kwargs) incident, occurred, is_valid_date = client.mail_to_incident(msg_result, service, user_key) if not is_valid_date: # if we can't trust the date store the msg id in the ignore list demisto.info(f'appending to ignore list msg id: {msg_id}. name: {incident.get("name")}') ignore_list_used = True ignore_ids.append(msg_id) else: # save new msg date for lookback emails_ids_and_dates.append((msg_id, occurred)) # update last run only if we trust the occurred timestamp if is_valid_date and occurred > next_last_fetch: next_last_fetch = occurred + timedelta(seconds=1) # avoid duplication due to weak time query if (not is_valid_date) or (occurred >= last_fetch): incidents.append(incident) else: demisto.info(f'skipped incident with lower date: {occurred} than fetch: {last_fetch} name: {incident.get("name")}') demisto.info(f"extracted {len(incidents)} incidents") next_page_token = result.get("nextPageToken", "") msg_to_exclude_in_next_fetch = [] if next_page_token: # we still have more results demisto.info( f"keeping current last fetch: {last_fetch} as result has additional pages to fetch." f" token: {next_page_token}. Ignoring incremented last_fatch: {next_last_fetch}" ) else: # if we are not in a tokenized search and we didn't use the ignore ids we can reset it if (not page_token) and (not ignore_list_used) and (len(ignore_ids) > 0): demisto.info(f"resetting ignore list of len: {len(ignore_ids)}") ignore_ids = [] demisto.debug(f"will use new last fetch date (no next page token): {next_last_fetch}") last_fetch = next_last_fetch if emails_ids_and_dates: # Editing the next fetch one minute back to look back. And adding the ids from the last minute to avoid duplicates latest_msg = max([parse_date(client, msg_date, "date") for _, msg_date in emails_ids_and_dates + lookback_ids_and_dates]) lookback_date = latest_msg - timedelta(minutes=LOOK_BACK_IN_MINUTES) # type: ignore msg_to_exclude_in_next_fetch = [ (msg_id, parse_date(client, msg_date)) for msg_id, msg_date in emails_ids_and_dates + lookback_ids_and_dates if parse_date(client, msg_date, "date") >= lookback_date # type: ignore ] new_last_run = { "gmt_time": client.get_date_isoformat_server(last_fetch), "next_gmt_time": client.get_date_isoformat_server(next_last_fetch), "page_token": next_page_token, "ignore_ids": ignore_ids, "ignore_list_used": ignore_list_used, "lookback_msg": msg_to_exclude_in_next_fetch or lookback_ids_and_dates or [], } demisto.debug(f"Gmail: save the following details to LastRun: {new_last_run}") demisto.setLastRun(new_last_run) return incidents def parse_date(client: Client, date: str | datetime, return_type: str = "str") -> datetime | str: """ Parses a date value using the client's server format utilities. Args: client (Client): The client instance with server date parsing methods. date (str | datetime): The date to parse. Can be a string or datetime object. return_type (str, optional): Determines the return format. - If "date": returns a `datetime` object. - If "str": returns an ISO-formatted date string. Defaults to "str". Returns: datetime | str: The date in the format specified by `return_type`. Returns the input unmodified if the type doesn't match the expected conversion. """ if return_type == "date": return client.parse_date_isoformat_server(date) if isinstance(date, str) else date return client.get_date_isoformat_server(date) if isinstance(date, datetime) else date def auth_link_command(client: Client): link = client.generate_auth_link() markdown = f""" ## Gmail Auth Link Please follow the following **[link]({link})**. After Completing the authentication process, copy the received code to the **Auth Code** configuration parameter of the integration instance. Save the integration instance and then run *!gmail-auth-test* to test that the authentication is properly set. """ return { "ContentsFormat": formats["text"], "Type": entryTypes["note"], "Contents": markdown, "ReadableContentsFormat": formats["markdown"], "HumanReadable": markdown, } def auth_test_command(client): client.search("me", "", "", "", "", "", "", "", "", "", [], 10, "", False, False) return "Authentication test completed successfully." def get_unix_date(args: dict) -> tuple[str, str]: """ Converts 'before' and 'after' date arguments to Unix timestamp strings. Args: args (dict): Dictionary containing optional 'before' and 'after' keys. Defaults to 'now' and '1 day ago' respectively if not provided. Returns: tuple[str, str]: A tuple containing the Unix timestamp strings for 'before' and 'after'. Returns empty strings if the input could not be parsed to a datetime. """ before_dt = arg_to_datetime(args.get("before", "now")) after_dt = arg_to_datetime(args.get("after", "1 day ago")) before = str(int(before_dt.timestamp())) if isinstance(before_dt, datetime) else "" after = str(int(after_dt.timestamp())) if isinstance(after_dt, datetime) else "" return before, after def get_incidents_command(client: Client): """ Retrieves a list of emails from Gmail within a specific date range and formats them as incidents. Args: client (GmailClient): The Gmail API client. args (dict): Command arguments, e.g., max_results, labels, before, after. Returns: CommandResults: The list of incidents formatted for Cortex XSOAR. """ args = demisto.args() max_results = int(args.get("max_results", 10)) label_ids = argToList(args.get("labels")) query = args.get("query", "") subject = args.get("subject", "") _from = args.get("from", "") to = args.get("to", "") filename = args.get("filename", "") _in = args.get("in", "") has_attachments = argToBoolean(args.get("has_attachments", "false")) before, after = get_unix_date(args) demisto.debug(f"Fetching emails with filters - subject: {subject}, from: {_from}, to: {to}, before: {before}, after: {after}") emails = [] try: emails, final_query = client.search( user_id="me", subject=subject, _from=_from, to=to, before=before, after=after, filename=filename, _in=_in, query=query, label_ids=label_ids, max_results=max_results, has_attachments=has_attachments, ) demisto.debug(f"\n{emails=}\n") except Exception as e: return_error(f"Failed to fetch emails: {str(e)}") demisto.info(f"search func raw_response: {emails, final_query}") if not emails: demisto.debug("No emails found for the given query.") return CommandResults(readable_output="No incidents found.") hr_emails = [] for email in emails: hr_emails.append( { "id": email.get("id"), "snippet": email.get("snippet"), "labelIds": email.get("labelIds"), "from": next( (h.get("value") for h in email.get("payload", {}).get("headers", []) if h.get("name", "").lower() == "from"), None, ), "internalDatetime": timestamp_to_datestring(email.get("internalDate", 0), is_utc=True), } ) demisto.debug(f"Retrieved {len(emails)} incidents using query: {final_query}\n\n{hr_emails=}\n\n") return CommandResults( readable_output=tableToMarkdown( "Retrieved Gmail Incidents", hr_emails, headers=["from", "snippet", "internalDatetime", "labelIds", "id"] ), raw_response=emails, ) """ COMMANDS MANAGER / SWITCH PANEL """ def main(): # pragma: no cover global EMAIL global SEND_AS command = demisto.command() client = Client() demisto.info(f"Command being called is {command}") commands = { "test-module": test_module, "send-mail": send_mail_command, "reply-mail": reply_mail_command, "fetch-incidents": fetch_incidents, "gmail-auth-link": auth_link_command, "gmail-auth-test": auth_test_command, "gmail-get-attachments": get_attachments_command, } try: if command == "fetch-incidents": demisto.incidents(fetch_incidents(client)) sys.exit(0) if command in commands: demisto.results(commands[command](client)) if command == "gmail-get-incidents": return_results(get_incidents_command(client)) except Exception as e: return_error(f"An error occurred: {e}", error=e) finally: return_metrics() # python2 uses __builtin__ python3 uses builtins if __name__ == "__builtin__" or __name__ == "builtins" or __name__ == "__main__": main()