GoogleCloudLogging

With Google Cloud Logging, users can centralize all their logs in a single location, making it easier to troubleshoot issues and gain insights from their data.

Cloud Services · Google Cloud Logging

Details

IDGoogleCloudLogging
ProviderGoogle
CategoryCloud Services
From Version6.8.0
Docker Imagedemisto/google-api-py3:1.0.0.10182333
Supported ModulesAgentix XSIAM

README

With Google Cloud Logging, users can centralize all their logs in a single location, making it easier to troubleshoot issues and gain insights from their data.

Configure Google Cloud Logging in Cortex

Parameter Description Required
Service Account JSON User’s Service Account key in JSON format. True
Trust any certificate (not secure)   False
Use system proxy settings   False

Commands

You can execute these commands from the CLI, as part of an automation, or in a playbook.
After you successfully execute a command, a DBot message appears in the War Room with the command details.

gcp-logging-log-entries-list


Lists log entries. Use this method to retrieve log entries that originated from a project/folder/organization/billing account.

Required Permissions

This command requires one of the following OAuth scopes:

  • https://www.googleapis.com/auth/logging.read
  • https://www.googleapis.com/auth/logging.admin
  • https://www.googleapis.com/auth/cloud-platform.read-only
  • https://www.googleapis.com/auth/cloud-platform

The command requires one or more of the following IAM permissions on the specified resource:

  • logging.logEntries.list
  • logging.privateLogEntries.list
  • logging.views.access

Base Command

gcp-logging-log-entries-list

Input

Argument Name Description Required
project_name A comma-separated list of projects names of parent resources from which to retrieve log entries. A maximum of 100 resources may be specified. Optional
organization_name A comma-separated list of organizations names of parent resources from which to retrieve log entries. A maximum of 100 resources may be specified. Optional
billing_account_name A comma-separated list of billing accounts names of parent resources from which to retrieve log entries. A maximum of 100 resources may be specified. Optional
folder_name A comma-separated list of folders names of parent resources from which to retrieve log entries. A maximum of 100 resources may be specified. Optional
filter When specified, the results returned are limited to log entries that match the filter. Referencing a parent resource that is not listed in resourceNames will cause the filter to return no results. The maximum length of a filter is 20,000 characters. E.g., “protoPayload.requestMetadata.callerIp:1.1.1.1 AND protoPayload.serviceName:name”. Optional
order_by How the results should be sorted. Possible values are: timestamp asc, timestamp desc. Default is timestamp asc. Optional
limit The maximum number of objects to return. Optional
page_size The maximum number of results to return from this request. Default is 50. If the value is negative or exceeds 1000, the request is rejected. Optional
next_token If present, then retrieve the next batch of results from the preceding call to this method. pageToken must be the value of nextPageToken from the previous response. Optional

Context Output

Path Type Description
GoogleCloudLogging.LogsEntry.logName String The resource name of the log to which this log entry belongs.
GoogleCloudLogging.LogsEntry.resource.type String The monitored resource type.
GoogleCloudLogging.LogsEntry.resource.labels Unknown Values for all of the labels listed in the associated monitored resource descriptor.
GoogleCloudLogging.LogsEntry.resource.labels.project_id Unknown The project ID.
GoogleCloudLogging.LogsEntry.resource.labels.cluster_name Unknown The cluster name.
GoogleCloudLogging.LogsEntry.timestamp String The time the event described by the log entry occurred.
GoogleCloudLogging.LogsEntry.receiveTimestamp String The time the log entry was received by Logging.
GoogleCloudLogging.LogsEntry.severity String The severity of the log entry. The default value is LogSeverity.DEFAULT.
GoogleCloudLogging.LogsEntry.insertId String A unique identifier for the log entry.
GoogleCloudLogging.LogsEntry.httpRequest.requestMethod String The request method.
GoogleCloudLogging.LogsEntry.httpRequest.requestUrl String The scheme (http, https), the host name, the path and the query portion of the URL that was requested.
GoogleCloudLogging.LogsEntry.httpRequest.requestSize String The size of the HTTP request message in bytes, including the request headers and the request body.
GoogleCloudLogging.LogsEntry.httpRequest.status Number The response code indicating the status of response.
GoogleCloudLogging.LogsEntry.httpRequest.responseSize String The size of the HTTP response message sent back to the client, in bytes, including the response headers and the response body.
GoogleCloudLogging.LogsEntry.httpRequest.userAgent String The user agent sent by the client.
GoogleCloudLogging.LogsEntry.httpRequest.remoteIp String The IP address (IPv4 or IPv6) of the client that issued the HTTP request.
GoogleCloudLogging.LogsEntry.httpRequest.serverIp String The IP address (IPv4 or IPv6) of the origin server that the request was sent to.
GoogleCloudLogging.LogsEntry.httpRequest.referer String The referer URL of the request.
GoogleCloudLogging.LogsEntry.httpRequest.latency String The request processing latency on the server, from the time the request was received until the response was sent.
GoogleCloudLogging.LogsEntry.httpRequest.cacheLookup Boolean Whether or not a cache lookup was attempted.
GoogleCloudLogging.LogsEntry.httpRequest.cacheHit Boolean Whether or not an entity was served from cache (with or without validation).
GoogleCloudLogging.LogsEntry.httpRequest.cacheValidatedWithOriginServer Boolean Whether or not the response was validated with the origin server before being served from cache. This field is only meaningful if cacheHit is True.
GoogleCloudLogging.LogsEntry.httpRequest.cacheFillBytes String The number of HTTP response bytes inserted into cache. Set only when a cache fill was attempted.
GoogleCloudLogging.LogsEntry.httpRequest.protocol String Protocol used for the request.
GoogleCloudLogging.LogsEntry.labels Unknown A map of key, value pairs that provides additional information about the log entry. The labels can be user-defined or system-defined.
GoogleCloudLogging.LogsEntry.operation.id String An arbitrary operation identifier. Log entries with the same identifier are assumed to be part of the same operation.
GoogleCloudLogging.LogsEntry.operation.producer String An arbitrary producer identifier. The combination of ID and producer must be globally unique.
GoogleCloudLogging.LogsEntry.operation.first Boolean Set this to True if this is the first log entry in the operation.
GoogleCloudLogging.LogsEntry.operation.last Boolean Set this to True if this is the last log entry in the operation.
GoogleCloudLogging.LogsEntry.trace String The REST resource name of the trace being written to Cloud Trace in association with this log entry.
GoogleCloudLogging.LogsEntry.spanId String The ID of the Cloud Trace span associated with the current operation in which the log is being written.
GoogleCloudLogging.LogsEntry.traceSampled Boolean The sampling decision of the trace associated with the log entry.
GoogleCloudLogging.LogsEntry.sourceLocation.file String Source file name. Depending on the runtime environment, this might be a simple name or a fully-qualified name.
GoogleCloudLogging.LogsEntry.sourceLocation.line String Line within the source file. 1-based; 0 indicates no line number available.
GoogleCloudLogging.LogsEntry.sourceLocation.function String Human-readable name of the function or method being invoked, with optional context such as the class or package name.
GoogleCloudLogging.LogsEntry.split.uid String A globally unique identifier for all log entries in a sequence of split log entries.
GoogleCloudLogging.LogsEntry.split.index Number The index of this LogEntry in the sequence of split log entries.
GoogleCloudLogging.LogsEntry.split.totalSplits Number The total number of log entries that the original LogEntry was split into.
GoogleCloudLogging.LogsEntry.protoPayload.@type String A URI identifying the type.
GoogleCloudLogging.LogsEntry.authenticationInfo.principalEmail String Identifies the principal that granted the role to the service account.
GoogleCloudLogging.LogsEntry.textPayload String The log entry payload, represented as a Unicode string (UTF-8).
GoogleCloudLogging.LogsEntry.jsonPayload Unknown The log entry payload, represented as a structure that is expressed as a JSON object.
GoogleCloudLogging.nextPageToken String If there might be more results than those appearing in this response, then nextPageToken is included.

Command example

!gcp-logging-log-entries-list project_name="project_id"

Context Example

{
    "GoogleCloudLogging": {
        "LogsEntry": [
            {
                "insertId": "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
                "labels": {
                    "authorization.k8s.io/decision": "allow",
                    "authorization.k8s.io/reason": "authorization.k8s.io/reason"
                },
                "logName": "logName1",
                "operation": {
                    "first": true,
                    "id": "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
                    "last": true,
                    "producer": "producer"
                },
                "protoPayload": {
                    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
                    "authenticationInfo": {
                        "principalEmail": "user@example.com"
                    },
                    "authorizationInfo": [
                        {
                            "granted": true,
                            "permission": "permission",
                            "resource": "resource"
                        }
                    ],
                    "methodName": "methodName",
                    "requestMetadata": {
                        "callerIp": "callerIp",
                        "callerSuppliedUserAgent": "callerSuppliedUserAgent"
                    },
                    "resourceName": "resourceName",
                    "serviceName": "serviceName",
                    "status": {
                        "code": 0
                    }
                },
                "receiveTimestamp": "2023-05-06T14:39:56.974311Z",
                "resource": {
                    "labels": {
                        "cluster_name": "cluster_name",
                        "location": "some_location",
                        "project_id": "project_id"
                    },
                    "type": "type"
                },
                "timestamp": "2023-05-06T14:39:56.974311Z"
            },
            {
                "insertId": "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
                "labels": {
                    "authorization.k8s.io/decision": "allow",
                    "authorization.k8s.io/reason": "authorization.k8s.io/reason"
                },
                "logName": "logName2",
                "operation": {
                    "first": true,
                    "id": "XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX",
                    "last": true,
                    "producer": "producer"
                },
                "protoPayload": {
                    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
                    "authenticationInfo": {
                        "principalEmail": "user@example.com"
                    },
                    "authorizationInfo": [
                        {
                            "granted": true,
                            "permission": "permission",
                            "resource": "resource1"
                        }
                    ],
                    "methodName": "methodName",
                    "requestMetadata": {
                        "callerIp": "callerIp",
                        "callerSuppliedUserAgent": "callerSuppliedUserAgent"
                    },
                    "resourceName": "resource1",
                    "serviceName": "appengine.googleapis.com",
                    "status": {
                        "code": 0
                    }
                },
                "receiveTimestamp": "2023-04-06T14:39:56.974311Z",
                "resource": {
                    "labels": {
                        "cluster_name": "cluster_name",
                        "location": "some_location",
                        "project_id": "project_id"
                    },
                    "type": "type"
                },
                "timestamp": "2023-04-06T14:39:56.974311Z"
            }
        ],
        "nextPageToken": "xxx-xxx"
    }
}

Human Readable Output

Lists log entries

TimeStamp Log Name Insert ID Principal Email Type Project ID Cluster Name
2023-05-06T14:39:56.974311Z logName1 XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX user@example.com type project_id cluster_name
2023-04-06T14:39:56.974311Z logName2 XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXX user2@example.com type project_id cluster_name

Next page token

nextPageToken
xxx-xxx

Troubleshooting

If you encounter the following error message: ‘Failed to generate/refresh token. Subject email or service account credentials are invalid. Reason: invalid_grant: Invalid JWT Signature.’, please ensure that your Service Account JSON, permissions and service account scopes are correct.

Configuration parameters

  • credentials — (required)
  • insecure — Trust any certificate (not secure)
  • proxy — Use system proxy settings

Commands (1)

  • gcp-logging-log-entries-list

    Lists log entries. Use this method to retrieve log entries that originated from a project/folder/organization/billing account.

from typing import Any

import demistomock as demisto
import urllib3
from CommonServerPython import *  # noqa # pylint: disable=unused-wildcard-import
from GSuiteApiModule import *  # noqa: E402

from CommonServerUserPython import *  # noqa

# Disable insecure warnings
urllib3.disable_warnings()


""" CONSTANTS """

DATE_FORMAT = "%Y-%m-%dT%H:%M:%SZ"  # ISO8601 format with UTC, default in XSOAR
SCOPES = ["https://www.googleapis.com/auth/logging.read"]
MAX_LIMIT = 1000


""" HELPER FUNCTIONS """


def prepare_gsuite_client() -> GSuiteClient:
    """
    Creates a client.

    Args:
        request_body (dict): The request body.
        client (GSuiteClient): GSuiteClient client.

    Returns:
        A gsuite client.
    """
    credentials_json = demisto.params().get("credentials", {}).get("password")
    verify_certificate = not demisto.params().get("insecure", False)
    proxy = demisto.params().get("proxy", False)
    headers = {"Content-Type": "application/json"}
    try:
        service_account_json = GSuiteClient.safe_load_non_strict_json(credentials_json)
        client = GSuiteClient(
            service_account_json,
            base_url="https://logging.googleapis.com/",
            verify=verify_certificate,
            proxy=proxy,
            headers=headers,
        )
    except Exception as e:
        raise e
    return client


def get_entries_request(client: GSuiteClient, request_body: dict) -> dict:
    """
    Gets a request body and execute the request.

    Args:
        request_body (dict): The request body.
        client (GSuiteClient): GSuiteClient client.

    Returns:
        The request response.
    """
    client.set_authorized_http(scopes=SCOPES)
    return client.http_request(url_suffix="v2/entries:list", method="POST", body=request_body)


""" COMMAND FUNCTIONS """


def test_module(params: dict) -> str:
    """Tests API connectivity and authentication'

    Returning 'ok' indicates that the integration works like it is supposed to.
    Connection to the service is successful.
    Raises exceptions if something goes wrong.

    :type client: ``Client``
    :param Client: client to use

    :return: 'ok' if test passed, anything else will fail the test.
    :rtype: ``str``
    """
    try:
        prepare_gsuite_client()
    except ValueError as e:
        raise e
    return "ok"


def get_all_results(client: GSuiteClient, limit: int, request_body: dict) -> dict:
    """
    Gets lists log entries when limit is bigger then 1000.

    Args:
        client (GSuiteClient): GSuiteClient client.
        limit (int): limit argument from XSOAR.
        request_body (dict): The request body.

    Returns:
        CommandResults containing the lists log entries.
    """
    request_body.update({"pageSize": MAX_LIMIT})
    demisto.debug(f"Request body: {request_body}")
    response = get_entries_request(client, request_body)
    entries = response.get("entries", [])
    next_page = response.get("nextPageToken")
    next_response = {}
    limit -= MAX_LIMIT
    number_of_results_to_retrieve = limit
    page_size = MAX_LIMIT
    while number_of_results_to_retrieve != 0 and next_page:
        if number_of_results_to_retrieve >= MAX_LIMIT:
            request_body |= {"pageSize": page_size, "pageToken": next_page}
            number_of_results_to_retrieve -= MAX_LIMIT
        else:
            request_body |= {"pageSize": number_of_results_to_retrieve, "pageToken": next_page}
            number_of_results_to_retrieve -= number_of_results_to_retrieve
        demisto.debug(f"Request body: {request_body}")
        next_response = get_entries_request(client, request_body)
        next_page = next_response.get("nextPageToken")
        entries.extend(next_response.get("entries", []))
        next_response.update({"entries": entries})
    return next_response


def create_readable_output(response: list[dict]) -> str:
    """
    Gets readable output.

    Args:
        response (list[dict]): A List of logs entries.

    Returns:
        readable output string.
    """
    hr = [
        {
            "TimeStamp": entry.get("timestamp"),
            "Log Name": entry.get("logName"),
            "Insert ID": entry.get("insertId"),
            "Principal Email": entry.get("protoPayload", {}).get("authenticationInfo", {}).get("principalEmail"),
            "Type": entry.get("resource", {}).get("type", {}),
            "Project ID": entry.get("resource", {}).get("labels", {}).get("project_id"),
            "Cluster Name": entry.get("resource", {}).get("labels", {}).get("cluster_name"),
            "Service Name": entry.get("serviceName"),
        }
        for entry in response
    ]
    return tableToMarkdown(
        "Lists log entries",
        t=hr,
        headers=["TimeStamp", "Log Name", "Insert ID", "Principal Email", "Type", "Project ID", "Cluster Name", "Service Name"],
        removeNull=True,
    )


def log_entries_list_command(client: GSuiteClient, args: dict[str, Any]) -> CommandResults:
    """
    Gets lists log entries. Use this method to retrieve log entries that originated from a
       project/folder/organization/billing account.

    Args:
        client (GSuiteClient): GSuiteClient client.
        args (dict): Command arguments from XSOAR.

    Returns:
        CommandResults containing the lists log entries.
    """
    resource_project_name = argToList(args.get("project_name", []))
    resource_organization_name = argToList(args.get("organization_name", []))
    resource_billing_account_name = argToList(args.get("billing_account_name", []))
    resource_folders_names = argToList(args.get("folder_name", []))

    if not (resource_project_name or resource_organization_name or resource_billing_account_name or resource_folders_names):
        raise DemistoException(
            "At least one of the following resources must be provided: "
            "project_name, organization_name, billing_account_name, or folder_name."
        )
    resources = []
    resources.extend([f"projects/{project_name}" for project_name in resource_project_name if resource_project_name])
    resources.extend(
        [f"organizations/{organization_name}" for organization_name in resource_organization_name if resource_organization_name]
    )
    resources.extend(
        [
            f"billingAccounts/{billing_account_name}"
            for billing_account_name in resource_billing_account_name
            if resource_billing_account_name
        ]
    )
    resources.extend([f"folders/{folders_names}" for folders_names in resource_folders_names if resource_folders_names])

    limit = arg_to_number(args.get("limit"))
    page_size = arg_to_number(args.get("page_size")) or 50
    request_body = {"resourceNames": resources, "filter": args.get("filter"), "orderBy": args.get("order_by")}
    if limit:
        request_body["pageSize"] = limit or 50
    elif args.get("next_token"):
        request_body["pageSize"] = page_size
        request_body["pageToken"] = args.get("next_token")
    else:
        request_body["pageSize"] = limit or 50
    response = {}
    try:
        if limit and limit > 1000:
            # If the pageSize value is negative or exceeds 1000, the request is rejected.
            request_body["pageSize"] = 1000
            response = get_all_results(client, limit, request_body)
        else:
            demisto.debug(f"Request body: {request_body}")
            response = get_entries_request(client, request_body)
    except ValueError as e:
        raise ValueError(e) from e
    return CommandResults(
        outputs_key_field="insertId",
        outputs={
            "GoogleCloudLogging(true)": {"nextPageToken": response.get("nextPageToken")},
            "GoogleCloudLogging.LogsEntry(val.insertId === obj.insertId)": response.get("entries"),
        },
        readable_output=create_readable_output(response.get("entries", []))
        + tableToMarkdown(
            "Next page token",
            t={"nextPageToken": response.get("nextPageToken", "").replace("--", "\--")},
            headers=["nextPageToken"],
            removeNull=True,
        ),
    )


""" MAIN FUNCTION """


def main() -> None:
    """main function, parses params and runs command functions"""
    command = demisto.command()
    try:
        client = prepare_gsuite_client()
        if command == "test-module":
            # This is the call made when pressing the integration Test button.
            result = test_module(demisto.params())
            return_results(result)

        elif command == "gcp-logging-log-entries-list":
            return_results(log_entries_list_command(client, demisto.args()))

        else:
            raise NotImplementedError(f"Command {command} is not implemented")

    # Log exceptions and return errors
    except Exception as e:
        return_error(f"Failed to execute {command} command.\nError:\n{e!s}")


""" ENTRY POINT """


if __name__ in ("__main__", "__builtin__", "builtins"):
    main()