Google Workspace Integration

Version 3.11.1 (View all)
Managed integration release status
Subscription level
What's this?
Basic
Developed by
What's this?
Elastic
Ingestion method(s) API
Minimum Kibana version(s) 9.5.3
9.4.6
8.19.21
Important - Managed integrations on Elastic Cloud Hosted

You can use this integration as an Elastic Managed integration on Elastic Cloud Hosted deployments running this version or later.

The Google Workspace integration collects and parses data from the different Google Workspace audit reports APIs.

These blogs from our Security Labs will help you know more about the Google Workspace and how to it setup:

  1. To understand what Google Workspace is in Part One - Surveying the Land
  2. To set it up, step by step, in Part Two - Setup Threat Detection with Elastic

It is compatible with a subset of applications under the Google Reports API v1. As of today it supports:

Google Workspace Service Description
SAML help View users’ successful and failed sign-ins to SAML applications.
User Accounts help Audit actions carried out by users on their own accounts including password changes, account recovery details and 2-Step Verification enrollment.
Login help Track user sign-in activity to your domain.
Rules help View a record of actions to review your user’s attempts to share sensitive data.
Admin help View administrator activity performed within the Google Admin console.
Drive help Record user activity within Google Drive including content creation in such as Google Docs, as well as content created elsewhere that your users upload to Drive such as PDFs and Microsoft Word files.
Groups help Track changes to groups, group memberships and group messages.
Group Enterprise help The Group Enterprise activity report returns information about various types of Enterprise Groups Audit activity events.
Device help The Mobile activity report returns information about various types of Device Audit activity events.
Token help The Token activity report returns information about various types of OAuth Token Audit activity events.
Access Transparency help The Access Transparency activity report returns information about various types of Access Transparency activity events.
Context Aware Access help The Context Aware Access activity report returns information about various types of Context-Aware Access Audit activity events.
GCP The GCP activity report returns information about various types of Google Cloud Platform activity events.
Chrome The Chrome activity reports return information about Chrome browser and Chrome OS events.
Data Studio Track and audit user interactions and changes made to Looker Studio assets.
Calendar The Calendar activity report returns information about how your account's users manage and modify their Google Calendar events.
Chat The Chat activity report returns information about how your account's users use and manage Spaces.
Vault The Vault activity report returns information about various types of Vault Audit activity events.
Meet The Meet activity report returns information about various aspects of call events.
Keep The Keep activity report returns information about how your account's users manage and modify their notes.
Gmail Reports The Gmail activity report returns information about message delivery, SMTP connections, spam classification and policy rule matches.
Gemini The Gemini in Workspace Apps activity report returns information about generative AI feature usage across Workspace applications.

In order to ingest data from the Google Reports API you must:

  • Have an administrator account. As a security best practice, instead of using a Super Admin account, you can create a custom role with the following permissions. This applies the principle of least privilege.

    Admin console privileges

    • OUs: Read
    • Users: Read
    • Chrome Management > Managed browsers: Read
    • View Reports
    • Alert Center: Full access

    Admin API privileges

    • Reports
    • Groups: Read
    • OU: Read
    • Users: Read

    You also need to enable the following APIs in your Google Cloud project:

    • Admin SDK API
    • Google Workspace Alert Center API
  • Set up a ServiceAccount using the administrator account.

  • Set up access to the Admin SDK API for the ServiceAccount.

  • Enable Domain-Wide Delegation for your ServiceAccount.

The Reports API data streams use the following oauth2 scope:

  • https://www.googleapis.com/auth/admin.reports.audit.readonly

The users data stream needs different scopes. See the Google Workspace Users section.

Once you have downloaded your service account credentials as a JSON file, you are ready to set up your integration.

Click the Advanced option of Google Workspace Audit Reports. The default value of "API Host" is https://www.googleapis.com. The API Host will be used for collecting access_transparency, admin, calendar, chat, chrome, context_aware_access, data_studio, device, drive, gcp, gemini, gmail_reports, groups, group_enterprise, keep, login, meet, rules, saml, token, user accounts and vault logs.

Note

The Delegated Account value in the configuration, is expected to be the email of the administrator account, and not the email of the ServiceAccount.

The access_transparency, admin, context_aware_access, device, drive, gcp, groups, group_enterprise, login, rules, saml, token and user_accounts data streams request activity from the Reports API in time windows. Three settings control those windows:

  • Initial Interval (package level, default 24h): how far back the first collection reaches when a data stream starts with no saved position.
  • Lag Time (per data stream, default 2h): how far behind the current time the end of every window stays. Google publishes activity with a delay that varies by report; see Data retention and lag times. Events newer than now - Lag Time are not requested yet, and are collected on a later interval once they have settled.
  • Chunk Duration (per data stream, under Advanced options, disabled by default): the maximum span of activity requested in one interval. When set, each interval requests a window of at most this duration starting from the last completed position, and the position only moves forward after the whole window has been collected. If collection is interrupted, for example by an agent restart or a policy update, only that one chunk is requested again.

Without Chunk Duration, an interval requests everything from the last completed position up to now - Lag Time in a single window. For a data stream with a large backlog or a high event rate, such as token or drive on a busy domain, that window can take longer to page through than the time between interruptions. Each interruption then restarts the same window from the beginning, no progress is saved, and the data stream appears to stall while it re-sends events it has already sent.

Recommended Chunk Duration values:

Data stream volume Suggested value
High (for example token, drive, login on large domains) 1h
Moderate 6h
Low (for example rules, saml, access_transparency) 24h

A chunk should be small enough to be fully collected between the interruptions you expect, and must not exceed 720h (30 days). Once the data stream has caught up, the window end is limited by now - Lag Time, so Chunk Duration has no effect on steady-state collection. Leaving the setting empty, or setting it to 0, disables chunking. Existing saved positions are reused unchanged when the setting is turned on or off.

Note

The integration provides two Gmail data streams. This section is for gmail, which queries logs exported to Google BigQuery. If you prefer the Google Reports API, enable gmail_reports instead — it shares the service account and admin.reports.audit.readonly scope used by the other Reports API streams and skips the BigQuery configuration steps below.

The integration collects and parses Gmail audit logs data available for reporting in Google Workspace. You must first export Google Workspace logs to Google BigQuery. This involves exporting all activity log events and usage reports to Google BigQuery. Only certain Google Workspace editions support this feature. For more details see About reporting logs and BigQuery. The integration uses the BigQuery API to query logs from BigQuery.

In order to ingest data from the Google BigQuery API, you must:

  1. Enable BigQuery API if not already
  • In the Google Cloud console, navigate to APIs & Services > Library.
  • Search for BigQuery API and select it.
  • Click Enable.
  1. Create a service account:
  • In the Google Cloud console, navigate to APIs & Services > Credentials.
  • Click Create Credentials > Service account.
  • In the setup:
    • Enter a name for the service account.
    • Click Create and Continue.
    • (Optional) Grant project access.
    • Click Continue.
    • (Optional) Grant user access.
    • Click Done.
  1. Generate a JSON Key:
  • From the Credentials page, click on the name of your new service account.
  • Go to the Keys tab.
  • Click Add Key > Create new key.
  • Choose JSON format and click Create.
  • Save the downloaded JSON key securely.
  1. Grant IAM Role to service account:
  • Go to IAM & Admin > IAM in the Cloud Console.
  • Click Grant access.
  • Paste the service account email in the New principals field.
  • Click Select a role, search for and select BigQuery Job User.
  • Click Save.
  1. Set up a BigQuery project for reporting logs
  • Go to IAM & Admin page for your project.
  • Add a project editor for your project.
    • Click Grant access.
    • Enter gapps-reports@system.gserviceaccount.com in the New principals field.
    • In Select a role, select Project, then Editor.
    • Click Save.
  • Add a Google Workspace administrator account as a project editor by following the same steps above.
  • For more details see Set up a BigQuery project for reporting logs
  1. Set up a BigQuery Export configuration:
  • Sign in to your Google Admin console with a super administrator account.
  • Navigate to Reporting > Data Integrations (Requires having the Reports administrator privilege).
    Education administrators go to Menu Reporting > BigQuery export, which opens the Data integrations page.
  • Point to the BigQuery Export card and click Edit.
  • To activate BigQuery logs, check the Enable Google Workspace data export to Google BigQuery box.
  • (Optional) To export sensitive parameters of DLP rules, check the Allow export of sensitive content from DLP rule logs box.
  • Under BigQuery project ID, select the project where you want to store the logs.
    Choose a project for which gapps-reports@system.gserviceaccount.com has an editor role.
  • Under New dataset within project, enter the name of the dataset to use for storing the logs in the project.
    A new dataset will be created with this name in your BigQuery project.
  • (Optional) Check the Restrict the dataset to a specific geographic location box > select the location from the menu.
  • Click Save.
  • For more details see Set up a BigQuery Export configuration.
  1. Grant Dataset Permissions:
  • Go to Google Cloud console and search for BigQuery.
  • Click your Google Cloud project on the left pane.
  • Locate the dataset, click the three-dot menu > Share > Manage Permissions.
  • Click Add principal.
  • Paste the service account email in New principals.
  • Select BigQuery Data Viewer as the role.
  • Click Save.

This integration will make use of the following oauth2 scope:

  • https://www.googleapis.com/auth/bigquery

Once you have downloaded your service account credentials as a JSON file, you are ready to set up your integration for collecting Gmail logs.

Note

For Gmail data stream, the default value of "BigQuery API Host" is https://bigquery.googleapis.com. The BigQuery API Host will be used for collecting gmail logs only.

The Google Workspace Integration collects and parses data received from the Google Workspace Alert Center API using HTTP JSON Input.

  • Alert Data Stream has been tested against Google Workspace Alert Center API (v1).

  • Following Alert types have been supported in the current integration version:

    1. Customer takeout initiated
    2. Malware reclassification
    3. Misconfigured whitelist
    4. Phishing reclassification
    5. Suspicious message reported
    6. User reported phishing
    7. User reported spam spike
    8. Leaked password
    9. Suspicious login
    10. Suspicious login (less secure app)
    11. Suspicious programmatic login
    12. User suspended
    13. User suspended (spam)
    14. User suspended (spam through relay)
    15. User suspended (suspicious activity)
    16. Google Operations
    17. Configuration problem
    18. Government attack warning
    19. Device compromised
    20. Suspicious activity
    21. AppMaker Default Cloud SQL setup
    22. Activity Rule
    23. Data Loss Prevention
    24. Apps outage
    25. Primary admin changed
    26. SSO profile added
    27. SSO profile updated
    28. SSO profile deleted
    29. Super admin password reset
    30. Account suspension warning
    31. Calendar settings changed
    32. Chrome devices auto-update expiration warning
    33. Customer takeout initiated
    34. Drive settings changed
    35. Email settings changed
    36. Gmail potential employee spoofing
    37. Mobile settings changed
    38. New user added
    39. Reporting Rule
    40. Suspended user made active
    41. User deleted
    42. User granted Admin privilege
    43. User suspended (spam)
    44. User's Admin privileges revoked
    45. Users password changed
    46. Google Voice configuration problem detected

In order to ingest data from the Google Alert Center API, you must:

This integration will make use of the following oauth2 scope:

  • https://www.googleapis.com/auth/apps.alerts

Once Service Account credentials are downloaded as a JSON file, then the integration can be setup to collect data.

Note

The Delegated Account value in the configuration, is expected to be the email of the administrator account, and not the email of the ServiceAccount.

Note

The default value of the "Page Size" is set to 1000. This option is available under 'Alert' Advance options. Set the parameter "Page Size" according to the requirement. For Alert Data Stream, The default value of "Alert Center API Host" is https://alertcenter.googleapis.com. The Alert Center API Host will be used for collecting alert logs only.

The users data stream collects an inventory of the user accounts in your Google Workspace tenant from the Admin SDK Directory API. Each collection cycle produces one document per user, describing what exists: identity, 2-Step Verification, admin, suspension and archive state, organizational unit, manager and other directory attributes. Documents have event.kind: asset.

This is different from the user_accounts data stream, which contains audit events about actions users took on their own accounts (what happened). Changes to users made by administrators are in the admin data stream.

The data stream is turned off by default.

The Directory API has no filter for changed users, so each cycle reads the whole directory again. The default interval is 24h. A tenant of 50,000 users needs about 100 requests per cycle.

Two optional settings add more information to each user document. Both are off by default:

  • Collect group membership adds the groups each user is a direct member of (user.group.id, user.group.name and google_workspace.users.groups). Membership inherited through nested groups is not included.
  • Collect admin roles and privileges adds the admin roles assigned to each user (user.roles), their privileges (user.entity.attributes.permissions) and the scope each role applies to (user.entity.relationships.administers.entity.id): an organizational unit ID, or the customer ID for roles that apply to the whole account, such as super administrator. The full assignments are in google_workspace.users.role_assignments. Only roles assigned directly to the user are included. Roles a user gets through a group are not, because the Directory API only returns those for a single user at a time.

user.roles always includes super_admin or delegated_admin when Google flags the user as one, even when the roles option is off.

When an option is on and its requests succeed, every user document carries google_workspace.users.groups or google_workspace.users.role_assignments, as an empty list ([]) if the user has no direct membership or assignment. When the option is off, or its requests failed in that cycle, the field is left out of the document, and a failure also produces an error document. Elasticsearch does not index empty lists, so the difference is visible in the document source but cannot be used in a query.

The users data stream uses the same service account and Delegated Account as the Reports API data streams. It needs these extra scopes and privileges:

Setting OAuth scope to add Privilege the delegated account needs
Always https://www.googleapis.com/auth/admin.directory.user.readonly Users → Read and Organizational Units → Read, on all organizational units
Collect group membership https://www.googleapis.com/auth/admin.directory.group.readonly Groups → Read. If you use a Groups Reader or Groups Editor role assignment instead, it must cover all groups.
Collect admin roles and privileges https://www.googleapis.com/auth/admin.directory.rolemanagement.readonly The delegated account must be a super administrator. Google does not offer a privilege that grants read access to admin roles, so other accounts get HTTP 403 on these requests.

Google has merged the former "Admin console privileges" and "Admin API privileges" into one Admin privileges section, so one Users → Read privilege covers both. See Administrator privilege definitions.

To add the scopes to the service account's domain-wide delegation (a super administrator must do this):

  1. In the Admin console go to Security → Access and data control → API controls → Manage Domain Wide Delegation.
  2. Find the service account's client ID and click Edit. Do not use Add new for a client ID that is already listed, and do not delete and re-add it: deleting it stops every application that uses it straight away.
  3. Add the scopes you need, separated by commas. Keep https://www.googleapis.com/auth/admin.reports.audit.readonly and any other scopes already listed.
  4. Click Authorize, then open View details and check that every scope is listed.

Use the exact scope strings in the table. Google does not accept a broader scope in place of the one the integration requests. Scope changes can take up to 24 hours to take effect. If your organization uses multi-party approval, another super administrator must approve the change first.

Note

The integration requests all the scopes the data stream needs in one token request. If any of them is not delegated yet, Google issues no token and the whole users data stream stops, including the basic user inventory. Turn on an optional setting only after its scope is delegated and in effect. The other data streams of the integration are not affected.

The data stream calls https://admin.googleapis.com. You can change this with the Directory API Host setting under the data stream's advanced options. If your network uses an egress allowlist, add admin.googleapis.com.

  • user.entity.lifecycle.last_activity comes from the user's last sign-in time. Google says this value "might be inaccurate" when listing all users, so treat it as approximate. Users who have never signed in have no value. google_workspace.users.agreed_to_terms: false also shows that a user has never completed a first sign-in.
  • Deleted users are not included.
  • The data stream stops with oauth2: cannot fetch token and the input health is Degraded. Google rejected the token request. The error usually contains unauthorized_client, access_denied, invalid_grant, invalid_scope or admin_policy_enforced. Check that every scope for the settings you enabled is delegated exactly as in the table above, and that enough time has passed for the change to take effect.
  • An error document reports HTTP 403 on a roles, role assignments, groups or members request. The delegated account lacks the privilege for that option. Users are still collected, without that information. For the roles option, the delegated account must be a super administrator.
  • Fewer users than expected. The delegated account's role is probably limited to some organizational units. Grant Users → Read on all organizational units.

Agentless integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to Agentless integrations and the Agentless integrations FAQ.

Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features.

This is a list of Google Workspace Reports fields that are mapped to ECS that are common to all data sets.

Google Workspace Reports ECS Fields
items[].id.time @timestamp
items[].id.uniqueQualifier event.id
items[].id.applicationName event.provider
items[].events[].name event.action
items[].customerId organization.id
items[].ipAddress source.ip, related.ip, source.as.*, source.geo.*
items[].actor.email source.user.email, source.user.name, source.user.domain
items[].actor.profileId source.user.id

This is the saml dataset.

This is the users dataset. It contains an inventory of user accounts, not events. See the Google Workspace Users section.

This is the user_accounts dataset.

This is the login dataset.

This is the rules dataset.

This is the admin dataset.

This is the drive dataset.

This is the groups dataset.

This is the alert dataset.

This is the device dataset.

This is the group_enterprise dataset.

This is the token dataset.

This is the access_transparency dataset.

This is the context_aware_access dataset.

This is the gcp dataset.

This is the chrome dataset.

This is the data_studio dataset.

This is the calendar dataset.

This is the chat dataset.

This is the vault dataset.

This is the meet dataset.

This is the keep dataset.

This is the gmail_reports dataset. It is collected from the Google Reports API and emits one document per event, so a single activity record containing several events produces several documents.

This is the gemini dataset.

This is the gmail dataset.

This integration includes one or more Kibana dashboards that visualizes the data collected by the integration. The screenshots below illustrate how the ingested data is displayed.