read SA data & write to cloud logging #1145
Conversation
| cd sdk-codegen/examples/python/extract-logs-write-to-cloud-logging | ||
| ``` | ||
|
|
||
| - Setup Python Virtual environment |
There was a problem hiding this comment.
Is python virtual environment necessary for development?
Seems like Python version 3.8.2 is what's necessary.
There was a problem hiding this comment.
A virtual environment is not required. However, it is generally encouraged to install dependencies in a virtual environment
There was a problem hiding this comment.
This sample script is not the place for this. We should only list bare minimum requirements to run the sample script.
| - Install dependencies | ||
| ``` | ||
| pip install looker-sdk | ||
| pip install --upgrade google-cloud-logging |
There was a problem hiding this comment.
Does the --upgrade assume the developer already has google-cloud-logging installed?
There was a problem hiding this comment.
This is a mistake. --upgrade option is not required.
| ## Requirements | ||
| - Looker Instance in which you have Admin or `see_system_activity` permission | ||
| - Google Cloud Project with Cloud Logging API enabled | ||
| - [pyenv](https://github.com/pyenv/pyenv#installation) installed |
There was a problem hiding this comment.
Should add the python version and gcloud to this
|
|
||
| | GCP Audit Log Field | Looker System Actvity Field | | ||
| | ----------- | ----------- | | ||
| | [logName](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#:~:text=Fields-,logName,-string) | `looker_system_activity_logs` | |
There was a problem hiding this comment.
Do the GCP services that consume Audit Logs have expectations on the values in the Audit Log?
This example would set precedent and probably be defacto standard for Audit logs coming from Looker. I assume compliance/auditing expects certain values for Audit Logs.
There was a problem hiding this comment.
Yes GCP Audit logs have expected values.
However, some of these values do not apply or make sense for a Looker resource. For example, GCP resource always falls under a project/resource but a Looker resource doesn't. A project in Looker means something different. The mapping is on a best effort basis.
Technically the output of this script is not a GCP Audit log (when you filter for GCP Audit log, these logs won't appear). I'm only using the Audit log format in this example for similarity when querying these logs.
There was a problem hiding this comment.
Good to know. So this is only for querying standard GCP Audit Logs with Looker System activity Logs included.
However, my main worry is compliance/auditing issues. Like i previously asked, is there any GCP services that consume audit logs that may require specific values? (And have you consulted them on this)
There was a problem hiding this comment.
Yes, all GCP resources require certain values in audit logs(E.g Project, organization, IAM details). I already mentioned that some of these values don't apply to Looker because technically Looker is not a GCP resource (yet)
No, I did not consult with the Audit Logs team. This repo is only an example showing a way you can programmatically export System Activity logs today, until the product team integrates Looker as a GCP service. It's my understanding that only the product team can get this type of compliance approval you're referring to.
System Activity logs is NOT a GCP Audit Log. And I'm sure the Audit team will echo that if I consult with them. This is not an official solution, but an example to demonstrate one of the workarounds you could do until the product team provides an official solution to this.
I'm only mapping the SA logs to look like GCP Audit logs since most customers are familiar with the format and I mentioned in the readme that this is a best effort basis
|
|
||
| ## GCP Audit Log Fields to Looker System Activity Mapping | ||
|
|
||
| | GCP Audit Log Field | Looker System Actvity Field | |
There was a problem hiding this comment.
Can the second column be Looker System Activity Field OR Value? So that the other hardcoded values can be documented here too?
| "event.id", | ||
| "event.name", | ||
| "event.category", | ||
| "event.sudo_user_id", | ||
| "event.created_time", | ||
| "user.email", | ||
| "user.name", | ||
| "permission_set.permissions", | ||
| "permission_set.name", | ||
| "permission_set.id", | ||
| "model_set.models", | ||
| "model_set.name", | ||
| "event_attribute.name", | ||
| "event_attribute.value", | ||
| "event_attribute.id", | ||
| "group.id", | ||
| "group.name", | ||
| "group.external_group_id", | ||
| "model_set.id", | ||
| "user.dev_branch_name" |
There was a problem hiding this comment.
Small nitpick but can we alphabetize?
| @@ -0,0 +1,66 @@ | |||
| ## Overview | |||
|
|
|||
| A Python script that extracts Looker system/audit logs from [System Activity](https://docs.looker.com/admin-options/system-activity) and exports the Logs to Cloud Logging. This example tries to format the output logs like a [GCP Audit Log](https://cloud.google.com/logging/docs/audit/understanding-audit-logs) as best as possible. See [mapping](#gcp-audit-log-fields-to-looker-system-activity-mapping) for comparison between Looker System Activity Fields and GCP Audit Log Fields | |||
There was a problem hiding this comment.
First lines should state what it does, not be saved for the note.
A Python script that extracts [System Activity](https://docs.looker.com/admin-options/system-activity) data from the last 10 minutes, formats the data as Audit Logs, and exports the logs to Cloud Logging. The data formatting/mapping is best effort. See [data mapping](#gcp-audit-log-fields-to-looker-system-activity-mapping) below.
Then can keep note as:
**_NOTE:_** You can schedule this script to run every 10 minutes using a cron job or equivalent to continually create and export logs.
| ], | ||
| filters={"event.created_time": "10 minutes"}, | ||
| sorts=["event.created_time desc"], | ||
| filter_config={"event.created_time": [{ |
There was a problem hiding this comment.
This should be set to null unless i'm misunderstanding behavior or api docs.
https://developers.looker.com/api/explorer/4.0/types/Query/WriteQuery
The filter_config represents the state of the filter UI on the explore page for a given query. When running a query via the Looker UI, this parameter takes precedence over "filters". When creating a query or modifying an existing query, "filter_config" should be set to null. Setting it to any other value could cause unexpected filtering behavior. The format should be considered opaque.
There was a problem hiding this comment.
true. having both filters & filter_config is redundant
| cd extract-looker-logs/ | ||
| ``` | ||
|
|
||
| - Setup python virtual environment |
| "event.id", | ||
| "event.name", | ||
| "event.category", | ||
| "event.sudo_user_id", | ||
| "event.created_time", | ||
| "event_attribute.name", | ||
| "event_attribute.value", | ||
| "event_attribute.id", | ||
| "group.id", | ||
| "group.name", | ||
| "group.external_group_id", | ||
| "model_set.id", | ||
| "model_set.models", | ||
| "model_set.name", | ||
| "permission_set.permissions", | ||
| "permission_set.name", | ||
| "permission_set.id", | ||
| "user.email", | ||
| "user.name", | ||
| "user.dev_branch_name" |
There was a problem hiding this comment.
Missed some alphabetical ordering here.
|
|
||
| - Clone the repo | ||
| ``` | ||
| git clone https://github.com/itodotimothy6/extract-looker-logs.git |
There was a problem hiding this comment.
Corrected to the right repo
An example python script that shows how to use Looker API to extract System Activity logs and use the Logging API to write to Cloud Logging
Fixes #1144 1144 🦕