Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EDU-709 - audit logging pages for GCP #3047

Merged
merged 23 commits into from
Sep 13, 2024
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
36af67d
splitting export page
MasonEgger Aug 27, 2024
6491eed
adding gcp export
MasonEgger Aug 28, 2024
d6d1f4d
first draft of audit logging with content from engineering
MasonEgger Aug 28, 2024
e58c1fa
fixing broken code sample
MasonEgger Aug 28, 2024
4c95877
removing line about example consuming audit log as content moved
MasonEgger Sep 3, 2024
0927fff
fixing broken anchor link
MasonEgger Sep 3, 2024
8d76e91
removing export as its in another commit
MasonEgger Sep 10, 2024
dfae828
adding export back
MasonEgger Sep 10, 2024
c26aec2
Update docs/production-deployment/cloud/audit-logging-aws.mdx
MasonEgger Sep 12, 2024
6dab0a3
Update docs/production-deployment/cloud/audit-logging-aws.mdx
MasonEgger Sep 12, 2024
68fe68b
Update docs/production-deployment/cloud/audit-logging.mdx
MasonEgger Sep 12, 2024
ae022ca
Update docs/production-deployment/cloud/audit-logging-aws.mdx
MasonEgger Sep 12, 2024
e52062e
moving examples back to main page
MasonEgger Sep 12, 2024
32d421b
removing gcp pubsub
MasonEgger Sep 12, 2024
f9fa253
Update docs/production-deployment/cloud/audit-logging.mdx
MasonEgger Sep 12, 2024
00534a5
updates per Rafaels feedback
MasonEgger Sep 12, 2024
10b917a
adding things that went missing
MasonEgger Sep 12, 2024
627ede4
adding correct example of audit log
MasonEgger Sep 12, 2024
74f4c55
Merge branch 'main' into gcp-edits-cloud-audit-logging
MasonEgger Sep 12, 2024
0db9129
update from Jwahir
MasonEgger Sep 12, 2024
056a043
removing GCP reference
MasonEgger Sep 12, 2024
3025d88
Merge branch 'main' into gcp-edits-cloud-audit-logging
MasonEgger Sep 13, 2024
99e8a06
final tweaks
MasonEgger Sep 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions docs/production-deployment/cloud/audit-logging-aws.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
---
id: audit-logging-aws
title: Audit Logging - AWS Kinesis
sidebar_label: Amazon Kinesis
description: Audit Logging in Temporal Cloud provides forensic information, integrating with AWS Kinesis for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes.
slug: /cloud/audit-logging-aws
toc_max_heading_level: 4
keywords:
- audit logging
- explanation
- how-to
- operations
- temporal cloud
- term
- troubleshooting
- aws
- kinesis
tags:
- audit-logging
- explanation
- how-to
- operations
- temporal-cloud
- term
- troubleshooting
- aws
- kinesis
---

## How to configure Audit Logging using AWS Kinesis {#configure-audit-logging}
MasonEgger marked this conversation as resolved.
Show resolved Hide resolved

To set up Audit Logging, you must have an Amazon Web Services (AWS) account and set up Kinesis Data Streams.

1. If you don't have an AWS account, follow the instructions from AWS in [Create and activate an AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/).
2. To set up Kinesis Data Streams, open the [AWS Management Console](https://aws.amazon.com/console/), search for Kinesis, and start the setup process.

You can use [this AWS CloudFormation template](https://temporal-auditlogs-config.s3.us-west-2.amazonaws.com/cloudformation/iam-role-for-temporal-audit-logs.yaml) to create an IAM role with access to a Kinesis stream you have in your account.

Be aware that Kinesis has a rate limit of 1,000 messages per second and quotas for both the number of records written and the size of the records.
For more information, see [Why is my Kinesis data stream throttling?](https://aws.amazon.com/premiumsupport/knowledge-center/kinesis-data-stream-throttling/)

### Create an Audit Log sink

1. In the Temporal Cloud UI, select **Settings**.
1. On the **Settings** page, select **Integrations**.
1. In the **Audit Logging** card, select **Configure Audit Logs**.
1. On the **Audit Logging** page, choose your **Access method** (either **Auto** or **Manual**).
- **Auto:** Configure the AWS CloudFormation stack in your AWS account from the Cloud UI.
- **Manual:** Use a generated AWS CloudFormation template to set up Kinesis manually.
1. In **Kinesis ARN**, paste the Kinesis ARN from your AWS account.
1. In **Role name**, provide a name for a new IAM Role.
1. In **Select an AWS region**, select the appropriate region for your Kinesis stream.

If you chose the **Auto** access method, continue with the following steps:

1. Select **Save and launch stack**.
1. In **Stack name** in the AWS CloudFormation console, specify a name for the stack.
1. In the lower-right corner of the page, select **Create stack**.

If you chose the **Manual** access method, continue with the following steps:

1. Select **Save and download template**.
1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/).
1. Select **Create Stack**.
1. On the **Create stack** page, select **Template is ready** and **Update a template file**.
1. Select **Choose file** and specify the template you generated in step 1.
1. Select **Next** on this page and on the next two pages.
1. On the **Review** page, select **Create stack**.


## How to consume an Audit Log {#consume-an-audit-log}

After you create an Audit Log sink, wait for the logs to flow into the Kinesis stream.
You should see the first logs 2–10 minutes after you configure the sink.
Subsequent logs arrive every 2 minutes if any actions occurred during that 2-minute window.

:::note

You must configure and implement your own consumer of the Kinesis stream.
For an example, see [Example of consuming an Audit Log](#example-of-consuming-an-audit-log).

:::

### Example of consuming an Audit Log

The following Go code is an example of consuming Audit Logs from a Kinesis stream and delivering them to an S3 bucket.

```go
func main() {
fmt.Println("print audit log from S3")
cfg, err := config.LoadDefaultConfig(context.TODO(),
config.WithSharedConfigProfile("your_profile"),
)
if err != nil {
fmt.Println(err)
}
s3Client := s3.NewFromConfig(cfg)
response, err := s3Client.GetObject(
context.Background(),
&s3.GetObjectInput{
Bucket: aws.String("your_bucket_name"),
Key: aws.String("your_s3_file_path")})
if err != nil {
fmt.Println(err)
}
defer response.Body.Close()

content, err := io.ReadAll(response.Body)

fmt.Println(string(content))
}
```

The preceding code also prints the logs in the terminal.
The following is a sample result.

```json
{
"emit_time": "2023-11-14T07:56:55Z",
"level": "LOG_LEVEL_INFO",
"caller_ip_address":"10.1.2.3, 10.4.5.6",
"user_email": "[email protected]",
MasonEgger marked this conversation as resolved.
Show resolved Hide resolved
"operation": "DeleteUser",
"details": {
"target_users": ["d7dca96f-adcc-417d-aafc-e8f5d2ba9fe1"],
"search_attribute_update": {}
},
"status": "OK",
"category": "LOG_CATEGORY_ADMIN",
"log_id": "0mc69c0323b871293ce231dd1c7fb639",
"request_id": "445297d3-43a7-4793-8a04-1b1dd1999640",
"principal": {
"id": "988cb80b-d6be-4bb5-9c87-d09f93f58ed3",
"type": "user",
"name": "[email protected]"
}
}
```
150 changes: 34 additions & 116 deletions docs/production-deployment/cloud/audit-logging.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
id: audit-logging
title: Audit Logging - Temporal Cloud feature guide
sidebar_label: Audit Logging
description: Audit Logging in Temporal Cloud provides forensic information, integrating with Amazon Kinesis for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes.
description: Audit Logging in Temporal Cloud provides forensic information, integrating with a data streaming service for secure data handling and supporting key Admin and API Key operations. This streamlines audit and compliance processes.
slug: /cloud/audit-logging
toc_max_heading_level: 4
keywords:
Expand Down Expand Up @@ -30,8 +30,7 @@ These answers can help you evaluate the security of your organization, and they

## Which integrations are supported by Audit Logging? {#supported-integrations}

Audit Logging supports the [Amazon Kinesis](https://docs.aws.amazon.com/kinesis/) streaming-data platform.
By using [Amazon Kinesis Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html), you can route Temporal Audit Logs in [Amazon Simple Storage Service](https://docs.aws.amazon.com/s3/) (S3).
Audit Logging supports the [Amazon Kinesis](https://docs.aws.amazon.com/kinesis/) and the [GCP Pub/Sub](https://cloud.google.com/pubsub?hl=en) streaming-data platform.
MasonEgger marked this conversation as resolved.
Show resolved Hide resolved
We plan to release additional integrations.

## Which events are supported by Audit Logging? {#supported-events}
Expand Down Expand Up @@ -76,14 +75,14 @@ The following list specifies both the supported events and the Temporal APIs tha

### Audit Log format

The log sent to the Kinesis stream is JSON in the following format:
The log sent to the stream is JSON in the following format:
MasonEgger marked this conversation as resolved.
Show resolved Hide resolved

```json
{
"emit_time": // Time the operation was recorded
"level": // Level of the log entry, such as info, warning, or error
"user_email": // Email address of the user who initiated the operation
"caller_ip_address": // Customer IP address or server name
"caller_ip_address": // Customer IP address in the X-Forwarded-For format
"operation": // Operation that was performed
"details": // Details of the operation
"status": // Status, such as OK or ERROR
Expand All @@ -95,132 +94,51 @@ The log sent to the Kinesis stream is JSON in the following format:
}
```

MasonEgger marked this conversation as resolved.
Show resolved Hide resolved
## How to configure Audit Logging {#configure-audit-logging}

To set up Audit Logging, you must have an Amazon Web Services (AWS) account and set up Kinesis Data Streams.

1. If you don't have an AWS account, follow the instructions from AWS in [Create and activate an AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/).
2. To set up Kinesis Data Streams, open the [AWS Management Console](https://aws.amazon.com/console/), search for Kinesis, and start the setup process.

You can use [this AWS CloudFormation template](https://temporal-auditlogs-config.s3.us-west-2.amazonaws.com/cloudformation/iam-role-for-temporal-audit-logs.yaml) to create an IAM role with access to a Kinesis stream you have in your account.

Be aware that Kinesis has a rate limit of 1,000 messages per second and quotas for both the number of records written and the size of the records.
For more information, see [Why is my Kinesis data stream throttling?](https://aws.amazon.com/premiumsupport/knowledge-center/kinesis-data-stream-throttling/)

### Create an Audit Log sink

1. In the Temporal Cloud UI, select **Settings**.
1. On the **Settings** page, select **Integrations**.
1. In the **Audit Logging** card, select **Configure Audit Logs**.
1. On the **Audit Logging** page, choose your **Access method** (either **Auto** or **Manual**).
- **Auto:** Configure the AWS CloudFormation stack in your AWS account from the Cloud UI.
- **Manual:** Use a generated AWS CloudFormation template to set up Kinesis manually.
1. In **Kinesis ARN**, paste the Kinesis ARN from your AWS account.
1. In **Role name**, provide a name for a new IAM Role.
1. In **Select an AWS region**, select the appropriate region for your Kinesis stream.

If you chose the **Auto** access method, continue with the following steps:

1. Select **Save and launch stack**.
1. In **Stack name** in the AWS CloudFormation console, specify a name for the stack.
1. In the lower-right corner of the page, select **Create stack**.

If you chose the **Manual** access method, continue with the following steps:

1. Select **Save and download template**.
1. Open the [AWS CloudFormation console](https://console.aws.amazon.com/cloudformation/).
1. Select **Create Stack**.
1. On the **Create stack** page, select **Template is ready** and **Update a template file**.
1. Select **Choose file** and specify the template you generated in step 1.
1. Select **Next** on this page and on the next two pages.
1. On the **Review** page, select **Create stack**.

## How to consume an Audit Log {#consume-an-audit-log}

After you create an Audit Log sink, wait for the logs to flow into the Kinesis stream.
You should see the first logs 2–10 minutes after you configure the sink.
Subsequent logs arrive every 2 minutes if any actions occurred during that 2-minute window.

:::note

You must configure and implement your own consumer of the Kinesis stream.
For an example, see [Example of consuming an Audit Log](#example-of-consuming-an-audit-log).
The [`X-Forwarded-For`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For) format is a comma-separated list of IP addresses which should be evaluated from the last to the first, until meeting the first untrusted IP address of the list. This allows for instance to consider proxies in the path.

Temporal provides the caller IP address in that format to allow customers to identify a caller IP address even if one (or more proxies) are in the network path to reach Temporal Cloud.

:::

### Example of an Audit Log

The following example shows the contents of an Audit Log.

```json
{"emit_time":"2023-10-24T08:19:41Z","level":"LOG_LEVEL_INFO","user_email":"[email protected]","operation":"UpdateAccount","details":{"client_ca_fingerprints":["5bb99d14fa602f7d39b7d048674a2251"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb634","principal":{"id":"988cb80b-d6be-4bb5-9c87-d09f93f58ed3","type":"user","name":"[email protected]"}}
{"emit_time":"2023-10-24T08:19:41Z","level":"LOG_LEVEL_INFO","caller_ip_address":"10.1.2.3, 10.4.5.6","user_email":"[email protected]","operation":"UpdateAccount","details":{"client_ca_fingerprints":["5bb99d14fa602f7d39b7d048674a2251"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb634","principal":{"id":"988cb80b-d6be-4bb5-9c87-d09f93f58ed3","type":"user","name":"[email protected]"}}
**********
{"emit_time":"2023-10-25T21:16:42Z","level":"LOG_LEVEL_INFO","user_email":"[email protected]","operation":"DeleteUser","details":{"target_users":["0b741c47-e093-47d1-9b74-f2359129f78f"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb635","request_id":"445297d3-43a7-4793-8a04-1b1dd1999641","principal":{"id":"b160473e-e40d-4a81-90d1-f4218269e6e4","type":"user","name":"[email protected]"}}
{"emit_time":"2023-10-25T21:16:42Z","level":"LOG_LEVEL_INFO","caller_ip_address":"10.7.8.9","user_email":"[email protected]","operation":"DeleteUser","details":{"target_users":["0b741c47-e093-47d1-9b74-f2359129f78f"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb635","request_id":"445297d3-43a7-4793-8a04-1b1dd1999641","principal":{"id":"b160473e-e40d-4a81-90d1-f4218269e6e4","type":"user","name":"[email protected]"}}
**********
{"emit_time":"2023-11-03T19:31:45Z","level":"LOG_LEVEL_INFO","user_email":"[email protected]","operation":"InviteUsers","details":{"target_users":["[email protected]"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb636","principal":{"id":"35fdc757-9637-446b-b386-12ed475511ad","type":"user","name":"[email protected]"}}
{"emit_time":"2023-11-03T19:31:45Z","level":"LOG_LEVEL_INFO","caller_ip_address":"10.1.2.3, 10.10.11.12","user_email":"[email protected]","operation":"InviteUsers","details":{"target_users":["[email protected]"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb636","principal":{"id":"35fdc757-9637-446b-b386-12ed475511ad","type":"user","name":"[email protected]"}}
**********
{"emit_time":"2023-11-08T08:06:40Z","level":"LOG_LEVEL_INFO","user_email":"[email protected]","operation":"UpdateUser","details":{"target_users":["[email protected]"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb637","request_id":"445297d3-43a7-4793-8a04-1b1dd1999640","principal":{"id":"988cb80b-d6be-4bb5-9c87-d09f93f58ed3","type":"user","name":"[email protected]"}}
{"emit_time":"2023-11-08T08:06:40Z","level":"LOG_LEVEL_INFO","caller_ip_address":"10.1.2.3, 10.4.5.6","user_email":"[email protected]","operation":"UpdateUser","details":{"target_users":["[email protected]"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb637","request_id":"445297d3-43a7-4793-8a04-1b1dd1999640","principal":{"id":"988cb80b-d6be-4bb5-9c87-d09f93f58ed3","type":"user","name":"[email protected]"}}
**********
{"emit_time":"2023-11-08T08:14:09Z","level":"LOG_LEVEL_INFO","user_email":"[email protected]","operation":"UpdateNamespace","details":{"namespace":"audit-log-test.example-dev","client_ca_fingerprints":["f186d0bd971ff7d52dc6cc9d9b6f7644"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb638","principal":{"id":"988cb80b-d6be-4bb5-9c87-d09f93f58ed3","type":"user","name":"[email protected]"}}
{"emit_time":"2023-11-08T08:14:09Z","level":"LOG_LEVEL_INFO","caller_ip_address":"10.1.2.3, 10.4.5.6","user_email":"[email protected]","operation":"UpdateNamespace","details":{"namespace":"audit-log-test.example-dev","client_ca_fingerprints":["f186d0bd971ff7d52dc6cc9d9b6f7644"],"search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb638","principal":{"id":"988cb80b-d6be-4bb5-9c87-d09f93f58ed3","type":"user","name":"[email protected]"}}
**********
{"emit_time":"2023-11-08T09:20:22Z","level":"LOG_LEVEL_INFO","user_email":"[email protected]","operation":"UpdateUserNamespacePermissions","details":{"namespace":"audit-log-test.example-dev","search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb639","principal":{"id":"988cb80b-d6be-4bb5-9c87-d09f93f58ed3","type":"user","name":"[email protected]"}}
{"emit_time":"2023-11-08T09:20:22Z","level":"LOG_LEVEL_INFO","caller_ip_address":"10.1.2.3, 10.4.5.6","user_email":"[email protected]","operation":"UpdateUserNamespacePermissions","details":{"namespace":"audit-log-test.example-dev","search_attribute_update":{}},"status":"OK","category":"LOG_CATEGORY_ADMIN","log_id":"0mc69c0323b871293ce231dd1c7fb639","principal":{"id":"988cb80b-d6be-4bb5-9c87-d09f93f58ed3","type":"user","name":"[email protected]"}}
**********
```

### Example of consuming an Audit Log

The following Go code is an example of consuming Audit Logs from a Kinesis stream and delivering them to an S3 bucket.

```go
func main() {
fmt.Println("print audit log from S3")
cfg, err := config.LoadDefaultConfig(context.TODO(),
config.WithSharedConfigProfile("your_profile"),
)
if err != nil {
fmt.Println(err)
}
s3Client := s3.NewFromConfig(cfg)
response, err := s3Client.GetObject(
context.Background(),
&s3.GetObjectInput{
Bucket: aws.String("your_bucket_name"),
Key: aws.String("your_s3_file_path")})
if err != nil {
fmt.Println(err)
}
defer response.Body.Close()

content, err := io.ReadAll(response.Body)

fmt.Println(string(content))
}
```
## How to configure Audit Logging {#configure-audit-logging}

The preceding code also prints the logs in the terminal.
The following is a sample result.
Audit logging can be configured in both AWS Kinesis

* [AWS Kinesis Instructions](/cloud/audit-logging-aws)

## Consume an Audit Log {#consume-an-audit-log}

**How to consume an Audit Log**

After you create an Audit Log sink, wait for the logs to flow into the stream.
You should see the first logs 2–10 minutes after you configure the sink.
Subsequent logs arrive every 2 minutes if any actions occurred during that 2-minute window.

:::note

You must configure and implement your own consumer of the stream.

:::

```json
{
"emit_time": "2023-11-14T07:56:55Z",
"level": "LOG_LEVEL_INFO",
"user_email": "[email protected]",
"operation": "DeleteUser",
"details": {
"target_users": ["d7dca96f-adcc-417d-aafc-e8f5d2ba9fe1"],
"search_attribute_update": {}
},
"status": "OK",
"category": "LOG_CATEGORY_ADMIN",
"log_id": "0mc69c0323b871293ce231dd1c7fb639",
"request_id": "445297d3-43a7-4793-8a04-1b1dd1999640",
"principal": {
"id": "988cb80b-d6be-4bb5-9c87-d09f93f58ed3",
"type": "user",
"name": "[email protected]"
}
}
```

## How to troubleshoot Audit Logging {#troubleshoot-audit-logging}

Expand All @@ -229,9 +147,9 @@ The Audit Logging page of the Temporal Cloud UI provides the current status of a
- If an error is detected, a summary of the error appears below the page title.
- If the Audit Log sink is functioning normally, an **On** badge appears next to the page heading.

After an Admin Operation is performed, users can see Audit Log messages flow through Kinesis.
After an Admin Operation is performed, users can see Audit Log messages flow through the stream.

Upon successful configuration of the Audit Log sink and set up of Kinesis, you will receive events within the hour of setup.
Upon successful configuration of the Audit Log sink and set up of a stream, you will receive events within the hour of setup.
Temporal is able to retain Audit Log information for up to 30 days.
To retrieve logs up to the past 30 days, you will need to file a request.

Expand All @@ -247,4 +165,4 @@ When you no longer need Audit Logging, you can delete the Audit Log sink.
1. In the **Audit Logging** card, select **Configure Audit Logs**.
1. At the bottom of the **Audit Logging** page, choose **Delete**.

After you confirm the deletion, the Audit Log Sink is removed from your account and logs stop flowing to your Kinesis stream.
After you confirm the deletion, the Audit Log Sink is removed from your account and logs stop flowing to your stream.
Loading