Skip to content

Commit

Permalink
edit github webhook
Browse files Browse the repository at this point in the history
  • Loading branch information
WanYixian committed Nov 22, 2024
1 parent 8aae30c commit de7dc5a
Showing 1 changed file with 60 additions and 62 deletions.
122 changes: 60 additions & 62 deletions integrations/sources/github-webhook.mdx
Original file line number Diff line number Diff line change
@@ -1,52 +1,51 @@
---
title: "Ingest data from GitHub Webhook"
description: "Describes how to use ingest data from GitHub Webhook to RisingWave."
description: "Ingest GitHub events directly into your RisingWave database for real-time processing and analytics."
sidebarTitle: GitHub Webhook
---

GitHub Webhooks allow you to build or set up integrations that subscribe to certain events on GitHub.com. When one of those events is triggered, GitHub sends an HTTP POST payload to the webhook's configured URL. Webhooks can be used to update an external issue tracker, trigger CI builds, update a backup mirror, or even deploy to your production server.
GitHub Webhook allows you to build or set up integrations that subscribe to certain events on `GitHub.com`. When one of those events is triggered, GitHub sends an HTTP POST payload to the Webhook's configured URL. Webhooks can be used to update an external issue tracker, trigger CI builds, update a backup mirror, or even deploy to your production server.

In this guide, we'll walk through the steps to set up RisingWave as a destination for GitHub Webhooks. This enables you to ingest GitHub events directly into your RisingWave database for real-time processing and analytics.
This guide will walk through the steps to set up RisingWave as a destination for GitHub Webhooks.

## Steps to Ingest Data from GitHub via Webhook
## 1. Create a secret in RisingWave

### 1. Create a Secret in RisingWave

First, create a secret in RisingWave to securely store a secret string. This secret will be used to validate incoming webhook requests from GitHub.
First, create a secret in RisingWave to securely store a secret string. This secret will be used to validate incoming Webhook requests from GitHub.

```sql
CREATE SECRET test_secret WITH (backend = 'meta') AS 'TEST_WEBHOOK';
```

Explanation:
- `test_secret`: The name of the secret.
- `'TEST_WEBHOOK'`: The secret string used for signing and verifying webhook payloads. Replace this with a secure, random string.
| Parameter or clause | Description |
| :--------- | :-----------|
|`test_secret`| The name of the secret.|
| `TEST_WEBHOOK`| The secret string used for signing and verifying Webhook payloads. Replace this with a secure, random string.|

### 2. Create a Table in RisingWave to Receive Webhook Data
## 2. Create a table in RisingWave

Next, create a table configured to accept webhook data from GitHub.
Next, create a table configured to accept Webhook data from GitHub.

```sql
CREATE TABLE wbhtable (
data JSONB
) WITH (
connector = 'webhook'
) VALIDATE SECRET test_secret AS secure_compare(
headers->>'x-hub-signature-256',
headers->>'x-hub-signature-256', ## Example value: `sha256=f37a93a68fef1505d75e920a15d0543199557be72d2182e5cf8c15d7f9a6260f`
'sha256=' || encode(hmac(test_secret, data, 'sha256'), 'hex')
);
```

Explanation:
- `data JSONB`: Defines the name of column to store the JSON payload from the webhook. Currently, only `JSONB` type is supported for webhook tables.
- `headers->>'x-hub-signature-256'`: Extracts the signature provided by GitHub in the x-hub-signature-256 HTTP header. An example of the value is `sha256=f37a93a68fef1505d75e920a15d0543199557be72d2182e5cf8c15d7f9a6260f`. Note that in `secure_compare()` function, the whole HTTP header is interpreted as a JSONB object, and you can access the header value using the `->>` operator. But please only use the lower-case header names in the `->>` operator. The verification will fail, otherwise.
- `'sha256=' || encode(hmac(test_secret, data, 'sha256'), 'hex')`: Computes the expected signature by generating an HMAC SHA-256 hash of the payload (`data`) using the secret (`test_secret`), encodes it in hexadecimal, and prefixes it with `sha256=`.

The `secure_compare()` function compares the signature from the request header with the computed signature. If they match, the request is accepted; otherwise, it is rejected. This ensures that only authentic requests from GitHub are processed.
| Parameter or clause | Description |
| :--------- | :-----------|
|`data JSONB` | Defines the name of column to store the JSON payload from the Webhook. Currently, only `JSONB` type is supported for Webhook tables. |
| `headers->>'...'` | Extracts the signature provided by GitHub in the `x-hub-signature-256` HTTP header. <br/> <br/> In `secure_compare()` function, the whole HTTP header is interpreted as a JSONB object, and you can access the header value using the `->>` operator, but only the lower-case header names in the `->>` operator, otherwise the verification will fail. |
|`'sha256=' \|\| encode(...)` | Computes the expected signature. In the example above, it generates an `HMAC SHA-256` hash of the payload (`data`) using the secret (`test_secret`), encodes it in hexadecimal, and prefixes it with `sha256=`.|
| `secure_compare()` | Validates requests by matching the header signature against the computed signature, ensuring only authenticated requests are processed. |

In GitHub Webhook, you can choose between SHA-1 and SHA-256 HMAC algorithms for signing the payload. The example above uses SHA-256 for demonstration purposes. If you want to use SHA-1, replace `x-hub-signature-256` with `x-hub-signature` and `sha256` with `sha1` in the `VALIDATE` clause. An example is here:
In GitHub Webhook, you can choose between `SHA-1` and `SHA-256 HMAC` algorithms for signing the payload. The example above uses `SHA-256 HMAC`. If you want to use `SHA-1`, change `x-hub-signature-256` into `x-hub-signature`, `sha256` into `sha1` in the `VALIDATE` clause.

```sql
```sql Example using SHA-1
CREATE SECRET test_secret WITH ( backend = 'meta') AS 'TEST_WEBHOOK';
-- webhook table example github
create table wbhtable (
Expand All @@ -59,55 +58,54 @@ create table wbhtable (
);
```

### 3. Set Up the Webhook in GitHub
## 3. Set up Webhook in GitHub

After configuring RisingWave to accept webhook data, set up GitHub to send events to your RisingWave instance.
After configuring RisingWave to accept Webhook data, set up GitHub to send events to your RisingWave instance.

#### RisingWave Webhook URL
### RisingWave Webhook URL

The webhook URL should follow this format:
The Webhook URL should follow this format:
```
https://<HOST>/webhook/<database>/<schema_name>/<table_name>
```

Explanation:

- `<HOST>`: The hostname or IP address where your RisingWave instance is accessible. This could be a domain name or an IP address.
- `<database>`: The name of the RisingWave database where your table resides.
- `<schema_name>`: The schema name of your table, typically `public` unless specified otherwise.
- `<table_name>`: The name of the table you created to receive webhook data (e.g., `wbhtable` in the above example).

#### Configuring the Webhook in GitHub

1. Navigate to Your Repository Settings:

- Go to your GitHub repository.
- Click on the **Settings** tab.

2. Add a New Webhook:

- In the left sidebar, click on **Webhooks**.
- Click the **Add webhook** button.

3. Configure the Webhook Settings:

- **Payload URL**: Enter your RisingWave webhook URL.
- **Content type**: Select `application/json`.
- **Secret**: Enter the same secret string you used when creating the RisingWave secret (e.g., `'TEST_WEBHOOK'`). This ensures that GitHub signs the payloads using this secret, allowing RisingWave to validate them.
- **Which events would you like to trigger this webhook?**: Choose the events you want to subscribe to. For testing purposes, you might start with Just the push event.
- **Active**: Ensure the webhook is set to active.


4. Save the Webhook:
- Click the **Add webhook** button at the bottom of the page.

### 4. Push Data from GitHub via Webhook

With the webhook configured, GitHub will automatically send HTTP POST requests to your RisingWave webhook URL whenever the specified events occur (e.g., pushes to the repository). RisingWave will receive these requests, validate the signatures, and insert the payload data into the target table.

### 5. Further Event Processing
| Parameter | Description |
|-----------|-------------|
| `HOST` | The hostname or IP address where your RisingWave instance is accessible. This could be a domain name or an IP address. |
| `database` | The name of the RisingWave database where your table resides |
| `schema_name` | The schema name of your table, typically `public` unless specified otherwise. |
| `table_name` | The name of the table you created to receive Webhook data, e.g., `wbhtable`. |


### Configure Webhook in GitHub
<Steps>
<Step>
Go to your GitHub repository, and click on **Settings** tab.
</Step>
<Step>
In the left sidebar, click on **Webhooks** > **Add Webhook**.
</Step>
<Step>
Configure the Webhook settings:

- **Payload URL**: Enter your RisingWave Webhook URL.
- **Content type**: Select `application/json`.
- **Secret**: Enter the same secret string you used when creating the RisingWave secret (e.g., `'TEST_WEBHOOK'`). This ensures that GitHub signs the payloads using this secret, allowing RisingWave to validate them.
- **Which events would you like to trigger this Webhook?**: Choose the events you want to subscribe to. For testing purposes, you might start with Just the push event.
- **Active**: Ensure the Webhook is set to active.
</Step>
<Step>
Click **Add Webhook** at the bottom of the page to save.
</Step>
</Steps>
## 4. Push data from GitHub via Webhook

With the Webhook configured, GitHub will automatically send HTTP POST requests to your RisingWave Webhook URL whenever the specified events occur (e.g., pushes to the repository). RisingWave will receive these requests, validate the signatures, and insert the payload data into the target table.

## 5. Further event processing
The data in the table is already ready for further processing. You can access the fields using `data->'field_name'` in SQL queries.
You can create a Materialized View to extract specific fields from the JSON payload.

You can create a materialized view to extract specific fields from the JSON payload.

```sql
CREATE MATERIALIZED VIEW github_events AS
Expand Down

0 comments on commit de7dc5a

Please sign in to comment.