From 032e3035f9f1da3c538aaf8b22f878ea3e5b5e8e Mon Sep 17 00:00:00 2001 From: "devin-ai-integration[bot]" <158243242+devin-ai-integration[bot]@users.noreply.github.com> Date: Mon, 16 Dec 2024 18:42:39 +1100 Subject: [PATCH] docs: reformat all FDW documentation (#384) * docs: reformat documentation for all FDW services - Add consistent Operations, Usage, and Notes sections - Preserve all technical content and functionality - Improve documentation structure and readability - Follow stripe.md template format Fixes #383 Co-Authored-By: copple@supabase.io * docs: reformat stripe.md documentation to match accounts template Co-Authored-By: copple@supabase.io * docs instructions * docs: standardize preparation sections across all FDW documentation Co-Authored-By: copple@supabase.io * docs: restore Query Pushdown Support sections Co-Authored-By: copple@supabase.io * small re-ordering * docs: add missing sections * update CONTRIBUTING.md --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: copple@supabase.io Co-authored-by: Copple <10214025+kiwicopple@users.noreply.github.com> Co-authored-by: Bo Lu Co-authored-by: Bo Lu --- .gitignore | 1 + CONTRIBUTING.md | 4 + docs/catalog/airtable.md | 51 +-- docs/catalog/auth0.md | 59 ++-- docs/catalog/bigquery.md | 91 +++-- docs/catalog/cal.md | 251 +++++++++---- docs/catalog/calendly.md | 224 +++++++----- docs/catalog/clickhouse.md | 157 +++++---- docs/catalog/cognito.md | 59 ++-- docs/catalog/firebase.md | 97 ++++- docs/catalog/logflare.md | 104 +++--- docs/catalog/mssql.md | 116 +++--- docs/catalog/notion.md | 196 +++++----- docs/catalog/paddle.md | 157 +++++++-- docs/catalog/redis.md | 296 ++++++++++++---- docs/catalog/s3.md | 153 ++++++-- docs/catalog/snowflake.md | 73 ++-- docs/catalog/stripe.md | 706 ++++++++++++++++++++++++++----------- 18 files changed, 1903 insertions(+), 892 deletions(-) create mode 100644 CONTRIBUTING.md diff --git a/.gitignore b/.gitignore index 12c4a2e03..b5a69196d 100644 --- a/.gitignore +++ b/.gitignore @@ -16,3 +16,4 @@ cmake*/ results/ wrappers/results/ wrappers/regression.* +.venv/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..ce57918f8 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,4 @@ +# Contributing + +Any contributions to wrappers are welcome, please visit https://fdw.dev/contributing/core/ for more details. + diff --git a/docs/catalog/airtable.md b/docs/catalog/airtable.md index 0d38b79e6..093df8ffa 100644 --- a/docs/catalog/airtable.md +++ b/docs/catalog/airtable.md @@ -19,13 +19,19 @@ The Airtable Wrapper allows you to read data from your Airtable bases/tables wit ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Airtable, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the Airtable Wrapper + +Enable the `airtable_wrapper` FDW: ```sql create foreign data wrapper airtable_wrapper @@ -33,9 +39,9 @@ create foreign data wrapper airtable_wrapper validator airtable_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) -By default, Postgres stores FDW credentials inide `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. +By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. ```sql -- Save your Airtable API key in Vault and retrieve the `key_id` @@ -72,7 +78,15 @@ We need to provide Postgres with the credentials to connect to Airtable, and any ); ``` -## Creating Foreign Tables +### Create a schema + +We recommend creating a schema to hold all the foreign tables: + +```sql +create schema if not exists airtable; +``` + +## Entities The Airtable Wrapper supports data reads from the Airtable API. @@ -82,14 +96,14 @@ The Airtable Wrapper supports data reads from Airtable's [Records](https://airta #### Operations -| Airtable | Select | Insert | Update | Delete | Truncate | -| -------- | :----: | :----: | :----: | :----: | :------: | -| Records | ✅ | ❌ | ❌ | ❌ | ❌ | +| Object | Select | Insert | Update | Delete | Truncate | +| ------- | :----: | :----: | :----: | :----: | :------: | +| Records | ✅ | ❌ | ❌ | ❌ | ❌ | #### Usage ```sql -create foreign table my_foreign_table ( +create foreign table airtable.my_foreign_table ( name text -- other fields ) @@ -100,13 +114,10 @@ options ( ); ``` -#### Options +#### Notes -The full list of foreign table options are below: - -- `base_id` - Airtable Base ID the table belongs to, required. -- `table_id` - Airtable table ID, required. -- `view_id` - Airtable view ID, optional. +- The table requires both `base_id` and `table_id` options +- Optional `view_id` can be specified to query a specific view ## Query Pushdown Support @@ -114,14 +125,12 @@ This FDW doesn't support query pushdown. ## Examples -Some examples on how to use Airtable foreign tables. - ### Query an Airtable table This will create a "foreign table" inside your Postgres database called `airtable_table`: ```sql -create foreign table airtable_table ( +create foreign table airtable.airtable_table ( name text, notes text, content text, @@ -138,7 +147,7 @@ options ( You can now fetch your Airtable data from within your Postgres database: ```sql -select * from airtable_table; +select * from airtable.airtable_table; ``` ### Query an Airtable view @@ -146,7 +155,7 @@ select * from airtable_table; We can also create a foreign table from an Airtable View called `airtable_view`: ```sql -create foreign table airtable_view ( +create foreign table airtable.airtable_view ( name text, notes text, content text, @@ -164,5 +173,5 @@ options ( You can now fetch your Airtable data from within your Postgres database: ```sql -select * from airtable_view; +select * from airtable.airtable_view; ``` diff --git a/docs/catalog/auth0.md b/docs/catalog/auth0.md index dcffe36b4..93b70d9fa 100644 --- a/docs/catalog/auth0.md +++ b/docs/catalog/auth0.md @@ -19,13 +19,19 @@ The Auth0 Wrapper allows you to read data from your Auth0 tenant for use within ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Auth0, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the Auth0 Wrapper + +Enable the `auth0_wrapper` FDW: ```sql create foreign data wrapper auth0_wrapper @@ -33,9 +39,9 @@ create foreign data wrapper auth0_wrapper validator auth0_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) -By default, Postgres stores FDW credentials inide `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. +By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. ```sql -- Save your Auth0 API key in Vault and retrieve the `key_id` @@ -73,32 +79,44 @@ We need to provide Postgres with the credentials to connect to Airtable, and any ); ``` -## Creating Foreign Tables +### Create a schema + +We recommend creating a schema to hold all the foreign tables: + +```sql +create schema if not exists auth0; +``` + +## Entities + +The Auth0 Wrapper supports data reads from Auth0 API. + +### Users The Auth0 Wrapper supports data reads from Auth0's [Management API List users endpoint](https://auth0.com/docs/api/management/v2/users/get-users) endpoint (_read only_). -| Auth0 | Select | Insert | Update | Delete | Truncate | -| ------- | :----: | :----: | :----: | :----: | :------: | -| Records | ✅ | ❌ | ❌ | ❌ | ❌ | +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Users | ✅ | ❌ | ❌ | ❌ | ❌ | -For example: +#### Usage ```sql -create foreign table my_foreign_table ( +create foreign table auth0.my_foreign_table ( name text -- other fields ) server auth0_server options ( - object 'users', + object 'users' ); ``` -### Foreign table options +#### Notes -The full list of foreign table options are below: - -- `objects` - Auth0 object to select from. Currently only supports `users` +- Currently only supports the `users` object ## Query Pushdown Support @@ -106,14 +124,12 @@ This FDW doesn't support query pushdown. ## Examples -Some examples on how to use Auth0 foreign tables. - -### Basic example +### Basic Auth0 Users Query -This will create a "foreign table" inside your Postgres database called `auth0_table`: +This example demonstrates querying Auth0 users data. ```sql -create foreign table auth0_table ( +create foreign table auth0.auth0_table ( created_at text, email text, email_verified bool, @@ -123,11 +139,10 @@ create foreign table auth0_table ( options ( object 'users' ); - ``` You can now fetch your Auth0 data from within your Postgres database: ```sql -select * from auth0; +select * from auth0.auth0_table; ``` diff --git a/docs/catalog/bigquery.md b/docs/catalog/bigquery.md index 29c709f80..82a0cdec0 100644 --- a/docs/catalog/bigquery.md +++ b/docs/catalog/bigquery.md @@ -34,13 +34,19 @@ The BigQuery Wrapper allows you to read and write data from BigQuery within your ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query BigQuery, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the BigQuery Wrapper + +Enable the `bigquery_wrapper` FDW: ```sql create foreign data wrapper bigquery_wrapper @@ -48,9 +54,9 @@ create foreign data wrapper bigquery_wrapper validator big_query_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) -By default, Postgres stores FDW credentials inide `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. +By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. ```sql -- Save your BigQuery service account json in Vault and retrieve the `key_id` @@ -104,18 +110,47 @@ We need to provide Postgres with the credentials to connect to BigQuery, and any ); ``` -## Creating Foreign Tables +### Create a schema + +We recommend creating a schema to hold all the foreign tables: + +```sql +create schema if not exists bigquery; +``` + +## Options -The BigQuery Wrapper supports data reads and writes from BigQuery. +The following options are available when creating BigQuery foreign tables: -| Integration | Select | Insert | Update | Delete | Truncate | -| ----------- | :----: | :----: | :----: | :----: | :------: | -| BigQuery | ✅ | ✅ | ✅ | ✅ | ❌ | +- `table` - Source table or view name in BigQuery, required +- `location` - Source table location (default: 'US') +- `timeout` - Query request timeout in milliseconds (default: 30000) +- `rowid_column` - Primary key column name (required for data modification) -For example: +You can also use a subquery as the table option: ```sql -create foreign table my_bigquery_table ( +table '(select * except(props), to_json_string(props) as props from `my_project.my_dataset.my_table`)' +``` + +Note: When using subquery, full qualified table name must be used. + +## Entites + +### Tables + +The BigQuery Wrapper supports data reads and writes from BigQuery tables and views. + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Tables | ✅ | ✅ | ✅ | ✅ | ❌ | + +#### Usage + +```sql +create foreign table bigquery.my_bigquery_table ( id bigint, name text, ts timestamp @@ -127,23 +162,11 @@ create foreign table my_bigquery_table ( ); ``` -### Foreign table options - -The full list of foreign table options are below: - -- `table` - Source table or view name in BigQuery, required. - - This can also be a subquery enclosed in parentheses, for example, - - ```sql - table '(select * except(props), to_json_string(props) as props from `my_project.my_dataset.my_table`)' - ``` - - **Note**: When using subquery in this option, full qualitified table name must be used. +#### Notes -- `location` - Source table location, optional. Default is 'US'. -- `timeout` - Query request timeout in milliseconds, optional. Default is '30000' (30 seconds). -- `rowid_column` - Primary key column name, optional for data scan, required for data modify +- Supports `where`, `order by` and `limit` clause pushdown +- When using `rowid_column`, it must be specified for data modification operations +- Data in the streaming buffer cannot be updated or deleted until the buffer is flushed (up to 90 minutes) ## Query Pushdown Support @@ -151,7 +174,7 @@ This FDW supports `where`, `order by` and `limit` clause pushdown. ## Inserting Rows & the Streaming Buffer -This foreign data wrapper uses BigQuery’s `insertAll` API method to create a `streamingBuffer` with an associated partition time. **Within that partition time, the data cannot be updated, deleted, or fully exported**. Only after the time has elapsed (up to 90 minutes according to [BigQuery’s documentation](https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery)); can you perform operations. +This foreign data wrapper uses BigQuery’s `insertAll` API method to create a `streamingBuffer` with an associated partition time. **Within that partition time, the data cannot be updated, deleted, or fully exported**. Only after the time has elapsed (up to 90 minutes according to [BigQuery’s documentation](https://cloud.google.com/bigquery/docs/streaming-data-into-bigquery)), can you perform operations. If you attempt an `UPDATE` or `DELETE` statement on rows while in the streamingBuffer, you will get an error of `UPDATE` or `DELETE` statement over table datasetName - note that tableName would affect rows in the streaming buffer, which is not supported. @@ -181,7 +204,7 @@ insert into your_project_id.your_dataset_id.people values This example will create a "foreign table" inside your Postgres database called `people` and query its data: ```sql -create foreign table people ( +create foreign table bigquery.people ( id bigint, name text, ts timestamp @@ -192,7 +215,7 @@ create foreign table people ( location 'EU' ); -select * from people; +select * from bigquery.people; ``` ### Data modify example @@ -200,7 +223,7 @@ select * from people; This example will modify data in a "foreign table" inside your Postgres database called `people`, note that `rowid_column` option is mandatory: ```sql -create foreign table people ( +create foreign table bigquery.people ( id bigint, name text, ts timestamp @@ -213,15 +236,15 @@ create foreign table people ( ); -- insert new data -insert into people(id, name, ts) +insert into bigquery.people(id, name, ts) values (4, 'Yoda', '2023-01-01 12:34:56'); -- update existing data -update people +update bigquery.people set name = 'Anakin Skywalker' where id = 1; -- delete data -delete from people +delete from bigquery.people where id = 2; ``` diff --git a/docs/catalog/cal.md b/docs/catalog/cal.md index 4ecea3b2a..ff67f1efc 100644 --- a/docs/catalog/cal.md +++ b/docs/catalog/cal.md @@ -19,13 +19,13 @@ The Cal Wrapper is a WebAssembly(Wasm) foreign data wrapper which allows you to ## Supported Data Types -| Postgres Data Type | Cal.com Data Type | -| ------------------ | ------------------ | -| boolean | Boolean | -| bigint | Number | -| double precision | Number | -| text | String | -| jsonb | Json | +| Postgres Data Type | Cal.com Data Type | +| ------------------ | ----------------- | +| boolean | Boolean | +| bigint | Number | +| double precision | Number | +| text | String | +| jsonb | Json | The Cal.com API uses JSON formatted data, please refer to [Cal.com API docs](https://cal.com/docs/api-reference/v2/introduction) for more details. @@ -41,13 +41,19 @@ The Cal.com API uses JSON formatted data, please refer to [Cal.com API docs](htt ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Cal.com, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the Wasm foreign data wrapper: +### Enable the Cal.com Wrapper + +Enable the Wasm foreign data wrapper: ```sql create foreign data wrapper wasm_wrapper @@ -55,7 +61,7 @@ create foreign data wrapper wasm_wrapper validator wasm_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. @@ -64,7 +70,7 @@ By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server insert into vault.secrets (name, secret) values ( 'cal', - '' -- Cal.com API key + '' -- Cal.com API key ) returning key_id; ``` @@ -113,20 +119,40 @@ We recommend creating a schema to hold all the foreign tables: create schema if not exists cal; ``` -## Creating Foreign Tables +## Options + +The full list of foreign table options are below: + +- `object` - Object name in Cal.com, required. + +Supported objects are listed below: + +| Object name | +| ------------ | +| my_profile | +| event-types | +| bookings | +| calendars | +| schedules | +| conferencing | + +## Entities -The Cal.com Wrapper supports data reads from below objects in Cal.com. +The Cal.com Wrapper supports data reads and booking writes from the Cal.com API. -| Integration | Select | Insert | Update | Delete | Truncate | -| -------------| :----: | :----: | :----: | :----: | :------: | -| My Profile | ✅ | ❌ | ❌ | ❌ | ❌ | -| Event Types | ✅ | ❌ | ❌ | ❌ | ❌ | -| Bookings | ✅ | ✅ | ❌ | ❌ | ❌ | -| Calendars | ✅ | ❌ | ❌ | ❌ | ❌ | -| Schedules | ✅ | ❌ | ❌ | ❌ | ❌ | -| Conferencing | ✅ | ❌ | ❌ | ❌ | ❌ | +### Profile -For example: +This is an object representing your Cal.com user profile. + +Ref: [Cal.com API docs](https://cal.com/docs/api-reference/v2/introduction) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------- | :----: | :----: | :----: | :----: | :------: | +| Profile | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage ```sql create foreign table cal.my_profile ( @@ -139,7 +165,27 @@ create foreign table cal.my_profile ( options ( object 'my_profile' ); +``` + +#### Notes + +- The `attrs` column contains all profile attributes in JSON format + +### Event Types + +This is an object representing Cal.com event types. + +Ref: [Cal.com API docs](https://cal.com/docs/api-reference/v2/introduction) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------- | :----: | :----: | :----: | :----: | :------: | +| Event Types | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table cal.event_types ( attrs jsonb ) @@ -147,7 +193,37 @@ create foreign table cal.event_types ( options ( object 'event-types' ); - +``` + +#### Notes + +- The `attrs` column contains all event type attributes in JSON format +- Extract specific fields using JSON operators, for example: + ```sql + select + etg->'profile'->>'name' as profile, + et->>'id' as id, + et->>'title' as title + from cal.event_types t + cross join json_array_elements((attrs->'eventTypeGroups')::json) etg + cross join json_array_elements((etg->'eventTypes')::json) et; + ``` + +### Bookings + +This is an object representing Cal.com bookings. + +Ref: [Cal.com API docs](https://cal.com/docs/api-reference/v2/bookings/create-a-booking) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| -------- | :----: | :----: | :----: | :----: | :------: | +| Bookings | ✅ | ✅ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table cal.bookings ( attrs jsonb ) @@ -156,7 +232,45 @@ create foreign table cal.bookings ( object 'bookings', rowid_column 'attrs' ); +``` + +#### Notes + +- Supports both reading and creating bookings +- The `attrs` column contains all booking attributes in JSON format +- Example of creating a booking: + ```sql + insert into cal.bookings(attrs) + values ( + '{ + "start": "2024-12-12T10:30:00.000Z", + "eventTypeId": 123456, + "attendee": { + "name": "Test Name", + "email": "test.name@example.com", + "timeZone": "America/New_York" + } + }'::jsonb + ); + ``` +- Additional fields like `guests` or `metadata` can be added to the booking JSON +- For more details on booking options, refer to [Cal.com documentation](https://cal.com/docs/api-reference/v2/bookings/create-a-booking) + +### Calendars + +This is an object representing Cal.com calendars. +Ref: [Cal.com API docs](https://cal.com/docs/api-reference/v2/introduction) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| --------- | :----: | :----: | :----: | :----: | :------: | +| Calendars | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table cal.calendars ( attrs jsonb ) @@ -164,7 +278,27 @@ create foreign table cal.calendars ( options ( object 'calendars' ); +``` + +#### Notes + +- The `attrs` column contains all calendar attributes in JSON format + +### Schedules + +This is an object representing Cal.com schedules. +Ref: [Cal.com API docs](https://cal.com/docs/api-reference/v2/introduction) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| --------- | :----: | :----: | :----: | :----: | :------: | +| Schedules | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table cal.schedules ( id bigint, name text, @@ -174,7 +308,27 @@ create foreign table cal.schedules ( options ( object 'schedules' ); +``` + +#### Notes + +- The `attrs` column contains additional schedule attributes in JSON format + +### Conferencing + +This is an object representing Cal.com conferencing settings. +Ref: [Cal.com API docs](https://cal.com/docs/api-reference/v2/introduction) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------ | :----: | :----: | :----: | :----: | :------: | +| Conferencing | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table cal.conferencing ( id bigint, attrs jsonb @@ -185,28 +339,9 @@ create foreign table cal.conferencing ( ); ``` -!!! note - - - All the supported columns are listed above, other columns are not allowd. - - The `attrs` is a special column which stores all the object attributes in JSON format, you can extract any attributes needed from it. See more examples below. - - `bookings` table support data insertion, see an example below. +#### Notes -### Foreign table options - -The full list of foreign table options are below: - -- `object` - Object name in Cal.com, required. - - Supported objects are listed below: - - | Object name | - | ------------------------ | - | my_profile | - | event-types | - | bookings | - | calendars | - | schedules | - | conferencing | +- The `attrs` column contains all conferencing attributes in JSON format ## Query Pushdown Support @@ -218,13 +353,7 @@ Below are some examples on how to use Cal.com foreign tables. ### Basic example -This example will create a "foreign table" inside your Postgres database and query its data. First, we can create a schema to hold all the Cal.com foreign tables. - -```sql -create schema if not exists cal; -``` - -Then create the foreign table and query it, for example: +This example will create a "foreign table" inside your Postgres database and query its data. ```sql create foreign table cal.my_profile ( @@ -263,7 +392,7 @@ create foreign table cal.event_types ( options ( object 'event-types' ); - + -- extract bookings select bk->>'id' as id, @@ -289,15 +418,15 @@ Once we know an event type ID (we can get it from above example, here we suppose ``` insert into cal.bookings(attrs) values ( - '{ - "start": "2024-12-12T10:30:00.000Z", - "eventTypeId": 123456, - "attendee": { - "name": "Test Name", - "email": "test.name@example.com", - "timeZone": "America/New_York" - } - }'::jsonb + '{ + "start": "2024-12-12T10:30:00.000Z", + "eventTypeId": 123456, + "attendee": { + "name": "Test Name", + "email": "test.name@example.com", + "timeZone": "America/New_York" + } + }'::jsonb ); ``` diff --git a/docs/catalog/calendly.md b/docs/catalog/calendly.md index 4f931ffbf..786d4e65b 100644 --- a/docs/catalog/calendly.md +++ b/docs/catalog/calendly.md @@ -33,19 +33,25 @@ The Calendly API uses JSON formatted data, please refer to [Calendly API docs](h ## Available Versions -| Version | Wasm Package URL | Checksum | -| ------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| Version | Wasm Package URL | Checksum | +| ------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | | 0.1.0 | `https://github.com/supabase/wrappers/releases/download/wasm_calendly_fdw_v0.1.0/calendly_fdw.wasm` | `51a19fa4b8c40afb5dcf6dc2e009189aceeba65f30eec75d56a951d78fc8893f` | ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Calendly, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the Wasm foreign data wrapper: +### Enable the Calendly Wrapper + +Enable the Wasm foreign data wrapper: ```sql create foreign data wrapper wasm_wrapper @@ -53,7 +59,7 @@ create foreign data wrapper wasm_wrapper validator wasm_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. @@ -62,51 +68,11 @@ By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server insert into vault.secrets (name, secret) values ( 'calendly', - '' -- Calendly personal access token + '' -- Calendly personal access token ) returning key_id; ``` -### Connecting to Calendly - -We need to provide Postgres with the credentials to access Calendly and any additional options. We can do this using the `create server` command: - -=== "With Vault" - - ```sql - create server calendly_server - foreign data wrapper wasm_wrapper - options ( - fdw_package_url 'https://github.com/supabase/wrappers/releases/download/wasm_calendly_fdw_v0.1.0/calendly_fdw.wasm', - fdw_package_name 'supabase:calendly-fdw', - fdw_package_version '0.1.0', - fdw_package_checksum '51a19fa4b8c40afb5dcf6dc2e009189aceeba65f30eec75d56a951d78fc8893f', - -- find your organization uri using foreign table 'calendly.current_user', see below example for details - organization 'https://api.calendly.com/organizations/81da9c7f-3e19-434a-c3d2-0325e375cdef', - api_url 'https://api.calendly.com', -- optional - api_key_id '' -- The Key ID from above. - ); - ``` - -=== "Without Vault" - - ```sql - create server calendly_server - foreign data wrapper wasm_wrapper - options ( - fdw_package_url 'https://github.com/supabase/wrappers/releases/download/wasm_calendly_fdw_v0.1.0/calendly_fdw.wasm', - fdw_package_name 'supabase:calendly-fdw', - fdw_package_version '0.1.0', - fdw_package_checksum '51a19fa4b8c40afb5dcf6dc2e009189aceeba65f30eec75d56a951d78fc8893f', - -- find your organization uri using foreign table 'calendly.current_user', see below example for details - organization 'https://api.calendly.com/organizations/81da9c7f-3e19-434a-c3d2-0325e375cdef', - api_url 'https://api.calendly.com', -- optional - api_key 'eyJraWQiOiIxY2UxZ...' -- Calendly personal access token - ); - ``` - -Note the `fdw_package_*` options are required, which specify the Wasm package metadata. You can get the available package version list from [above](#available-versions). - ### Create a schema We recommend creating a schema to hold all the foreign tables: @@ -115,25 +81,39 @@ We recommend creating a schema to hold all the foreign tables: create schema if not exists calendly; ``` -## Creating Foreign Tables +## Options -The Calendly Wrapper supports data reads from below objects in calendly. +The full list of foreign table options are below: -| Integration | Select | Insert | Update | Delete | Truncate | -| ----------------------- | :----: | :----: | :----: | :----: | :------: | -| Current User | ✅ | ❌ | ❌ | ❌ | ❌ | -| Event Types | ✅ | ❌ | ❌ | ❌ | ❌ | -| Groups | ✅ | ❌ | ❌ | ❌ | ❌ | -| Organization Membership | ✅ | ❌ | ❌ | ❌ | ❌ | -| Scheduled Events | ✅ | ❌ | ❌ | ❌ | ❌ | +- `object` - Object name in Calendly, required. + +Supported objects are listed below: + +| Object name | +| ------------------------ | +| current_user | +| event_types | +| groups | +| organization_memberships | +| scheduled_events | -For example: +## Entities + +### Current User + +This is an object representing your Calendly user profile. + +Ref: [Calendly API docs](https://developer.calendly.com/api-docs) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------ | :----: | :----: | :----: | :----: | :------: | +| Current User | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage ```sql --- Get the current user used for the API request --- Note: we can query this table to retrieve the organization uri: --- select attrs->>'current_organization' as org_uri --- from calendly.current_user; create foreign table calendly.current_user ( uri text, slug text, @@ -145,7 +125,32 @@ create foreign table calendly.current_user ( options ( object 'current_user' ); +``` + +#### Notes + +- The `attrs` column contains additional user attributes in JSON format +- Use this table to retrieve the organization URI for server configuration, for example: + ```sql + select attrs->>'current_organization' as org_uri + from calendly.current_user; + ``` +### Event Types + +This is an object representing Calendly event types. + +Ref: [Calendly API docs](https://developer.calendly.com/api-docs) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------- | :----: | :----: | :----: | :----: | :------: | +| Event Types | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table calendly.event_types ( uri text, created_at timestamp, @@ -156,7 +161,33 @@ create foreign table calendly.event_types ( options ( object 'event_types' ); - +``` + +#### Notes + +- The `attrs` column contains all event type attributes in JSON format +- Access profile and custom questions through JSON attributes: + ```sql + select attrs->'profile'->>'name' as profile_name, + attrs->'custom_questions'->0->>'name' as first_question_name + from calendly.event_types; + ``` + +### Groups + +This is an object representing Calendly groups. + +Ref: [Calendly API docs](https://developer.calendly.com/api-docs) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Groups | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table calendly.groups ( uri text, created_at timestamp, @@ -167,7 +198,27 @@ create foreign table calendly.groups ( options ( object 'groups' ); +``` + +#### Notes + +- The `attrs` column contains all group attributes in JSON format + +### Organization Memberships + +This is an object representing Calendly organization memberships. + +Ref: [Calendly API docs](https://developer.calendly.com/api-docs) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------------------- | :----: | :----: | :----: | :----: | :------: | +| Organization Membership | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage +```sql create foreign table calendly.organization_memberships ( uri text, created_at timestamp, @@ -178,7 +229,27 @@ create foreign table calendly.organization_memberships ( options ( object 'organization_memberships' ); +``` + +#### Notes + +- The `attrs` column contains all membership attributes in JSON format + +### Scheduled Events + +This is an object representing Calendly scheduled events. + +Ref: [Calendly API docs](https://developer.calendly.com/api-docs) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ---------------- | :----: | :----: | :----: | :----: | :------: | +| Scheduled Events | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table calendly.scheduled_events ( uri text, created_at timestamp, @@ -191,26 +262,9 @@ create foreign table calendly.scheduled_events ( ); ``` -!!! note - - - All the supported columns are listed above, other columns are not allowd. - - The `attrs` is a special column which stores all the object attributes in JSON format, you can extract any attributes needed from it. See more examples below. +#### Notes -### Foreign table options - -The full list of foreign table options are below: - -- `object` - Object name in Calendly, required. - - Supported objects are listed below: - - | Object name | - | ------------------------ | - | current_user | - | event_types | - | groups | - | organization_memberships | - | scheduled_events | +- The `attrs` column contains all event attributes in JSON format ## Query Pushdown Support @@ -222,13 +276,7 @@ Below are some examples on how to use Calendly foreign tables. ### Basic example -This example will create a "foreign table" inside your Postgres database and query its data. First, we can create a schema to hold all the Calendly foreign tables. - -```sql -create schema if not exists calendly; -``` - -Then create the foreign table and query it, for example: +This example will create a "foreign table" inside your Postgres database and query its data. ```sql create foreign table calendly.current_user ( @@ -273,7 +321,7 @@ create foreign table calendly.event_types ( options ( object 'event_types' ); - + select attrs->'profile'->>'name' as profile_name from calendly.event_types; diff --git a/docs/catalog/clickhouse.md b/docs/catalog/clickhouse.md index 72bb1e37a..dbcd5ef3b 100644 --- a/docs/catalog/clickhouse.md +++ b/docs/catalog/clickhouse.md @@ -19,31 +19,37 @@ The ClickHouse Wrapper allows you to read and write data from ClickHouse within ## Supported Data Types -| Postgres Type | ClickHouse Type | -| ---------------- | --------------- | -| boolean | UInt8 | -| smallint | Int16 | -| integer | UInt16 | -| integer | Int32 | -| bigint | UInt32 | -| bigint | Int64 | -| bigint | UInt64 | -| real | Float32 | -| double precision | Float64 | -| text | String | -| date | Date | -| timestamp | DateTime | +| Postgres Type | ClickHouse Type | +| ---------------- | ----------------- | +| boolean | UInt8 | +| smallint | Int16 | +| integer | UInt16 | +| integer | Int32 | +| bigint | UInt32 | +| bigint | Int64 | +| bigint | UInt64 | +| real | Float32 | +| double precision | Float64 | +| text | String | +| date | Date | +| timestamp | DateTime | | * | Nullable<T> | ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query ClickHouse, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the ClickHouse Wrapper + +Enable the `clickhouse_wrapper` FDW: ```sql create foreign data wrapper clickhouse_wrapper @@ -51,9 +57,9 @@ create foreign data wrapper clickhouse_wrapper validator click_house_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) -By default, Postgres stores FDW credentials inide `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. +By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. ```sql -- Save your ClickHouse credential in Vault and retrieve the `key_id` @@ -96,58 +102,78 @@ Some connection string examples: Check [more connection string parameters](https://github.com/suharev7/clickhouse-rs#dns). -## Creating Foreign Tables +### Create a schema -The ClickHouse Wrapper supports data reads and writes from ClickHouse. +We recommend creating a schema to hold all the foreign tables: -| Integration | Select | Insert | Update | Delete | Truncate | -| ----------- | :----: | :----: | :----: | :----: | :------: | -| ClickHouse | ✅ | ✅ | ✅ | ✅ | ❌ | +```sql +create schema if not exists clickhouse; +``` -For example: +## Options + +The following options are available when creating ClickHouse foreign tables: + +- `table` - Source table name in ClickHouse, required + +This can also be a subquery enclosed in parentheses, for example, ```sql -create foreign table my_clickhouse_table ( - id bigint, - name text -) - server clickhouse_server - options ( - table 'people' - ); +table '(select * from my_table)' ``` -### Foreign table options +### Parametrized views -The full list of foreign table options are below: +[Parametrized view](https://clickhouse.com/docs/en/sql-reference/statements/create/view#parameterized-view) is also supported in the subquery. In this case, you need to define a column for each parameter and use `where` to pass values to them. For example, -- `table` - Source table name in ClickHouse, required. +```sql + create foreign table test_vw ( + id bigint, + col1 text, + col2 bigint, + _param1 text, + _param2 bigint + ) + server clickhouse_server + options ( + table '(select * from my_view(column1=${_param1}, column2=${_param2}))' + ); + + select * from test_vw where _param1='aaa' and _param2=32; +``` - This can also be a subquery enclosed in parentheses, for example, +- `rowid_column` - Primary key column name, optional for data scan, required for data modify - ```sql - table '(select * from my_table)' - ``` +## Entities - [Parametrized view](https://clickhouse.com/docs/en/sql-reference/statements/create/view#parameterized-view) is also supported in the subquery. In this case, you need to define a column for each parameter and use `where` to pass values to them. For example, +### Tables - ```sql - create foreign table test_vw ( - id bigint, - col1 text, - col2 bigint, - _param1 text, - _param2 bigint - ) - server clickhouse_server - options ( - table '(select * from my_view(column1=${_param1}, column2=${_param2}))' - ); +The ClickHouse Wrapper supports data reads and writes from ClickHouse tables. - select * from test_vw where _param1='aaa' and _param2=32; - ``` +#### Operations -- `rowid_column` - Primary key column name, optional for data scan, required for data modify +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Tables | ✅ | ✅ | ✅ | ✅ | ❌ | + +#### Usage + +```sql +create foreign table clickhouse.my_table ( + id bigint, + name text +) + server clickhouse_server + options ( + table 'people' + ); +``` + +#### Notes + +- Supports `where`, `order by` and `limit` clause pushdown +- Supports parametrized views in subqueries +- When using `rowid_column`, it must be specified for data modification operations ## Query Pushdown Support @@ -155,18 +181,15 @@ This FDW supports `where`, `order by` and `limit` clause pushdown, as well as pa ## Examples -Some examples on how to use ClickHouse foreign tables. - -### Basic example +### Basic Query Example -This will create a "foreign table" inside your Postgres database called `people`: +This example demonstrates basic ClickHouse table operations. ```sql -- Run below SQLs on ClickHouse to create source table -drop table if exists people; create table people ( - id Int64, - name String + id Int64, + name String ) engine=MergeTree() order by id; @@ -178,7 +201,7 @@ insert into people values (1, 'Luke Skywalker'), (2, 'Leia Organa'), (3, 'Han So Create foreign table on Postgres database: ```sql -create foreign table people ( +create foreign table clickhouse.people ( id bigint, name text ) @@ -188,10 +211,10 @@ create foreign table people ( ); -- data scan -select * from people; +select * from clickhouse.people; -- data modify -insert into people values (4, 'Yoda'); -update people set name = 'Princess Leia' where id = 2; -delete from people where id = 3; +insert into clickhouse.people values (4, 'Yoda'); +update clickhouse.people set name = 'Princess Leia' where id = 2; +delete from clickhouse.people where id = 3; ``` diff --git a/docs/catalog/cognito.md b/docs/catalog/cognito.md index ad28b6bef..f4974ed7d 100644 --- a/docs/catalog/cognito.md +++ b/docs/catalog/cognito.md @@ -19,13 +19,19 @@ The Cognito wrapper allows you to read data from your Cognito Userpool within yo ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query AWS Cognito, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the Cognito Wrapper + +Enable the `cognito_wrapper` FDW: ```sql create foreign data wrapper cognito_wrapper @@ -33,9 +39,9 @@ create foreign data wrapper cognito_wrapper validator cognito_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) -By default, Postgres stores FDW credentials inide `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers are designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. +By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers are designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. ```sql insert into vault.secrets (name, secret) @@ -76,18 +82,32 @@ We need to provide Postgres with the credentials to connect to Cognito, and any ); ``` -## Creating Foreign Tables +### Create a schema + +We recommend creating a schema to hold all the foreign tables: + +```sql +create schema if not exists cognito; +``` + +## Entities -The Cognito Wrapper supports data reads from Cognito's [User Records](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools.html) endpoint (_read only_). +### Users -| Cognito | Select | Insert | Update | Delete | Truncate | -| ------- | :----: | :----: | :----: | :----: | :------: | -| Users | ✅ | ❌ | ❌ | ❌ | ❌ | +This is an object representing Cognito User Records. -For example: +Ref: [AWS Cognito User Records](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-identity-pools.html) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Users | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage ```sql -create foreign table cognito ( +create foreign table cognito.users ( username text, email text, status text, @@ -102,15 +122,10 @@ options ( ); ``` -!!! note - - Only columns listed above are accepted in the foreign table. +#### Notes -### Foreign table options - -The full list of foreign table options are below: - -- `object`: type of object we are querying. For now, only `users` is supported +- Only the columns listed above are accepted in the foreign table +- The `attributes` column contains additional user attributes in JSON format ## Query Pushdown Support @@ -118,14 +133,12 @@ This FDW doesn't support query pushdown. ## Examples -Some examples on how to use Cognito foreign tables. - ### Basic example This will create a "foreign table" inside your Postgres database called `cognito_table`: ```sql -create foreign table cognito_table ( +create foreign table cognito.users ( username text, email text, status text, @@ -143,5 +156,5 @@ options ( You can now fetch your Cognito data from within your Postgres database: ```sql -select * from cognito_table; +select * from cognito.users; ``` diff --git a/docs/catalog/firebase.md b/docs/catalog/firebase.md index c09492e51..bd03e2a2e 100644 --- a/docs/catalog/firebase.md +++ b/docs/catalog/firebase.md @@ -20,13 +20,19 @@ tags: ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Firebase, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the Firebase Wrapper + +Enable the `firebase_wrapper` FDW: ```sql create foreign data wrapper firebase_wrapper @@ -34,9 +40,9 @@ create foreign data wrapper firebase_wrapper validator firebase_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) -By default, Postgres stores FDW credentials inide `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. +By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. ```sql -- Save your Firebase credentials in Vault and retrieve the `key_id` @@ -84,19 +90,44 @@ We need to provide Postgres with the credentials to connect to Firebase, and any ); ``` -## Creating Foreign Tables +### Create a schema -The Firebase Wrapper supports reading data from below Firebase's objects: +We recommend creating a schema to hold all the foreign tables: -| Firebase | Select | Insert | Update | Delete | Truncate | -| ---------------------------- | :----: | :----: | :----: | :----: | :------: | -| Authentication Users | ✅ | ❌ | ❌ | ❌ | ❌ | -| Firestore Database Documents | ✅ | ❌ | ❌ | ❌ | ❌ | +```sql +create schema if not exists firebase; +``` + +## Options + +The full list of foreign table options are below: + +- `object` - Object name in Firebase, required. + + For Authenciation users, the object name is fixed to `auth/users`. For Firestore documents, its format is `firestore/`, note that collection id must be a full path id. For example, + + - `firestore/my-collection` + - `firestore/my-collection/my-document/another-collection` + + +## Entities + +### Authentication Users + +This is an object representing Firebase Authentication Users. + +Ref: [Firebase Authentication Users](https://firebase.google.com/docs/auth/users) + +#### Operations -For example: +| Object | Select | Insert | Update | Delete | Truncate | +| -------------------- | :----: | :----: | :----: | :----: | :------: | +| Authentication Users | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage ```sql -create foreign table firebase_users ( +create foreign table firebase.users ( uid text, email text, created_at timestamp, @@ -108,18 +139,46 @@ create foreign table firebase_users ( ); ``` -Note there is a meta column `attrs` in the foreign table, which contains all the returned data from Firebase as json format. +#### Notes -### Foreign table options +- The `attrs` column contains all user attributes in JSON format +- This is a special collection with unique metadata fields -The full list of foreign table options are below: +### Firestore Database Documents -- `object` - Object name in Firebase, required. +This is an object representing Firestore Database Documents. - For Authenciation users, the object name is fixed to `auth/users`. For Firestore documents, its format is `firestore/`, note that collection id must be a full path id. For example, +Ref: [Firestore Database](https://firebase.google.com/docs/firestore) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ---------------------------- | :----: | :----: | :----: | :----: | :------: | +| Firestore Database Documents | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql +create foreign table firebase.docs ( + name text, + created_at timestamp, + updated_at timestamp, + attrs jsonb +) + server firebase_server + options ( + object 'firestore/user-profiles' + ); +``` + +#### Notes +- The `name`, `created_at`, and `updated_at` are automatic metadata fields on all Firestore collections +- Collection ID must be a full path ID in the format `firestore/` +- Examples of valid collection paths: - `firestore/my-collection` - `firestore/my-collection/my-document/another-collection` +- The `attrs` column contains all document attributes in JSON format ## Query Pushdown Support @@ -134,7 +193,7 @@ Some examples on how to use Firebase foreign tables. To map a Firestore collection provide its location using the format `firestore/` as the `object` option as shown below. ```sql -create foreign table firebase_docs ( +create foreign table firebase.docs ( name text, created_at timestamp, updated_at timestamp, @@ -153,7 +212,7 @@ Note that `name`, `created_at`, and `updated_at`, are automatic metadata fields The `auth/users` collection is a special case with unique metadata. The following shows how to map Firebase users to PostgreSQL table. ```sql -create foreign table firebase_users ( +create foreign table firebase.users ( uid text, email text, created_at timestamp, diff --git a/docs/catalog/logflare.md b/docs/catalog/logflare.md index 9a54d0e4f..60736c0af 100644 --- a/docs/catalog/logflare.md +++ b/docs/catalog/logflare.md @@ -19,13 +19,19 @@ The Logflare Wrapper allows you to read data from Logflare endpoints within your ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Logflare, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the Logflare Wrapper + +Enable the `logflare_wrapper` FDW: ```sql create foreign data wrapper logflare_wrapper @@ -33,9 +39,9 @@ create foreign data wrapper logflare_wrapper validator logflare_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) -By default, Postgres stores FDW credentials inide `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. +By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. ```sql -- Save your Logflare API key in Vault and retrieve the `key_id` @@ -71,18 +77,38 @@ We need to provide Postgres with the credentials to connect to Logflare, and any ); ``` -## Creating Foreign Tables +### Create a schema + +We recommend creating a schema to hold all the foreign tables: + +```sql +create schema if not exists logflare; +``` + +## Options + +The full list of foreign table options are below: + +- `endpoint` - Logflare endpoint UUID or name, required. + +## Entities + +### Logflare -The Logflare Wrapper supports data reads from Logflare's endpoints. +This is an object representing Logflare endpoint data. -| Integration | Select | Insert | Update | Delete | Truncate | -| ----------- | :----: | :----: | :----: | :----: | :------: | -| Logflare | ✅ | ❌ | ❌ | ❌ | ❌ | +Ref: [Logflare docs](https://logflare.app) -For example: +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| -------- | :----: | :----: | :----: | :----: | :------: | +| Logflare | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage ```sql -create foreign table my_logflare_table ( +create foreign table logflare.my_logflare_table ( id bigint, name text, _result text @@ -93,19 +119,17 @@ create foreign table my_logflare_table ( ); ``` -### Meta column - -You can define a specific meta column `_result` (data type: `text`) in the foreign table. It will store the whole result record in JSON string format, so you can extract any fields from it using Postgres JSON queries like `_result::json->>'foo'`. See more examples below. - -### Query parameters +#### Notes -Logflare endpoint query parameters can be passed using specific parameter columns like `_param_foo` and `_param_bar`. See more examples below. - -### Foreign table options - -The full list of foreign table options are below: +##### Meta Column `_result`: + - Data type must be `text` + - Stores the whole result record in JSON string format + - Use JSON queries to extract fields: `_result::json->>'field_name'` -- `endpoint` - Logflare endpoint UUID or name, required. +##### Query Parameters: + - Use parameter columns with prefix `_param_` + - Example: `_param_org_id`, `_param_iso_timestamp_start` + - Parameters are passed to the Logflare endpoint ## Query Pushdown Support @@ -113,12 +137,9 @@ This FDW doesn't support query pushdown. ## Examples -Some examples on how to use Logflare foreign tables. - -### Basic example - -Assume the Logflare endpoint response is like below: +### Basic Example +Given a Logflare endpoint response: ```json [ { @@ -128,10 +149,10 @@ Assume the Logflare endpoint response is like below: ] ``` -Then we can define a foreign table like this: +You can create and query a foreign table: ```sql -create foreign table people ( +create foreign table logflare.people ( id bigint, name text, _result text @@ -141,19 +162,18 @@ create foreign table people ( endpoint '9dd9a6f6-8e9b-4fa4-b682-4f2f5cd99da3' ); -select * from people; +select * from logflare.people; ``` -### Query parameters example - -Suppose the Logflare endpoint accepts 3 parameters: +### Query Parameters Example -1. org_id -2. iso_timestamp_start -3. iso_timestamp_end +For an endpoint accepting parameters: -And its response is like below: +- org_id +- iso_timestamp_start +- iso_timestamp_end +With response format: ```json [ { @@ -165,10 +185,10 @@ And its response is like below: ] ``` -We can define a foreign table and parameter columns like this: +Create and query the table with parameters: ```sql -create foreign table runtime_hours ( +create foreign table logflare.runtime_hours ( db_size text, org_id text, runtime_hours numeric, @@ -182,18 +202,14 @@ create foreign table runtime_hours ( options ( endpoint 'my.custom.endpoint' ); -``` - -and query it with parameters like this: -```sql select db_size, org_id, runtime_hours, runtime_minutes from - runtime_hours + logflare.runtime_hours where _param_org_id = 123 and _param_iso_timestamp_start = '2023-07-01 02:03:04' and _param_iso_timestamp_end = '2023-07-02'; diff --git a/docs/catalog/mssql.md b/docs/catalog/mssql.md index fff27a435..0dfdd087a 100644 --- a/docs/catalog/mssql.md +++ b/docs/catalog/mssql.md @@ -36,13 +36,19 @@ The SQL Server Wrapper allows you to read data from Microsoft SQL Server within ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query SQL Server, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the SQL Server Wrapper + +Enable the `mssql_wrapper` FDW: ```sql create foreign data wrapper mssql_wrapper @@ -50,7 +56,7 @@ create foreign data wrapper mssql_wrapper validator mssql_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. @@ -64,6 +70,23 @@ values ( returning key_id; ``` +The connection string is an [ADO.NET connection string](https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/connection-strings), which specifies connection parameters in semicolon-delimited string. + +**Supported parameters** + +All parameter keys are handled case-insensitive. + +| Parameter | Allowed Values | Description | +| ---------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------- | +| Server | `` | The name or network address of the instance of SQL Server to which to connect. Format: `host,port` | +| User | `` | The SQL Server login account. | +| Password | `` | The password for the SQL Server account logging on. | +| Database | `` | The name of the database. | +| IntegratedSecurity | false | Windows/Kerberos authentication and SQL authentication. | +| TrustServerCertificate | true, false | Specifies whether the driver trusts the server certificate when connecting using TLS. | +| Encrypt | true, false, DANGER_PLAINTEXT | Specifies whether the driver uses TLS to encrypt communication. | +| ApplicationName | `` | Sets the application name for the connection. | + ### Connecting to SQL Server We need to provide Postgres with the credentials to connect to SQL Server. We can do this using the `create server` command: @@ -88,35 +111,44 @@ We need to provide Postgres with the credentials to connect to SQL Server. We ca ); ``` -The connection string is an [ADO.NET connection string](https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/connection-strings), which specifies connection parameters in semicolon-delimited string. +### Create a schema -**Supported parameters** +We recommend creating a schema to hold all the foreign tables: -All parameter keys are handled case-insensitive. +```sql +create schema if not exists mssql; +``` -| Parameter | Allowed Values | Description | -| ---------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------- | -| Server | `` | The name or network address of the instance of SQL Server to which to connect. Format: `host,port` | -| User | `` | The SQL Server login account. | -| Password | `` | The password for the SQL Server account logging on. | -| Database | `` | The name of the database. | -| IntegratedSecurity | false | Windows/Kerberos authentication and SQL authentication. | -| TrustServerCertificate | true, false | Specifies whether the driver trusts the server certificate when connecting using TLS. | -| Encrypt | true, false, DANGER_PLAINTEXT | Specifies whether the driver uses TLS to encrypt communication. | -| ApplicationName | `` | Sets the application name for the connection. | +## Options + +The full list of foreign table options are below: + +- `table` - Source table or view name in SQL Server, required. -## Creating Foreign Tables +This can also be a subquery enclosed in parentheses, for example, -The SQL Server Wrapper supports data reads from SQL Server. +```sql +table '(select * from users where id = 42 or id = 43)' +``` + +## Entities + +### SQL Server Tables + +This is an object representing SQL Server tables and views. + +Ref: [Microsoft SQL Server docs](https://www.microsoft.com/en-au/sql-server/) + +#### Operations -| Integration | Select | Insert | Update | Delete | Truncate | -| ----------- | :----: | :----: | :----: | :----: | :------: | -| SQL Server | ✅ | ❌ | ❌ | ❌ | ❌ | +| Object | Select | Insert | Update | Delete | Truncate | +| ---------- | :----: | :----: | :----: | :----: | :------: | +| table/view | ✅ | ❌ | ❌ | ❌ | ❌ | -For example: +#### Usage ```sql -create foreign table mssql_users ( +create foreign table mssql.users ( id bigint, name text, dt timestamp @@ -127,17 +159,15 @@ create foreign table mssql_users ( ); ``` -### Foreign table options +#### Notes -The full list of foreign table options are below: - -- `table` - Source table or view name in SQL Server, required. - - This can also be a subquery enclosed in parentheses, for example, - - ```sql - table '(select * from users where id = 42 or id = 43)' - ``` +- Supports both tables and views as data sources +- Can use subqueries in the `table` option +- Query pushdown supported for: + - `where` clauses + - `order by` clauses + - `limit` clauses +- See Data Types section for type mappings between PostgreSQL and SQL Server ## Query Pushdown Support @@ -145,9 +175,9 @@ This FDW supports `where`, `order by` and `limit` clause pushdown. ## Examples -Some examples on how to use SQL Server foreign tables. +### Basic Example -Let's prepare the source table in SQL Server first: +First, create a source table in SQL Server: ```sql -- Run below SQLs on SQL Server to create source table @@ -163,12 +193,10 @@ insert into users(id, name, dt) values (43, 'Bar', '2023-12-27'); insert into users(id, name, dt) values (44, 'Baz', '2023-12-26'); ``` -### Basic example - -This example will create a foreign table inside your Postgres database and query its data: +Then create and query the foreign table in PostgreSQL: ```sql -create foreign table mssql_users ( +create foreign table mssql.users ( id bigint, name text, dt timestamp @@ -178,15 +206,15 @@ create foreign table mssql_users ( table 'users' ); -select * from mssql_users; +select * from mssql.users; ``` -### Remote subquery example +### Remote Subquery Example -This example will create a foreign table based on a remote subquery and query its data: +Create a foreign table using a subquery: ```sql -create foreign table mssql_users_subquery ( +create foreign table mssql.users_subquery ( id bigint, name text, dt timestamp @@ -196,5 +224,5 @@ create foreign table mssql_users_subquery ( table '(select * from users where id = 42 or id = 43)' ); -select * from mssql_users_subquery; +select * from mssql.users_subquery; ``` diff --git a/docs/catalog/notion.md b/docs/catalog/notion.md index 5956b0438..2918c7be5 100644 --- a/docs/catalog/notion.md +++ b/docs/catalog/notion.md @@ -37,13 +37,19 @@ The Notion API uses JSON formatted data, please refer to [Notion API docs](https ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Notion, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the Wasm foreign data wrapper: +### Enable the Notion Wrapper + +Enable the Wasm foreign data wrapper: ```sql create foreign data wrapper wasm_wrapper @@ -51,7 +57,7 @@ create foreign data wrapper wasm_wrapper validator wasm_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. @@ -60,75 +66,52 @@ By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server insert into vault.secrets (name, secret) values ( 'notion', - '' -- Notion API key + '' -- Notion API key ) returning key_id; ``` -### Connecting to Notion +### Create a schema -We need to provide Postgres with the credentials to access Notion, and any additional options. We can do this using the `create server` command: +We recommend creating a schema to hold all the foreign tables: -=== "With Vault" +```sql +create schema if not exists notion; +``` - ```sql - create server notion_server - foreign data wrapper wasm_wrapper - options ( - fdw_package_url 'https://github.com/supabase/wrappers/releases/download/wasm_notion_fdw_v0.1.0/notion_fdw.wasm', - fdw_package_name 'supabase:notion-fdw', - fdw_package_version '0.1.0', - fdw_package_checksum 'e017263d1fc3427cc1df8071d1182cdc9e2f00363344dddb8c195c5d398a2099', - api_url 'https://api.notion.com/v1', -- optional - api_version '2022-06-28', -- optional - api_key_id '' -- The Key ID from above. - ); - ``` +## Options -=== "Without Vault" +The full list of foreign table options are below: - ```sql - create server notion_server - foreign data wrapper wasm_wrapper - options ( - fdw_package_url 'https://github.com/supabase/wrappers/releases/download/wasm_notion_fdw_v0.1.0/notion_fdw.wasm', - fdw_package_name 'supabase:notion-fdw', - fdw_package_version '0.1.0', - fdw_package_checksum 'e017263d1fc3427cc1df8071d1182cdc9e2f00363344dddb8c195c5d398a2099', - api_url 'https://api.notion.com/v1', -- optional - api_version '2022-06-28', -- optional - api_key 'secret_xxxxx' -- Notion API key - ); - ``` +- `object` - Object name in Notion, required. -Note the `fdw_package_*` options are required, which specify the Wasm package metadata. You can get the available package version list from [above](#available-versions). +Supported objects are listed below: -### Create a schema +| Object name | +| ----------- | +| block | +| page | +| database | +| user | -We recommend creating a schema to hold all the foreign tables: -```sql -create schema if not exists notion; -``` +## Entities + +### Block + +This is an object representing Notion Block content. -## Creating Foreign Tables +Ref: [Notion API docs](https://developers.notion.com/reference/intro) -The Notion Wrapper supports data reads from below objects in Notion. +#### Operations -| Integration | Select | Insert | Update | Delete | Truncate | -| ----------- | :----: | :----: | :----: | :----: | :------: | -| Block | ✅ | ❌ | ❌ | ❌ | ❌ | -| Page | ✅ | ❌ | ❌ | ❌ | ❌ | -| Database | ✅ | ❌ | ❌ | ❌ | ❌ | -| User | ✅ | ❌ | ❌ | ❌ | ❌ | +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Block | ✅ | ❌ | ❌ | ❌ | ❌ | -For example: +#### Usage ```sql --- Note: the 'page_id' isn't presented in the Notion Block's fields, it is --- added by the foreign data wrapper for development convenience. All blocks, --- including nested children blocks, belong to one page will have a same --- 'page_id' of that page. create foreign table notion.blocks ( id text, page_id text, @@ -142,7 +125,32 @@ create foreign table notion.blocks ( options ( object 'block' ); +``` + +#### Notes + +- The `attrs` column contains all user attributes in JSON format +- The `page_id` field is added by the FDW for development convenience +- All blocks, including nested children blocks, belong to one page will have the same `page_id` +- Query pushdown supported for both `id` and `page_id` columns +- Use `page_id` filter to fetch all blocks of a specific page recursively +- Querying all blocks without filters may take a long time due to recursive data requests + +### Page +This is an object representing Notion Pages. + +Ref: [Notion API docs](https://developers.notion.com/reference/intro) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Page | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table notion.pages ( id text, url text, @@ -155,7 +163,28 @@ create foreign table notion.pages ( options ( object 'page' ); +``` + +#### Notes +- The `attrs` column contains all page attributes in JSON format +- Query pushdown supported for `id` column + +### Database + +This is an object representing Notion Databases. + +Ref: [Notion API docs](https://developers.notion.com/reference/intro) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| -------- | :----: | :----: | :----: | :----: | :------: | +| Database | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql create foreign table notion.databases ( id text, url text, @@ -168,7 +197,28 @@ create foreign table notion.databases ( options ( object 'database' ); +``` + +#### Notes + +- The `attrs` column contains all database attributes in JSON format +- Query pushdown supported for `id` column + +### User + +This is an object representing Notion Users. + +Ref: [Notion API docs](https://developers.notion.com/reference/intro) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| User | ✅ | ❌ | ❌ | ❌ | ❌ | +#### Usage + +```sql create foreign table notion.users ( id text, name text, @@ -182,25 +232,11 @@ create foreign table notion.users ( ); ``` -!!! note - - - All the supported columns are listed above, other columns are not allowd. - - The `attrs` is a special column which stores all the object attributes in JSON format, you can extract any attributes needed from it. See more examples below. - -### Foreign table options - -The full list of foreign table options are below: - -- `object` - Object name in Notion, required. - - Supported objects are listed below: +#### Notes - | Object name | - | --------------------- | - | block | - | page | - | database | - | user | +- The `attrs` column contains all user attributes in JSON format +- Query pushdown supported for `id` column +- User email can be extracted using: `attrs->'person'->>'email'` ## Query Pushdown Support @@ -211,7 +247,7 @@ select * from notion.pages where id = '5a67c86f-d0da-4d0a-9dd7-f4cf164e6247'; ``` -will be translated Notion API call: `https://api.notion.com/v1/pages/5a67c86f-d0da-4d0a-9dd7-f4cf164e6247`. +will be translated to a Notion API call: `https://api.notion.com/v1/pages/5a67c86f-d0da-4d0a-9dd7-f4cf164e6247`. In addition to `id` column pushdown, `page_id` column pushdown is also supported for `Block` object. For example, @@ -220,7 +256,7 @@ select * from notion.blocks where page_id = '5a67c86f-d0da-4d0a-9dd7-f4cf164e6247'; ``` -will recursively fetch all children blocks of the Page with id '5a67c86f-d0da-4d0a-9dd7-f4cf164e6247'. This can dramatically reduce number of API calls and improve query speed. +will recursively fetch all children blocks of the Page with id '5a67c86f-d0da-4d0a-9dd7-f4cf164e6247'. This can dramatically reduce number of API calls and improve query performance. !!! note @@ -232,17 +268,9 @@ will recursively fetch all children blocks of the Page with id '5a67c86f-d0da-4d ## Examples -Below are some examples on how to use Notion foreign tables. - -### Basic example - -This example will create a "foreign table" inside your Postgres database and query its data. First, we can create a schema to hold all the Notion foreign tables. - -```sql -create schema if not exists notion; -``` +### Basic Example -Then create the foreign table and query it, for example: +This example will create a "foreign table" inside your Postgres database and query its data. ```sql create foreign table notion.pages ( @@ -268,7 +296,7 @@ where id = '5a67c86f-d0da-4d0a-9dd7-f4cf164e6247'; `attrs` is a special column which stores all the object attributes in JSON format, you can extract any attributes needed from it. See more examples below. -### Query JSON attributes +### Query JSON Attributes ```sql create foreign table notion.users ( diff --git a/docs/catalog/paddle.md b/docs/catalog/paddle.md index 93aa83658..80002fab3 100644 --- a/docs/catalog/paddle.md +++ b/docs/catalog/paddle.md @@ -44,13 +44,19 @@ The Paddle API uses JSON formatted data, please refer to [Paddle docs](https://d ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Paddle, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the Wasm foreign data wrapper: +### Enable the Paddle Wrapper + +Enable the Wasm foreign data wrapper: ```sql create foreign data wrapper wasm_wrapper @@ -58,7 +64,7 @@ create foreign data wrapper wasm_wrapper validator wasm_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. @@ -67,7 +73,7 @@ By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server insert into vault.secrets (name, secret) values ( 'paddle', - 'bb4e69088ea07a98a90565ac610c63654423f8f1e2d48b39b5' + '' -- Paddle API key ) returning key_id; ``` @@ -116,15 +122,80 @@ We recommend creating a schema to hold all the foreign tables: create schema if not exists paddle; ``` -## Creating Foreign Tables +## Options -The Paddle Wrapper supports data reads and writes from Paddle. +The full list of foreign table options are below: -| Integration | Select | Insert | Update | Delete | Truncate | -| ----------- | :----: | :----: | :----: | :----: | :------: | -| Paddle | ✅ | ✅ | ✅ | ❌ | ❌ | +- `object` - Object name in Paddle, required. + +Supported objects are listed below: + +| Object | +| --------------------- | +| products | +| prices | +| discounts | +| customers | +| transactions | +| reports | +| notification-settings | +| notifications | + +- `rowid_column` - Primary key column name, optional for data scan, required for data modify -For example: +## Entities + +### Products + +This is an object representing Paddle Products. + +Ref: [Paddle API docs](https://developer.paddle.com/api-reference/about/data-types) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| -------- | :----: | :----: | :----: | :----: | :------: | +| Products | ✅ | ✅ | ✅ | ❌ | ❌ | + +#### Usage + +```sql +create foreign table paddle.products ( + id text, + name text, + tax_category text, + status text, + description text, + created_at timestamp, + updated_at timestamp, + attrs jsonb +) + server paddle_server + options ( + object 'products', + rowid_column 'id' + ); +``` + +#### Notes + +- Requires `rowid_column` option for data modification operations +- Query pushdown supported for `id` column +- Product type can be extracted using: `attrs->>'type'` + +### Customers + +This is an object representing Paddle Customers. + +Ref: [Paddle API docs](https://developer.paddle.com/api-reference/about/data-types) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| --------- | :----: | :----: | :----: | :----: | :------: | +| Customers | ✅ | ✅ | ✅ | ❌ | ❌ | + +#### Usage ```sql create foreign table paddle.customers ( @@ -144,48 +215,60 @@ create foreign table paddle.customers ( ); ``` -### Foreign table options +#### Notes -The full list of foreign table options are below: +- Requires `rowid_column` option for data modification operations +- Query pushdown supported for `id` column +- Custom data stored in dedicated `custom_data` column -- `object` - Object name in Paddle, required. +### Subscriptions - Supported objects are listed below: +This is an object representing Paddle Subscriptions. - | Object name | - | --------------------- | - | products | - | prices | - | discounts | - | customers | - | transactions | - | reports | - | notification-settings | - | notifications | +Ref: [Paddle API docs](https://developer.paddle.com/api-reference/about/data-types) -- `rowid_column` - Primary key column name, optional for data scan, required for data modify +#### Operations -## Query Pushdown Support +| Object | Select | Insert | Update | Delete | Truncate | +| ------------- | :----: | :----: | :----: | :----: | :------: | +| Subscriptions | ✅ | ✅ | ✅ | ❌ | ❌ | -This FDW supports `where` clause pushdown with `id` as the filter. For example, +#### Usage ```sql -select * from paddle.customers where id = 'ctm_01hymwgpkx639a6mkvg99563sp'; +create foreign table paddle.subscriptions ( + id text, + status text, + created_at timestamp, + updated_at timestamp, + attrs jsonb +) + server paddle_server + options ( + object 'subscriptions', + rowid_column 'id' + ); ``` -## Examples +#### Notes -Below are some examples on how to use Paddle foreign tables. +- Requires `rowid_column` option for data modification operations +- Query pushdown supported for `id` column +- Subscription items status can be extracted using: `attrs#>'{items,status}'` -### Basic example +## Query Pushdown Support -This example will create a "foreign table" inside your Postgres database and query its data. First, we can create a schema to hold all the Paddle foreign tables. +This FDW supports `where` clause pushdown with `id` as the filter. For example, ```sql -create schema if not exists paddle; +select * from paddle.customers where id = 'ctm_01hymwgpkx639a6mkvg99563sp'; ``` -Then create the foreign table and query it, for example: +## Examples + +### Basic Example + +This example will create a "foreign table" inside your Postgres database and query its data. ```sql create foreign table paddle.customers ( @@ -209,7 +292,7 @@ select * from paddle.customers; `attrs` is a special column which stores all the object attributes in JSON format, you can extract any attributes needed or its associated sub objects from it. See more examples below. -### Query JSON attributes +### Query JSON Attributes ```sql create foreign table paddle.products ( @@ -250,13 +333,13 @@ select id, attrs#>'{items,status}' as item_status from paddle.subscriptions where id = 'sub_01hv959anj4zrw503h2acawb3p'; ``` -### Data modify example +### Data Modify Example This example will modify data in a "foreign table" inside your Postgres database, note that `rowid_column` option is mandatory for data modify: ```sql -- insert new data -insert into paddle.products (name, tax_category) +insert into paddle.products(name, tax_category) values ('my prod', 'standard'); -- update existing data diff --git a/docs/catalog/redis.md b/docs/catalog/redis.md index ec77b8d51..81f5b351d 100644 --- a/docs/catalog/redis.md +++ b/docs/catalog/redis.md @@ -37,13 +37,19 @@ All Redis values will be stored as `text` or `jsonb` columns in Postgres, below ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Redis, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the Redis Wrapper + +Enable the `redis_wrapper` FDW: ```sql create foreign data wrapper redis_wrapper @@ -51,7 +57,7 @@ create foreign data wrapper redis_wrapper validator redis_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. @@ -89,95 +95,239 @@ We need to provide Postgres with the credentials to connect to Redis. We can do ); ``` -The connection URL format is: +### Create a schema -``` -redis://[][:@][:port][/] +We recommend creating a schema to hold all the foreign tables: + +```sql +create schema if not exists redis; ``` -The connection URL also supports TLS: +## Options +The following options are available when creating Redis foreign tables: + +- `src_type` - Foreign table source type in Redis, required. + +This can be one of below types, + +| Source type | Description | +| ----------- | ------------------------------------------------------------------ | +| list | [Single list](https://redis.io/docs/data-types/lists/) | +| set | [Single set](https://redis.io/docs/data-types/sets/) | +| hash | [Single hash](https://redis.io/docs/data-types/hashes/) | +| zset | [Single sorted set](https://redis.io/docs/data-types/sorted-sets/) | +| stream | [Stream](https://redis.io/docs/data-types/streams/) | +| multi_list | Multiple lists, specified by `src_key` pattern | +| multi_set | Multiple sets, specified by `src_key` pattern | +| multi_hash | Multiple hashes, specified by `src_key` pattern | +| multi_zset | Multiple sorted sets, specified by `src_key` pattern | + +- `src_key` - Source object key in Redis, required. + +This key can be a pattern for `multi_*` type of foreign table. For other types, this key must return exact one value. For example, + +| Source Type | `src_key` examples | +| --------------------------------------------- | ------------------------------------------------------- | +| list, set, hash, zset, stream | `my_list`, `list:001`, `hash_foo`, `zset:1000` and etc. | +| multi_list, multi_set, multi_hash, multi_zset | `my_list:*`, `set:*`, `zset:*` and etc. | + + +## Entities + +### List + +This is an object representing a Redis List. + +Ref: [Redis docs](https://redis.io/docs/data-types/lists/) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| List | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql +create foreign table redis.list ( + element text +) + server redis_server + options ( + src_type 'list', + src_key 'my_list' + ); ``` -rediss://[][:@][:port][/] -``` -To enable insecure mode, append `#insecure` at the end of the URL: +#### Notes + +- Elements are stored in insertion order +- Query returns all elements in the list +- No query pushdown support + +### Set + +This is an object representing a Redis Set. +Ref: [Redis docs](https://redis.io/docs/data-types/sets/) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Set | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql +create foreign table redis.set ( + element text +) + server redis_server + options ( + src_type 'set', + src_key 'set' + ); ``` -rediss://[][:@][:port]/[]#insecure + +#### Notes + +- Elements are unique within the set +- No guaranteed order of elements +- No query pushdown support + +### Hash + +This is an object representing a Redis Hash. + +Ref: [Redis docs](https://redis.io/docs/data-types/hashes/) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Hash | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql +create foreign table redis.hash ( + key text, + value text +) + server redis_server + options ( + src_type 'hash', + src_key 'hash' + ); ``` -!!! note +#### Notes + +- Key-value pairs within the hash +- No query pushdown support +- Both key and value are returned as text - Client certificate and custom root certificates are not supported when using TLS. +### Sorted Set -## Creating Foreign Tables +This is an object representing a Redis Sorted Set. -The Redis Wrapper supports data reads from Redis. +Ref: [Redis docs](https://redis.io/docs/data-types/sorted-sets/) -| Integration | Select | Insert | Update | Delete | Truncate | -| ----------- | :----: | :----: | :----: | :----: | :------: | -| Redis | ✅ | ❌ | ❌ | ❌ | ❌ | +#### Operations -For example: +| Object | Select | Insert | Update | Delete | Truncate | +| ---------- | :----: | :----: | :----: | :----: | :------: | +| Sorted Set | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage ```sql -create foreign table redis_list ( +create foreign table redis.zset ( element text ) server redis_server options ( - src_type 'list', - src_key 'my_list' + src_type 'zset', + src_key 'zset' ); ``` -The foreign table columns names and types must be fixed for each source type, as listed below: +#### Notes -| src_type | Column name | Column type | -| --------------- | ----------- | ----------- | -| list, set, zset | `element` | text | -| hash | `key` | text | -| | `value` | text | -| stream | `id` | text | -| | `items` | jsonb | -| multi\_\* | `key` | text | -| | `items` | jsonb | +- Elements are ordered by their score +- Elements are unique within the set +- Score information is not exposed in the foreign table -**See below for the full list of `src_type` and descriptions.** +### Stream -### Foreign table options +This is an object representing a Redis Stream. -The full list of foreign table options are below: +Ref: [Redis docs](https://redis.io/docs/data-types/streams/) -- `src_type` - Foreign table source type in Redis, required. +#### Operations - This can be one of below types, +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| Stream | ✅ | ❌ | ❌ | ❌ | ❌ | - | Source type | Description | - | ----------- | ------------------------------------------------------------------ | - | list | [Single list](https://redis.io/docs/data-types/lists/) | - | set | [Single set](https://redis.io/docs/data-types/sets/) | - | hash | [Single hash](https://redis.io/docs/data-types/hashes/) | - | zset | [Single sorted set](https://redis.io/docs/data-types/sorted-sets/) | - | stream | [Stream](https://redis.io/docs/data-types/streams/) | - | multi_list | Multiple lists, specified by `src_key` pattern | - | multi_set | Multiple sets, specified by `src_key` pattern | - | multi_hash | Multiple hashes, specified by `src_key` pattern | - | multi_zset | Multiple sorted sets, specified by `src_key` pattern | +#### Usage -- `src_key` - Source object key in Redis, required. +```sql +create foreign table redis.stream ( + id text, + items jsonb +) + server redis_server + options ( + src_type 'stream', + src_key 'stream' + ); +``` + +#### Notes + +- Stream entries have unique IDs +- Items are stored in JSONB format +- Entries are ordered by their IDs + +### Multiple Objects + +Redis wrapper supports querying multiple objects of the same type using pattern matching. + +#### Operations + +| Object Type | Select | Insert | Update | Delete | Truncate | +| ------------- | :----: | :----: | :----: | :----: | :------: | +| Multiple List | ✅ | ❌ | ❌ | ❌ | ❌ | +| Multiple Set | ✅ | ❌ | ❌ | ❌ | ❌ | +| Multiple Hash | ✅ | ❌ | ❌ | ❌ | ❌ | +| Multiple ZSet | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql +create foreign table redis.multi_lists ( + key text, + items jsonb +) + server redis_server + options ( + src_type 'multi_list', + src_key 'list:*' + ); +``` - This key can be a pattern for `multi_*` type of foreign table. For other types, this key must return exact one value. For example, +#### Notes - | Source Type | `src_key` examples | - | --------------------------------------------- | ------------------------------------------------------- | - | list, set, hash, zset, stream | `my_list`, `list:001`, `hash_foo`, `zset:1000` and etc. | - | multi_list, multi_set, multi_hash, multi_zset | `my_list:*`, `set:*`, `zset:*` and etc. | +- Use pattern matching in `src_key` option +- Results include object key and items in JSONB format +- Items format varies by object type ## Query Pushdown Support -This FDW doesn't supports pushdown. +This FDW doesn't support pushdown. ## Examples @@ -213,7 +363,7 @@ This example will create foreign tables inside your Postgres database and query - List ```sql - create foreign table redis_list ( + create foreign table redis.list ( element text ) server redis_server @@ -222,7 +372,7 @@ This example will create foreign tables inside your Postgres database and query src_key 'list' ); - select * from redis_list; + select * from redis.list; ``` Query result: @@ -239,7 +389,7 @@ This example will create foreign tables inside your Postgres database and query - Set ```sql - create foreign table redis_set ( + create foreign table redis.set ( element text ) server redis_server @@ -248,7 +398,7 @@ This example will create foreign tables inside your Postgres database and query src_key 'set' ); - select * from redis_set; + select * from redis.set; ``` Query result: @@ -265,7 +415,7 @@ This example will create foreign tables inside your Postgres database and query - Hash ```sql - create foreign table redis_hash ( + create foreign table redis.hash ( key text, value text ) @@ -275,7 +425,7 @@ This example will create foreign tables inside your Postgres database and query src_key 'hash' ); - select * from redis_hash; + select * from redis.hash; ``` Query result: @@ -291,7 +441,7 @@ This example will create foreign tables inside your Postgres database and query - Sorted set ```sql - create foreign table redis_zset ( + create foreign table redis.zset ( element text ) server redis_server @@ -300,7 +450,7 @@ This example will create foreign tables inside your Postgres database and query src_key 'zset' ); - select * from redis_zset; + select * from redis.zset; ``` Query result: @@ -317,7 +467,7 @@ This example will create foreign tables inside your Postgres database and query - Stream ```sql - create foreign table redis_stream ( + create foreign table redis.stream ( id text, items jsonb ) @@ -327,7 +477,7 @@ This example will create foreign tables inside your Postgres database and query src_key 'stream' ); - select * from redis_stream; + select * from redis.stream; ``` Query result: @@ -347,7 +497,7 @@ This example will create several foreign tables using pattern in key and query m - List ```sql - create foreign table redis_multi_lists ( + create foreign table redis.multi_lists ( key text, items jsonb ) @@ -357,7 +507,7 @@ This example will create several foreign tables using pattern in key and query m src_key 'list:*' ); - select * from redis_multi_lists; + select * from redis.multi_lists; ``` Query result: @@ -373,7 +523,7 @@ This example will create several foreign tables using pattern in key and query m - Set ```sql - create foreign table redis_multi_sets ( + create foreign table redis.multi_sets ( key text, items jsonb ) @@ -383,7 +533,7 @@ This example will create several foreign tables using pattern in key and query m src_key 'set:*' ); - select * from redis_multi_sets; + select * from redis.multi_sets; ``` Query result: @@ -399,7 +549,7 @@ This example will create several foreign tables using pattern in key and query m - Hash ```sql - create foreign table redis_multi_hashes ( + create foreign table redis.multi_hashes ( key text, items jsonb ) @@ -409,7 +559,7 @@ This example will create several foreign tables using pattern in key and query m src_key 'hash:*' ); - select * from redis_multi_hashes; + select * from redis.multi_hashes; ``` Query result: @@ -425,7 +575,7 @@ This example will create several foreign tables using pattern in key and query m - Sorted set ```sql - create foreign table redis_multi_zsets ( + create foreign table redis.multi_zsets ( key text, items jsonb ) @@ -435,7 +585,7 @@ This example will create several foreign tables using pattern in key and query m src_key 'zset:*' ); - select * from redis_multi_zsets; + select * from redis.multi_zsets; ``` Query result: diff --git a/docs/catalog/s3.md b/docs/catalog/s3.md index e8a927c51..585e188e3 100644 --- a/docs/catalog/s3.md +++ b/docs/catalog/s3.md @@ -53,13 +53,19 @@ The S3 Wrapper uses Parquet file data types from [arrow_array::types](https://do ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query S3, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the foreign data wrapper: +### Enable the S3 Wrapper + +Enable the `s3_wrapper` FDW: ```sql create foreign data wrapper s3_wrapper @@ -67,9 +73,9 @@ create foreign data wrapper s3_wrapper validator s3_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) -By default, Postgres stores FDW credentials inide `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. +By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. ```sql -- Save your AWS credential in Vault and retrieve the `key_id` @@ -124,7 +130,7 @@ The full list of options are below: - `endpoint_url` (optional) - An optional URL to allow connection to S3-compliant providers (i.e. Wasabi, Cloudflare R2, Backblaze B2, DigitalOcean Spaces) - `path_style_url` (optional) - Whether to use [path-style URL](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access) access. This is required by some S3-compliant providers. `true` or `false`, default is `false`. -### Required S3 permissions +#### Required S3 permissions Below S3 permissions are needed: @@ -163,18 +169,41 @@ create server s3_server ); ``` -## Creating Foreign Tables +### Create a schema + +We recommend creating a schema to hold all the foreign tables: + +```sql +create schema if not exists s3; +``` + +## Options + +The following options are available when creating S3 foreign tables: + +- `uri` - S3 URI, required. For example, `s3://bucket/s3_table.csv` +- `format` - File format, required. `csv`, `jsonl`, or `parquet` +- `has_header` - If the CSV file has header, optional. `true` or `false`, default is `false` +- `compress` - Compression algorithm, optional. One of `gzip`, `bzip2`, `xz`, `zlib`, default is no compression + +## Entities + +### CSV Files + +This is an object representing CSV files in S3. -The S3 Wrapper supports data reads from S3. +Ref: [AWS S3 docs](https://aws.amazon.com/s3/) -| Integration | Select | Insert | Update | Delete | Truncate | -| ----------- | :----: | :----: | :----: | :----: | :------: | -| S3 | ✅ | ❌ | ❌ | ❌ | ❌ | +#### Operations -For example: +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| CSV | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage ```sql -create foreign table s3_table_csv ( +create foreign table s3.table_csv ( name text, sex text, age text, @@ -189,18 +218,86 @@ create foreign table s3_table_csv ( ); ``` -One file in S3 corresponds a foreign table in Postgres. For CSV and JSONL file, all columns must be present in the foreign table and type must be `text`. You can do custom transformations, like type conversion, by creating a view on top of the foreign table or using a subquery. +#### Notes -For Parquet file, no need to define all columns in the foreign table but column names must match between Parquet file and its foreign table. +- All columns must be defined in the foreign table +- All column types must be `text` +- Optional header support via `has_header` option +- Supports compression (gzip, bzip2, xz, zlib) -### Foreign table options +### JSON Lines Files -The full list of foreign table options are below: +This is an object representing JSONL files in S3. -- `uri` - S3 URI, required. For example, `s3://bucket/s3_table.csv` -- `format` - File format, required. `csv`, `jsonl`, or `parquet` -- `has_header` - If the CSV file has header, optional. `true` or `false`, default is `false` -- `compress` - Compression algorithm, optional. One of `gzip`, `bzip2`, `xz`, `zlib`, default is no compression +Ref: [JSONL docs](https://jsonlines.org/) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------ | :----: | :----: | :----: | :----: | :------: | +| JSONL | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql +create foreign table s3.table_jsonl ( + name text, + sex text, + age text, + height text, + weight text +) + server s3_server + options ( + uri 's3://bucket/s3_table.jsonl', + format 'jsonl' + ); +``` + +#### Notes + +- All columns must be defined in the foreign table +- All column types must be `text` +- Each line must be a valid JSON object +- Supports compression (gzip, bzip2, xz, zlib) + +### Parquet Files + +This is an object representing Parquet files in S3. + +Ref: [Parquet docs](https://parquet.apache.org/) + +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------- | :----: | :----: | :----: | :----: | :------: | +| Parquet | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + +```sql +create foreign table s3.table_parquet ( + id integer, + bool_col boolean, + bigint_col bigint, + float_col real, + date_string_col text, + timestamp_col timestamp +) + server s3_server + options ( + uri 's3://bucket/s3_table.parquet', + format 'parquet' + ); +``` + +#### Notes + +- Not all columns need to be defined in foreign table +- Column names must match between Parquet file and foreign table +- Supports various Postgres data types (see Data Types section) +- Compressed files are loaded entirely into memory +- Supports compression (gzip, bzip2, xz, zlib) ## Query Pushdown Support @@ -208,15 +305,13 @@ This FDW doesn't support query pushdown. ## Examples -Some examples on how to use S3 foreign tables. - -### Basic example +### Basic Example This will create some "foreign table" inside your Postgres database can read data from S3: ```sql -- CSV file, no compression -create foreign table s3_table_csv ( +create foreign table s3.table_csv ( name text, sex text, age text, @@ -231,7 +326,7 @@ create foreign table s3_table_csv ( ); -- JSON line file, no compression -create foreign table s3_table_jsonl ( +create foreign table s3.table_jsonl ( name text, sex text, age text, @@ -245,7 +340,7 @@ create foreign table s3_table_jsonl ( ); -- GZIP compressed CSV file -create foreign table s3_table_csv_gzip ( +create foreign table s3.table_csv_gzip ( name text, sex text, age text, @@ -261,7 +356,7 @@ create foreign table s3_table_csv_gzip ( ); -- Parquet file, no compression -create foreign table s3_table_parquet ( +create foreign table s3.table_parquet ( id integer, bool_col boolean, bigint_col bigint, @@ -276,7 +371,7 @@ create foreign table s3_table_parquet ( ); -- GZIP compressed Parquet file -create foreign table s3_table_parquet_gz ( +create foreign table s3.table_parquet_gz ( id integer, bool_col boolean, bigint_col bigint, @@ -307,7 +402,7 @@ create server s3_server path_style_url 'true' ); -create foreign table s3_table_csv ( +create foreign table s3.supabase_table_csv ( id text, name text ) diff --git a/docs/catalog/snowflake.md b/docs/catalog/snowflake.md index 1f55e88f6..e441a31fd 100644 --- a/docs/catalog/snowflake.md +++ b/docs/catalog/snowflake.md @@ -42,13 +42,19 @@ The Snowflake Wrapper is a WebAssembly(Wasm) foreign data wrapper which allows y ## Preparation -Before you get started, make sure the `wrappers` extension is installed on your database: +Before you can query Snowflake, you need to enable the Wrappers extension and store your credentials in Postgres. + +### Enable Wrappers + +Make sure the `wrappers` extension is installed on your database: ```sql create extension if not exists wrappers with schema extensions; ``` -and then create the Wasm foreign data wrapper: +### Enable the Snowflake Wrapper + +Enable the Wasm foreign data wrapper: ```sql create foreign data wrapper wasm_wrapper @@ -56,7 +62,7 @@ create foreign data wrapper wasm_wrapper validator wasm_fdw_validator; ``` -### Secure your credentials (optional) +### Store your credentials (optional) By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. @@ -120,15 +126,31 @@ We recommend creating a schema to hold all the foreign tables: create schema if not exists snowflake; ``` -## Creating Foreign Tables +## Options + +The full list of foreign table options are below: + +- `table` - Source table or view name in Snowflake, required. + + This option can also be a subquery enclosed in parentheses. + +- `rowid_column` - Primary key column name, optional for data scan, required for data modify + +## Entities + +### Snowflake Tables/Views + +This is an object representing a Snowflake table or view. + +Ref: [Snowflake docs](https://docs.snowflake.com/en/sql-reference/sql/create-table) -The Snowflake Wrapper supports data reads and writes from Snowflake. +#### Operations -| Integration | Select | Insert | Update | Delete | Truncate | +| Object | Select | Insert | Update | Delete | Truncate | | ----------- | :----: | :----: | :----: | :----: | :------: | -| Snowflake | ✅ | ✅ | ✅ | ✅ | ❌ | +| table/view | ✅ | ✅ | ✅ | ✅ | ❌ | -For example: +#### Usage ```sql create foreign table snowflake.mytable ( @@ -145,19 +167,14 @@ create foreign table snowflake.mytable ( ); ``` -### Foreign table options +#### Notes -The full list of foreign table options are below: - -- `table` - Source table or view name in Snowflake, required. - - This can also be a subquery enclosed in parentheses, for example, - - ```sql - table '(select * from mydatabase.public.mytable where id = 42)' - ``` - -- `rowid_column` - Primary key column name, optional for data scan, required for data modify +- Supports both tables and views as data sources +- Can use subqueries in `table` option +- Requires `rowid_column` for data modification operations +- Supports query pushdown for `where`, `order by`, and `limit` clauses +- Column names must match between Snowflake and foreign table +- Data types must be compatible according to type mapping table ## Query Pushdown Support @@ -165,7 +182,7 @@ This FDW supports `where`, `order by` and `limit` clause pushdown. ## Examples -Some examples on how to use Snowflake foreign tables. +### Basic Example Let's prepare the source table in Snowflake first: @@ -189,15 +206,7 @@ insert into mydatabase.public.mytable(id, name, num, dt, ts) values (43, 'bar', 56.78, '2024-05-19', '2024-05-19 12:34:56'); ``` -### Basic example - -This example will create a "foreign table" inside your Postgres database and query its data. First, we can create a schema to hold all the Snowflake foreign tables. - -```sql -create schema if not exists snowflake; -``` - -Then create the foreign table and query it, for example: +This example will create a "foreign table" inside your Postgres database and query its data. ```sql create foreign table snowflake.mytable ( @@ -216,9 +225,9 @@ create foreign table snowflake.mytable ( select * from snowflake.mytable; ``` -### Data modify example +### Data Modify Example -This example will modify data in a "foreign table" inside your Postgres database, note that `rowid_column` option is mandatory for data modify: +This example will modify data in a "foreign table" inside your Postgres database, note that `rowid_column` option is required for data modify: ```sql -- insert new data diff --git a/docs/catalog/stripe.md b/docs/catalog/stripe.md index 36b74821f..ca32f01d8 100644 --- a/docs/catalog/stripe.md +++ b/docs/catalog/stripe.md @@ -9,7 +9,9 @@ tags: # Stripe -[Stripe](https://stripe.com) is an API driven online payment processing utility. `supabase/wrappers` exposes below endpoints. +[Stripe](https://stripe.com) is an API driven online payment processing utility. + +The Stripe Wrapper allows you to read data from Stripe within your Postgres database. !!! warning @@ -37,26 +39,26 @@ create foreign data wrapper stripe_wrapper validator stripe_fdw_validator; ``` -### Store your credentials +### Store your credentials (optional) -We need to provide Postgres with the credentials to connect to Stripe, and any additional options. We can do this using the `create server` command: +By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. -=== "With Vault" +```sql +-- Save your Stripe API key in Vault and retrieve the `key_id` +insert into vault.secrets (name, secret) +values ( + 'stripe', + '' +) +returning key_id; +``` + +### Connecting to Stripe - By default, Postgres stores FDW credentials inside `pg_catalog.pg_foreign_server` in plain text. Anyone with access to this table will be able to view these credentials. +We need to provide Postgres with the credentials to connect to Stripe, and any additional options. We can do this using the `create server` command: - Wrappers is designed to work with [Vault](https://supabase.com/docs/guides/database/vault), which provides an additional level of security for storing credentials. We recommend using Vault to store your credentials. - ```sql - -- Save your Stripe API key in Vault and retrieve the `key_id` - insert into vault.secrets (name, secret) - values ( - 'stripe', - 'YOUR_SECRET' - ) - returning key_id; - ``` - Reference the credentials using the Key ID or Key Name: +=== "With Vault" ```sql create server stripe_server @@ -75,7 +77,7 @@ We need to provide Postgres with the credentials to connect to Stripe, and any a create server stripe_server foreign data wrapper stripe_wrapper options ( - api_key '', -- Stripe API key, required + api_key '', -- Stripe API key, required api_url 'https://api.stripe.com/v1/', -- Stripe API base URL, optional. Default is 'https://api.stripe.com/v1/' api_version '2024-06-20' -- Stripe API version, optional. Default is your Stripe account’s default API version. ); @@ -86,61 +88,39 @@ We need to provide Postgres with the credentials to connect to Stripe, and any a We recommend creating a schema to hold all the foreign tables: ```sql -create schema stripe; +create schema if not exists stripe; ``` -## Creating Foreign Tables +## Entities The Stripe Wrapper supports data read and modify from Stripe API. -| Object | Select | Insert | Update | Delete | Truncate | -| ----------------------------------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | -| [Accounts](https://stripe.com/docs/api/accounts/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Balance](https://stripe.com/docs/api/balance) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Balance Transactions](https://stripe.com/docs/api/balance_transactions/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Charges](https://stripe.com/docs/api/charges/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Checkout Sessions](https://stripe.com/docs/api/checkout/sessions/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Customers](https://stripe.com/docs/api/customers/list) | ✅ | ✅ | ✅ | ✅ | ❌ | -| [Disputes](https://stripe.com/docs/api/disputes/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Events](https://stripe.com/docs/api/events/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Files](https://stripe.com/docs/api/files/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [File Links](https://stripe.com/docs/api/file_links/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Invoices](https://stripe.com/docs/api/invoices/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Mandates](https://stripe.com/docs/api/mandates) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Meters](https://docs.stripe.com/api/billing/meter/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [PaymentIntents](https://stripe.com/docs/api/payment_intents/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Payouts](https://stripe.com/docs/api/payouts/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Prices](https://stripe.com/docs/api/prices/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Products](https://stripe.com/docs/api/products/list) | ✅ | ✅ | ✅ | ✅ | ❌ | -| [Refunds](https://stripe.com/docs/api/refunds/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [SetupAttempts](https://stripe.com/docs/api/setup_attempts/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [SetupIntents](https://stripe.com/docs/api/setup_intents/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Subscriptions](https://stripe.com/docs/api/subscriptions/list) | ✅ | ✅ | ✅ | ✅ | ❌ | -| [Tokens](https://stripe.com/docs/api/tokens) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Topups](https://stripe.com/docs/api/topups/list) | ✅ | ❌ | ❌ | ❌ | ❌ | -| [Transfers](https://stripe.com/docs/api/transfers/list) | ✅ | ❌ | ❌ | ❌ | ❌ | - -The Stripe foreign tables mirror Stripe's API. - -We can then create the foreign table, for example: - -```sql -create foreign table stripe.accounts ( - id text, - business_type text, - country text, - email text, - type text, - created timestamp, - attrs jsonb -) - server stripe_server - options ( - object 'accounts' - ); -``` - -`attrs` is a special column which stores all the object attributes in JSON format, you can extract any attributes needed or its associated sub objects from it. See more examples below. +| Object | Select | Insert | Update | Delete | Truncate | +| --------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Accounts](#accounts) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Balance](#balance) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Balance Transactions](#balance-transactions) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Charges](#charges) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Checkout Sessions](#checkout-sessions) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Customers](#customers) | ✅ | ✅ | ✅ | ✅ | ❌ | +| [Disputes](#disputes) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Events](#events) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Files](#files) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [File Links](#file-links) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Invoices](#invoices) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Mandates](#mandates) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Meters](#meters) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [PaymentIntents](#payment-intents) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Payouts](#payouts) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Prices](#prices) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Products](#products) | ✅ | ✅ | ✅ | ✅ | ❌ | +| [Refunds](#refunds) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [SetupAttempts](#setupattempts) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [SetupIntents](#setupintents) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Subscriptions](#subscriptions) | ✅ | ✅ | ✅ | ✅ | ❌ | +| [Tokens](#tokens) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Topups](#top-ups) | ✅ | ❌ | ❌ | ❌ | ❌ | +| [Transfers](#transfers) | ✅ | ❌ | ❌ | ❌ | ❌ | ### Accounts @@ -174,16 +154,23 @@ create foreign table stripe.accounts ( #### Notes -- While any column is allowed in a where clause, it is most efficient to filter by `id`. +- While any column is allowed in a where clause, it is most efficient to filter by `id` +- Use the `attrs` jsonb column to access additional account details ### Balance -_read only_ - -Shows the balance currently on your Stripe account. +This is an object representing your Stripe account's current balance. Ref: [Stripe docs](https://stripe.com/docs/api/balance) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Balance](https://stripe.com/docs/api/balance) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.balance ( balance_type text, @@ -197,14 +184,27 @@ create foreign table stripe.balance ( ); ``` -### Balance Transactions +#### Notes + +- Balance is a read-only object that shows the current funds in your Stripe account +- The balance is broken down by source types (e.g., card, bank account) and currencies +- Use the `attrs` jsonb column to access additional balance details like pending amounts +- While any column is allowed in a where clause, filtering options are limited as this is a singleton object -_read only_ +### Balance Transactions -Balance transactions represent funds moving through your Stripe account. They're created for every type of transaction that comes into or flows out of your Stripe account balance. +This is an object representing funds moving through your Stripe account. Balance transactions are created for every type of transaction that comes into or flows out of your Stripe account balance. Ref: [Stripe docs](https://stripe.com/docs/api/balance_transactions/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------------------------------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Balance Transactions](https://stripe.com/docs/api/balance_transactions/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.balance_transactions ( id text, @@ -224,19 +224,29 @@ create foreign table stripe.balance_transactions ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- type +- Balance transactions are read-only records of all funds movement in your Stripe account +- Each transaction includes amount, currency, fees, and net amount information +- Use the `attrs` jsonb column to access additional transaction details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - type ### Charges -_read only_ - -To charge a credit or a debit card, you create a Charge object. You can retrieve and refund individual charges as well as list all charges. Charges are identified by a unique, random ID. +This is an object representing a charge on a credit or debit card. You can retrieve and refund individual charges as well as list all charges. Charges are identified by a unique, random ID. Ref: [Stripe docs](https://stripe.com/docs/api/charges/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Charges](https://stripe.com/docs/api/charges/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.charges ( id text, @@ -256,19 +266,29 @@ create foreign table stripe.charges ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- customer +- Charges are read-only records of payment transactions in your Stripe account +- Each charge includes amount, currency, customer, and payment status information +- Use the `attrs` jsonb column to access additional charge details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - customer ### Checkout Sessions -_read only_ - -A Checkout Session represents your customer's session as they pay for one-time purchases or subscriptions through Checkout or Payment Links. We recommend creating a new Session each time your customer attempts to pay. +This is an object representing your customer's session as they pay for one-time purchases or subscriptions through Checkout or Payment Links. We recommend creating a new Session each time your customer attempts to pay. Ref: [Stripe docs](https://stripe.com/docs/api/checkout/sessions/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Checkout Sessions](https://stripe.com/docs/api/checkout/sessions/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.checkout_sessions ( id text, @@ -284,21 +304,31 @@ create foreign table stripe.checkout_sessions ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- customer -- payment_intent -- subscription +- Checkout Sessions are read-only records of customer payment sessions in your Stripe account +- Each session includes customer, payment intent, and subscription information +- Use the `attrs` jsonb column to access additional session details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - customer + - payment_intent + - subscription ### Customers -_read and modify_ - -Contains customers known to Stripe. +This is an object representing your Stripe customers. You can create, retrieve, update, and delete customers. Ref: [Stripe docs](https://stripe.com/docs/api/customers/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| --------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Customers](https://stripe.com/docs/api/customers/list) | ✅ | ✅ | ✅ | ✅ | ❌ | + +#### Usage + ```sql create foreign table stripe.customers ( id text, @@ -315,19 +345,46 @@ create foreign table stripe.customers ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +Example operations: -- id -- email +```sql +-- create a new customer +insert into stripe.customers(email, name, description) +values ('jane@example.com', 'Jane Smith', 'Premium customer'); + +-- update a customer +update stripe.customers +set name = 'Jane Doe' +where email = 'jane@example.com'; + +-- delete a customer +delete from stripe.customers +where id = 'cus_xxx'; +``` -### Disputes +#### Notes -_read only_ +- Customers can be created, retrieved, updated, and deleted through SQL operations +- Each customer can have an email, name, and description +- Use the `attrs` jsonb column to access additional customer details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - email -A dispute occurs when a customer questions your charge with their card issuer. +### Disputes + +This is an object representing a dispute that occurs when a customer questions your charge with their card issuer. Ref: [Stripe docs](https://stripe.com/docs/api/disputes/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Disputes](https://stripe.com/docs/api/disputes/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.disputes ( id text, @@ -346,20 +403,30 @@ create foreign table stripe.disputes ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- charge -- payment_intent +- Disputes are read-only records of customer payment disputes in your Stripe account +- Each dispute includes amount, currency, charge, and payment intent information +- Use the `attrs` jsonb column to access additional dispute details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - charge + - payment_intent ### Events -_read only_ - -Events are our way of letting you know when something interesting happens in your account. +This is an object representing events that occur in your Stripe account, letting you know when something interesting happens. Ref: [Stripe docs](https://stripe.com/docs/api/events/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Events](https://stripe.com/docs/api/events/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.events ( id text, @@ -374,19 +441,29 @@ create foreign table stripe.events ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- type +- Events are read-only records of activities in your Stripe account +- Each event includes type, API version, and timestamp information +- Use the `attrs` jsonb column to access additional event details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - type ### Files -_read only_ - This is an object representing a file hosted on Stripe's servers. Ref: [Stripe docs](https://stripe.com/docs/api/files/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| --------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Files](https://stripe.com/docs/api/files/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.files ( id text, @@ -406,19 +483,30 @@ create foreign table stripe.files ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- purpose +- Files are read-only records of files hosted on Stripe's servers +- Each file includes filename, purpose, size, type, and URL information +- Files may have an expiration date specified in expires_at +- Use the `attrs` jsonb column to access additional file details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - purpose ### File Links -_read only_ - -To share the contents of a `File` object with non-Stripe users, you can create a `FileLink`. +This is an object representing a link that can be used to share the contents of a `File` object with non-Stripe users. Ref: [Stripe docs](https://stripe.com/docs/api/file_links/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [File Links](https://stripe.com/docs/api/file_links/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.file_links ( id text, @@ -435,14 +523,31 @@ create foreign table stripe.file_links ( ); ``` -### Invoices +#### Notes + +- File Links are read-only records that provide shareable access to Stripe files +- Each link includes a reference to the file and a public URL +- Links can be configured to expire at a specific time +- Use the `expired` boolean to check if a link has expired +- Use the `attrs` jsonb column to access additional link details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - file -_read only_ +### Invoices -Invoices are statements of amounts owed by a customer, and are either generated one-off, or generated periodically from a subscription. +This is an object representing statements of amounts owed by a customer, which are either generated one-off or periodically from a subscription. Ref: [Stripe docs](https://stripe.com/docs/api/invoices/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Invoices](https://stripe.com/docs/api/invoices/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.invoices ( id text, @@ -459,24 +564,34 @@ create foreign table stripe.invoices ( options ( object 'invoices' ); - ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- customer -- status -- subscription +- Invoices are read-only records of amounts owed by customers +- Each invoice includes customer, subscription, status, and amount information +- Invoices track billing periods with period_start and period_end timestamps +- Use the `attrs` jsonb column to access additional invoice details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - customer + - status + - subscription ### Mandates -_read only_ - -A Mandate is a record of the permission a customer has given you to debit their payment method. +This is an object representing a record of the permission a customer has given you to debit their payment method. Ref: [Stripe docs](https://stripe.com/docs/api/mandates) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Mandates](https://stripe.com/docs/api/mandates) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.mandates ( id text, @@ -491,18 +606,28 @@ create foreign table stripe.mandates ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id +- Mandates are read-only records of customer payment permissions +- Each mandate includes payment method, status, and type information +- Use the `attrs` jsonb column to access additional mandate details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id ### Meters -_read only_ - -A billing meter is a resource that allows you to track usage of a particular event. +This is an object representing a billing meter that allows you to track usage of a particular event. Ref: [Stripe docs](https://docs.stripe.com/api/billing/meter) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Meters](https://docs.stripe.com/api/billing/meter) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.meter ( id text, @@ -518,18 +643,29 @@ create foreign table stripe.meter ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id +- Meters are read-only records for tracking event usage in billing +- Each meter includes display name, event name, and time window information +- The status field indicates whether the meter is active +- Use the `attrs` jsonb column to access additional meter details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id ### Payment Intents -_read only_ - -A payment intent guides you through the process of collecting a payment from your customer. +This is an object representing a guide through the process of collecting a payment from your customer. Ref: [Stripe docs](https://stripe.com/docs/api/payment_intents/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Payment Intents](https://stripe.com/docs/api/payment_intents/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.payment_intents ( id text, @@ -546,19 +682,30 @@ create foreign table stripe.payment_intents ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- customer +- Payment Intents are read-only records that guide the payment collection process +- Each intent includes customer, amount, currency, and payment method information +- The created timestamp tracks when the payment intent was initiated +- Use the `attrs` jsonb column to access additional payment intent details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - customer ### Payouts -_read only_ - -A `Payout` object is created when you receive funds from Stripe, or when you initiate a payout to either a bank account or debit card of a connected Stripe account. +This is an object representing funds received from Stripe or initiated payouts to a bank account or debit card of a connected Stripe account. Ref: [Stripe docs](https://stripe.com/docs/api/payouts/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Payouts](https://stripe.com/docs/api/payouts/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.payouts ( id text, @@ -577,19 +724,31 @@ create foreign table stripe.payouts ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- status +- Payouts are read-only records of fund transfers +- Each payout includes amount, currency, and status information +- The arrival_date indicates when funds will be available +- The statement_descriptor appears on your bank statement +- Use the `attrs` jsonb column to access additional payout details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - status ### Prices -_read only_ - -A `Price` object is needed for all of your products to facilitate multiple currencies and pricing options. +This is an object representing pricing configurations for products to facilitate multiple currencies and pricing options. Ref: [Stripe docs](https://stripe.com/docs/api/prices/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Prices](https://stripe.com/docs/api/prices/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.prices ( id text, @@ -607,19 +766,31 @@ create foreign table stripe.prices ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- active +- Prices are read-only records that define product pricing configurations +- Each price includes currency, unit amount, and product reference +- The active boolean indicates if the price can be used +- The type field specifies the pricing model (e.g., one-time, recurring) +- Use the `attrs` jsonb column to access additional price details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - active ### Products -_read and modify_ - -All products available in Stripe. +This is an object representing all products available in Stripe. Ref: [Stripe docs](https://stripe.com/docs/api/products/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Products](https://stripe.com/docs/api/products/list) | ✅ | ✅ | ✅ | ✅ | ❌ | + +#### Usage + ```sql create foreign table stripe.products ( id text, @@ -638,19 +809,31 @@ create foreign table stripe.products ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- active +- Products can be created, read, updated, and deleted +- Each product includes name, description, and active status +- The default_price links to the product's default Price object +- The updated timestamp tracks the last modification time +- Use the `attrs` jsonb column to access additional product details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - active ### Refunds -_read only_ - -`Refund` objects allow you to refund a charge that has previously been created but not yet refunded. +This is an object representing refunds for charges that have previously been created but not yet refunded. Ref: [Stripe docs](https://stripe.com/docs/api/refunds/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Refunds](https://stripe.com/docs/api/refunds/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.refunds ( id text, @@ -669,20 +852,32 @@ create foreign table stripe.refunds ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- charge -- payment_intent +- Refunds are read-only records of charge reversals +- Each refund includes amount, currency, and status information +- The charge and payment_intent fields link to the original transaction +- The reason field provides context for the refund +- Use the `attrs` jsonb column to access additional refund details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - charge + - payment_intent ### SetupAttempts -_read only_ - -A `SetupAttempt` describes one attempted confirmation of a SetupIntent, whether that confirmation was successful or unsuccessful. +This is an object representing attempted confirmations of SetupIntents, tracking both successful and unsuccessful attempts. Ref: [Stripe docs](https://stripe.com/docs/api/setup_attempts/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [SetupAttempts](https://stripe.com/docs/api/setup_attempts/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.setup_attempts ( id text, @@ -702,19 +897,31 @@ create foreign table stripe.setup_attempts ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- setup_intent +- SetupAttempts are read-only records of payment setup confirmation attempts +- Each attempt includes customer, payment method, and status information +- The setup_intent field links to the associated SetupIntent +- The usage field indicates the intended payment method usage +- Use the `attrs` jsonb column to access additional attempt details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - setup_intent ### SetupIntents -_read only_ - -A `SetupIntent` guides you through the process of setting up and saving a customer's payment credentials for future payments. +This is an object representing a guide through the process of setting up and saving customer payment credentials for future payments. Ref: [Stripe docs](https://stripe.com/docs/api/setup_intents/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [SetupIntents](https://stripe.com/docs/api/setup_intents/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.setup_intents ( id text, @@ -733,20 +940,32 @@ create foreign table stripe.setup_intents ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- customer -- payment_method +- SetupIntents are read-only records for saving customer payment credentials +- Each intent includes customer, payment method, and status information +- The client_secret is used for client-side confirmation +- The usage field indicates how the payment method will be used +- Use the `attrs` jsonb column to access additional intent details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - customer + - payment_method ### Subscriptions -_read and modify_ - -Customer recurring payment schedules. +This is an object representing customer recurring payment schedules. Ref: [Stripe docs](https://stripe.com/docs/api/subscriptions/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ----------------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Subscriptions](https://stripe.com/docs/api/subscriptions/list) | ✅ | ✅ | ✅ | ✅ | ❌ | + +#### Usage + ```sql create foreign table stripe.subscriptions ( id text, @@ -763,28 +982,41 @@ create foreign table stripe.subscriptions ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- customer -- price -- status +- Subscriptions can be created, read, updated, and deleted +- Each subscription includes customer and currency information +- The current_period_start and current_period_end track billing cycles +- The rowid_column option enables modification operations +- Use the `attrs` jsonb column to access additional subscription details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - customer + - price + - status ### Tokens -_read only_ - -Tokenization is the process Stripe uses to collect sensitive card or bank account details, or personally identifiable information (PII), directly from your customers in a secure manner. +This is an object representing a secure way to collect sensitive card, bank account, or personally identifiable information (PII) from customers. Ref: [Stripe docs](https://stripe.com/docs/api/tokens) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| --------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Tokens](https://stripe.com/docs/api/tokens) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.tokens ( id text, - customer text, - currency text, - current_period_start timestamp, - current_period_end timestamp, + type text, + client_ip text, + created timestamp, + livemode boolean, + used boolean, attrs jsonb ) server stripe_server @@ -793,14 +1025,32 @@ create foreign table stripe.tokens ( ); ``` -### Top-ups +#### Notes + +- Tokens are read-only, single-use objects for secure data collection +- Each token includes type information (card, bank_account, pii, etc.) +- The client_ip field records where the token was created +- The used field indicates if the token has been used +- Use the `attrs` jsonb column to access token details like card or bank information +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - type + - used -_read only_ +### Top-ups -To top up your Stripe balance, you create a top-up object. +This is an object representing a way to add funds to your Stripe balance. Ref: [Stripe docs](https://stripe.com/docs/api/topups/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| --------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Top-ups](https://stripe.com/docs/api/topups/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.topups ( id text, @@ -817,19 +1067,30 @@ create foreign table stripe.topups ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- status +- Top-ups are read-only records of balance additions +- Each top-up includes amount and currency information +- The status field tracks the top-up state (e.g., succeeded, failed) +- Use the `attrs` jsonb column to access additional top-up details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - status ### Transfers -_read only_ - -A Transfer object is created when you move funds between Stripe accounts as part of Connect. +This is an object representing fund movements between Stripe accounts as part of Connect. Ref: [Stripe docs](https://stripe.com/docs/api/transfers/list) +#### Operations + +| Object | Select | Insert | Update | Delete | Truncate | +| ------------------------------------------------------- | :----: | :----: | :----: | :----: | :------: | +| [Transfers](https://stripe.com/docs/api/transfers/list) | ✅ | ❌ | ❌ | ❌ | ❌ | + +#### Usage + ```sql create foreign table stripe.transfers ( id text, @@ -846,10 +1107,15 @@ create foreign table stripe.transfers ( ); ``` -While any column is allowed in a where clause, it is most efficient to filter by: +#### Notes -- id -- destination +- Transfers are read-only records of fund movements between accounts +- Each transfer includes amount, currency, and destination information +- The destination field identifies the receiving Stripe account +- Use the `attrs` jsonb column to access additional transfer details +- While any column is allowed in a where clause, it is most efficient to filter by: + - id + - destination ## Query Pushdown Support @@ -861,7 +1127,7 @@ For example, this query select * from stripe.customers where id = 'cus_xxx'; ``` -will be translated Stripe API call: `https://api.stripe.com/v1/customers/cus_xxx`. +will be translated to a Stripe API call: `https://api.stripe.com/v1/customers/cus_xxx`. For supported filter columns for each object, please check out foreign table documents above. @@ -897,10 +1163,22 @@ from stripe.subscriptions where id = 'sub_xxx'; ### Data modify ```sql -insert into stripe.customers(email,name,description) values ('test@test.com', 'test name', null); -update stripe.customers set description='hello fdw' where id ='cus_xxx'; -update stripe.customers set attrs='{"metadata[foo]": "bar"}' where id ='cus_xxx'; -delete from stripe.customers where id ='cus_xxx'; +-- insert +insert into stripe.customers(email,name,description) +values ('test@test.com', 'test name', null); + +-- update +update stripe.customers +set description='hello fdw' +where id = 'cus_xxx'; + +update stripe.customers +set attrs='{"metadata[foo]": "bar"}' +where id = 'cus_xxx'; + +-- delete +delete from stripe.customers +where id = 'cus_xxx'; ``` To insert into an object with sub-fields, we need to create the foreign table with column name exactly same as the API required. For example, to insert a `subscription` object we can define the foreign table following [the Stripe API docs](https://docs.stripe.com/api/subscriptions/create): @@ -923,7 +1201,7 @@ create foreign table stripe.subscriptions ( And then we can insert a subscription like below: ```sql -insert into stripe.subscriptions (customer, "items[0][price]") +insert into stripe.subscriptions(customer, "items[0][price]") values ('cus_Na6dX7aXxi11N4', 'price_1MowQULkdIwHu7ixraBm864M'); ```