Skip to content

Commit

Permalink
Update Iceberg rest catalog (#166)
Browse files Browse the repository at this point in the history
* iceberg rest catalog

* minor update

* Update apache-iceberg.mdx
  • Loading branch information
WanYixian authored Jan 2, 2025
1 parent f74131d commit 56d65e0
Show file tree
Hide file tree
Showing 2 changed files with 34 additions and 18 deletions.
26 changes: 17 additions & 9 deletions integrations/destinations/apache-iceberg.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ WITH (
| catalog.url | Conditional. The URL of the catalog. It is required when catalog.type is not storage. |
| primary\_key | The primary key for an upsert sink. It is only applicable to the upsert mode. |
| commit\_checkpoint\_interval | Optional. Commit every N checkpoints (N > 0). Default value is 10. <br/>The behavior of this field also depends on the `sink_decouple` setting:<ul><li>If `sink_decouple` is true (the default), the default value of `commit_checkpoint_interval` is 10.</li> <li>If `sink_decouple` is set to false, the default value of `commit_checkpoint_interval` is 1.</li> <li>If `sink_decouple` is set to false and `commit_checkpoint_interval` is set to larger than 1, an error will occur.</li></ul>|
| catalog.credential | Optional. Credential for accessing the Iceberg catalog, used to exchange for a token in the OAuth2 client credentials flow. Applicable only in the `rest` catalog.|
| catalog.token| Optional. A Bearer token for accessing the Iceberg catalog, used for interaction with the server. Applicable only in the `rest` catalog. |
| catalog.oauth2-server-uri | Optional. The `oauth2-server-uri` for accessing the Iceberg catalog, serving as the token endpoint URI to fetch a token if the `rest` catalog is not the authorization server. Applicable only in the `rest` catalog.|
| catalog.scope | Optional. Scope for accessing the Iceberg catalog, providing additional scope for OAuth2. Applicable only in the `rest` catalog. |

## Data type mapping

Expand Down Expand Up @@ -101,16 +105,20 @@ with (
connector = 'iceberg',
type = 'append-only',
force_append_only = true,
s3.endpoint = 'http://minio-0:9301',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-1',
s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
s3.region = 'ap-southeast-2',
s3.access.key = 'xxxx',
s3.secret.key = 'xxxx',
s3.path.style.access = 'true',
catalog.type = 'rest',
catalog.name = 'demo',
catalog.uri = 'http://rest:8181',
warehouse.path = 's3://icebergdata/demo',
database.name = 's1',
table.name = 't1'
catalog.uri = 'http://localhost:8181/api/catalog',
warehouse.path = 'quickstart_catalog',
database.name = 'ns',
table.name = 't1',
catalog.credential='123456:123456',
catalog.scope='PRINCIPAL_ROLE:ALL',
catalog.oauth2-server-uri='xxx'
catalog.scope='xxx',
);
```

Expand Down
26 changes: 17 additions & 9 deletions integrations/sources/apache-iceberg.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@ You don’t need to specify the column name for the Iceberg source, as RisingWav
| catalog.type | Optional. The catalog type used in this table. Currently, the supported values are `storage`, `rest`, `hive`, `jdbc`, and `glue`. If not specified, `storage` is used. For details, see [Catalogs](#catalogs). |
| warehouse.path | Conditional. The path of the Iceberg warehouse. Currently, only S3-compatible object storage systems, such as AWS S3 and MinIO, are supported. It's required if the `catalog.type` is not `rest`. |
| catalog.url | Conditional. The URL of the catalog. It is required when `catalog.type` is not `storage`. |
| catalog.credential | Optional. Credential for accessing the Iceberg catalog, used to exchange for a token in the OAuth2 client credentials flow. Applicable only in the `rest` catalog.|
| catalog.token| Optional. A Bearer token for accessing the Iceberg catalog, used for interaction with the server. Applicable only in the `rest` catalog. |
| catalog.oauth2-server-uri | Optional. The `oauth2-server-uri` for accessing the Iceberg catalog, serving as the token endpoint URI to fetch a token if the `rest` catalog is not the authorization server. Applicable only in the `rest` catalog.|
| catalog.scope | Optional. Scope for accessing the Iceberg catalog, providing additional scope for OAuth2. Applicable only in the `rest` catalog. |

## Data type mapping

Expand Down Expand Up @@ -83,16 +87,20 @@ Examples
create source source_demo_rest
with (
connector = 'iceberg',
s3.endpoint = 'http://minio-0:9301',
s3.access.key = 'xxxxxxxxxx',
s3.secret.key = 'xxxxxxxxxx',
s3.region = 'ap-southeast-1',
s3.endpoint = 'https://s3.ap-southeast-2.amazonaws.com',
s3.region = 'ap-southeast-2',
s3.access.key = 'xxxx',
s3.secret.key = 'xxxx',
s3.path.style.access = 'true',
catalog.type = 'rest',
catalog.name = 'demo',
catalog.uri = 'http://rest:8181',
warehouse.path = 's3://icebergdata/demo',
database.name = 's1',
table.name = 't1'
catalog.uri = 'http://localhost:8181/api/catalog',
warehouse.path = 'quickstart_catalog',
database.name = 'ns',
table.name = 't1',
catalog.credential='123456:123456',
catalog.scope='PRINCIPAL_ROLE:ALL',
catalog.oauth2-server-uri='xxx'
catalog.scope='xxx',
);
```
* Hive catalog: RisingWave supports the Hive catalog. You need to set `catalog.type` to `hive` to use it.
Expand Down

0 comments on commit 56d65e0

Please sign in to comment.