From d712691de4dde55fe5ca6bcf092bb1a12bf00845 Mon Sep 17 00:00:00 2001 From: Tristen Harr Date: Mon, 10 Jun 2024 16:42:30 -0500 Subject: [PATCH] update docs --- LICENSE | 201 ---------------------------------- LICENSE.txt | 201 ++++++++++++++++++++++++++++++++++ README.md | 232 +++++++++++++++++++++++----------------- docs/architecture.md | 109 +++++++++++++++++++ docs/code-of-conduct.md | 74 +++++++++++++ docs/contributing.md | 39 +++++++ docs/development.md | 70 ++++++++++++ docs/index.md | 13 +++ docs/logo.svg | 1 + docs/production.md | 30 ++++++ docs/security.md | 30 ++++++ docs/support.md | 11 ++ docs/usage/index.md | 5 + docs/usage/querying.md | 128 ++++++++++++++++++++++ 14 files changed, 847 insertions(+), 297 deletions(-) delete mode 100644 LICENSE create mode 100644 LICENSE.txt create mode 100644 docs/architecture.md create mode 100644 docs/code-of-conduct.md create mode 100644 docs/contributing.md create mode 100644 docs/development.md create mode 100644 docs/index.md create mode 100644 docs/logo.svg create mode 100644 docs/production.md create mode 100644 docs/security.md create mode 100644 docs/support.md create mode 100644 docs/usage/index.md create mode 100644 docs/usage/querying.md diff --git a/LICENSE b/LICENSE deleted file mode 100644 index eedc68b..0000000 --- a/LICENSE +++ /dev/null @@ -1,201 +0,0 @@ - Apache License - Version 2.0, January 2004 - http://www.apache.org/licenses/ - -TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION - -1. Definitions. - - "License" shall mean the terms and conditions for use, reproduction, - and distribution as defined by Sections 1 through 9 of this document. - - "Licensor" shall mean the copyright owner or entity authorized by - the copyright owner that is granting the License. - - "Legal Entity" shall mean the union of the acting entity and all - other entities that control, are controlled by, or are under common - control with that entity. For the purposes of this definition, - "control" means (i) the power, direct or indirect, to cause the - direction or management of such entity, whether by contract or - otherwise, or (ii) ownership of fifty percent (50%) or more of the - outstanding shares, or (iii) beneficial ownership of such entity. - - "You" (or "Your") shall mean an individual or Legal Entity - exercising permissions granted by this License. - - "Source" form shall mean the preferred form for making modifications, - including but not limited to software source code, documentation - source, and configuration files. - - "Object" form shall mean any form resulting from mechanical - transformation or translation of a Source form, including but - not limited to compiled object code, generated documentation, - and conversions to other media types. - - "Work" shall mean the work of authorship, whether in Source or - Object form, made available under the License, as indicated by a - copyright notice that is included in or attached to the work - (an example is provided in the Appendix below). - - "Derivative Works" shall mean any work, whether in Source or Object - form, that is based on (or derived from) the Work and for which the - editorial revisions, annotations, elaborations, or other modifications - represent, as a whole, an original work of authorship. For the purposes - of this License, Derivative Works shall not include works that remain - separable from, or merely link (or bind by name) to the interfaces of, - the Work and Derivative Works thereof. - - "Contribution" shall mean any work of authorship, including - the original version of the Work and any modifications or additions - to that Work or Derivative Works thereof, that is intentionally - submitted to Licensor for inclusion in the Work by the copyright owner - or by an individual or Legal Entity authorized to submit on behalf of - the copyright owner. For the purposes of this definition, "submitted" - means any form of electronic, verbal, or written communication sent - to the Licensor or its representatives, including but not limited to - communication on electronic mailing lists, source code control systems, - and issue tracking systems that are managed by, or on behalf of, the - Licensor for the purpose of discussing and improving the Work, but - excluding communication that is conspicuously marked or otherwise - designated in writing by the copyright owner as "Not a Contribution." - - "Contributor" shall mean Licensor and any individual or Legal Entity - on behalf of whom a Contribution has been received by Licensor and - subsequently incorporated within the Work. - -2. Grant of Copyright License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - copyright license to reproduce, prepare Derivative Works of, - publicly display, publicly perform, sublicense, and distribute the - Work and such Derivative Works in Source or Object form. - -3. Grant of Patent License. Subject to the terms and conditions of - this License, each Contributor hereby grants to You a perpetual, - worldwide, non-exclusive, no-charge, royalty-free, irrevocable - (except as stated in this section) patent license to make, have made, - use, offer to sell, sell, import, and otherwise transfer the Work, - where such license applies only to those patent claims licensable - by such Contributor that are necessarily infringed by their - Contribution(s) alone or by combination of their Contribution(s) - with the Work to which such Contribution(s) was submitted. If You - institute patent litigation against any entity (including a - cross-claim or counterclaim in a lawsuit) alleging that the Work - or a Contribution incorporated within the Work constitutes direct - or contributory patent infringement, then any patent licenses - granted to You under this License for that Work shall terminate - as of the date such litigation is filed. - -4. Redistribution. You may reproduce and distribute copies of the - Work or Derivative Works thereof in any medium, with or without - modifications, and in Source or Object form, provided that You - meet the following conditions: - - (a) You must give any other recipients of the Work or - Derivative Works a copy of this License; and - - (b) You must cause any modified files to carry prominent notices - stating that You changed the files; and - - (c) You must retain, in the Source form of any Derivative Works - that You distribute, all copyright, patent, trademark, and - attribution notices from the Source form of the Work, - excluding those notices that do not pertain to any part of - the Derivative Works; and - - (d) If the Work includes a "NOTICE" text file as part of its - distribution, then any Derivative Works that You distribute must - include a readable copy of the attribution notices contained - within such NOTICE file, excluding those notices that do not - pertain to any part of the Derivative Works, in at least one - of the following places: within a NOTICE text file distributed - as part of the Derivative Works; within the Source form or - documentation, if provided along with the Derivative Works; or, - within a display generated by the Derivative Works, if and - wherever such third-party notices normally appear. The contents - of the NOTICE file are for informational purposes only and - do not modify the License. You may add Your own attribution - notices within Derivative Works that You distribute, alongside - or as an addendum to the NOTICE text from the Work, provided - that such additional attribution notices cannot be construed - as modifying the License. - - You may add Your own copyright statement to Your modifications and - may provide additional or different license terms and conditions - for use, reproduction, or distribution of Your modifications, or - for any such Derivative Works as a whole, provided Your use, - reproduction, and distribution of the Work otherwise complies with - the conditions stated in this License. - -5. Submission of Contributions. Unless You explicitly state otherwise, - any Contribution intentionally submitted for inclusion in the Work - by You to the Licensor shall be under the terms and conditions of - this License, without any additional terms or conditions. - Notwithstanding the above, nothing herein shall supersede or modify - the terms of any separate license agreement you may have executed - with Licensor regarding such Contributions. - -6. Trademarks. This License does not grant permission to use the trade - names, trademarks, service marks, or product names of the Licensor, - except as required for reasonable and customary use in describing the - origin of the Work and reproducing the content of the NOTICE file. - -7. Disclaimer of Warranty. Unless required by applicable law or - agreed to in writing, Licensor provides the Work (and each - Contributor provides its Contributions) on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or - implied, including, without limitation, any warranties or conditions - of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A - PARTICULAR PURPOSE. You are solely responsible for determining the - appropriateness of using or redistributing the Work and assume any - risks associated with Your exercise of permissions under this License. - -8. Limitation of Liability. In no event and under no legal theory, - whether in tort (including negligence), contract, or otherwise, - unless required by applicable law (such as deliberate and grossly - negligent acts) or agreed to in writing, shall any Contributor be - liable to You for damages, including any direct, indirect, special, - incidental, or consequential damages of any character arising as a - result of this License or out of the use or inability to use the - Work (including but not limited to damages for loss of goodwill, - work stoppage, computer failure or malfunction, or any and all - other commercial damages or losses), even if such Contributor - has been advised of the possibility of such damages. - -9. Accepting Warranty or Additional Liability. While redistributing - the Work or Derivative Works thereof, You may choose to offer, - and charge a fee for, acceptance of support, warranty, indemnity, - or other liability obligations and/or rights consistent with this - License. However, in accepting such obligations, You may act only - on Your own behalf and on Your sole responsibility, not on behalf - of any other Contributor, and only if You agree to indemnify, - defend, and hold each Contributor harmless for any liability - incurred by, or claims asserted against, such Contributor by reason - of your accepting any such warranty or additional liability. - -END OF TERMS AND CONDITIONS - -APPENDIX: How to apply the Apache License to your work. - - To apply the Apache License to your work, attach the following - boilerplate notice, with the fields enclosed by brackets "[]" - replaced with your own identifying information. (Don't include - the brackets!) The text should be enclosed in the appropriate - comment syntax for the file format. We also recommend that a - file or class name and description of purpose be included on the - same "printed page" as the copyright notice for easier - identification within third-party archives. - -Copyright [yyyy] [name of copyright owner] - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. \ No newline at end of file diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..05842e4 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2023 Hasura Inc. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file diff --git a/README.md b/README.md index 72904dc..a1f061b 100644 --- a/README.md +++ b/README.md @@ -1,137 +1,177 @@ -## DuckDB Connector +# Hasura DuckDB Connector + -The DuckDB Data Connector allows for connecting to a Motherduck hosted DuckDB database, or a local DuckDB database file. This connector uses the [Typescript Data Connector SDK](https://github.com/hasura/ndc-sdk-typescript) and implements the [Data Connector Spec](https://github.com/hasura/ndc-spec). +[![Docs](https://img.shields.io/badge/docs-v3.x-brightgreen.svg?style=flat)](https://hasura.io/connectors/duckdb) +[![ndc-hub](https://img.shields.io/badge/ndc--hub-duckdb-blue.svg?style=flat)](https://hasura.io/connectors/duckdb) +[![License](https://img.shields.io/badge/license-Apache--2.0-purple.svg?style=flat)](./LICENSE.txt) +[![Status](https://img.shields.io/badge/status-alpha-yellow.svg?style=flat)](./README.md) -This connector currently only supports querying. +The Hasura DuckDB Connector allows for connecting to a DuckDB database or a MotherDuck hosted DuckDB database to give you an instant GraphQL API on top of your DuckDB data. -### Setting up the DuckDB connector using Hasura Cloud & MotherDuck +This connector is built using the [Typescript Data Connector SDK](https://github.com/hasura/ndc-sdk-typescript) and implements the [Data Connector Spec](https://github.com/hasura/ndc-spec). -#### Step 1: Prerequisites +* [Connector information in the Hasura Hub](https://hasura.io/connectors/duckdb) +* [Hasura V3 Documentation](https://hasura.io/docs/3.0/index/) -1. Install the [new Hasura CLI](https://hasura.io/docs/3.0/cli/installation/) — to quickly and easily create and manage your Hasura projects and builds. -2. Install the [Hasura VS Code extension](https://marketplace.visualstudio.com/items?itemName=HasuraHQ.hasura) — with support for other editors coming soon! -3. Have a [MotherDuck](https://motherduck.com/) hosted DuckDB database — for supplying data to your API. +## Features -#### Step 2: Login to Hasura +Below, you'll find a matrix of all supported features for the DuckDB connector: -After our prerequisites are taken care of, login to Hasura Cloud with the CLI: +| Feature | Supported | Notes | +| ------------------------------- | --------- | ----- | +| Native Queries + Logical Models | ❌ | | +| Simple Object Query | ✅ | | +| Filter / Search | ✅ | | +| Simple Aggregation | ❌ | | +| Sort | ✅ | | +| Paginate | ✅ | | +| Table Relationships | ✅ | | +| Views | ❌ | | +| Distinct | ❌ | | +| Remote Relationships | ✅ | | +| Custom Fields | ❌ | | +| Mutations | ❌ | | -`ddn login` +## Before you get Started -This will open up a browser window and initiate an OAuth2 login flow. If the browser window doesn't open automatically, use the link shown in the terminal output to launch the flow. +[Prerequisites or recommended steps before using the connector.] -#### Step 3: Create a new project +1. The [DDN CLI](https://hasura.io/docs/3.0/cli/installation) and [Docker](https://docs.docker.com/engine/install/) installed +2. A [supergraph](https://hasura.io/docs/3.0/getting-started/init-supergraph) +3. A [subgraph](https://hasura.io/docs/3.0/getting-started/init-subgraph) +4. Have a [MotherDuck](https://motherduck.com/) hosted DuckDB database, or a persitent DuckDB database file — for supplying data to your API. -We'll use the `create project` command to create a new project: +The steps below explain how to Initialize and configure a connector for local development. You can learn how to deploy a +connector — after it's been configured — [here](https://hasura.io/docs/3.0/getting-started/deployment/deploy-a-connector). -`ddn create project --dir ./ddn` +## Using the DuckDB connector -#### Step 4: Add a connector manifest +### Step 1: Authenticate your CLI session -Let's move into the project directory: +```bash +ddn auth login +``` -`cd ddn` +### Step 2: Initialize the connector -Create a subgraph: +```bash +ddn connector init duckdb --subgraph my_subgraph --hub-connector hasura/duckdb +``` -`ddn create subgraph duckdb` +In the snippet above, we've used the subgraph `my_subgraph` as an example; however, you should change this +value to match any subgraph which you've created in your project. -Then, create a connector manifest: -`ddn add connector-manifest duckdb_connector --subgraph duckdb --hub-connector hasura/duckdb --type cloud` +### Step 3: Modify the connector's port -#### Step 5: Edit the connector manifest +When you initialized your connector, the CLI generated a set of configuration files, including a Docker Compose file for +the connector. Typically, connectors default to port `8080`. Each time you add a connector, we recommend incrementing the published port by one to avoid port collisions. -You should have a connector manifest created at `ddn/duckdb/duckdb_connector/connector/duckdb_connector.build.hml` +As an example, if your connector's configuration is in `my_subgraph/connector/duckdb/dokcer-compose.duckdb.yaml`, you can modify the published port to +reflect a value that isn't currently being used by any other connectors: ```yaml -kind: ConnectorManifest -version: v1 -spec: - supergraphManifests: - - base -definition: - name: duckdb_connector - type: cloud - connector: - type: hub - name: hasura/duckdb:v0.0.9 - deployments: - - context: . - env: - DUCKDB_URL: - value: "" +ports: + - mode: ingress + target: 8080 + published: "8082" + protocol: tcp ``` -Fill in the value for the DUCKDB_URL environment variable with a DuckDB connection string which looks something like this: +### Step 4: Add environment variables -`md:?motherduck_token=ey` +Now that our connector has been scaffolded out for us, we need to provide a connection string so that the data source can be introspected and the +boilerplate configuration can be taken care of by the CLI. -(Make sure to save your changes to the file!) +The CLI has provided an `.env.local` file for our connector in the `my_subgraph/connector/duckdb` directory. We can add a key-value pair +of `DUCKDB_URL` along with the connection string itself to this file, and our connector will use this to connect to our database. -#### Step 6: Start a development session -Start a Hasura dev session using the following command: +The file, after adding the `DUCKDB_URL`, should look like this example if connecting to a MotherDuck hosted DuckDB instance: -`ddn dev` +```env +OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://local.hasura.dev:4317 +OTEL_SERVICE_NAME=my_subgraph_duckdb +DUCKDB_URL=md:?motherduck_token=eyJhbGc... +``` -You should see something like this: +To connect to a local DuckDB file, you can add the persistent DuckDB database file into the `my_subgraph/connector/duckdb` directory, and since all files in this directory will get mounted to the container at `/etc/connector/` you can then point the `DUCKDB_URL` to the local file. Assuming that the duckdb file was named `chinook.db` the file should look like this example: +```env +OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://local.hasura.dev:4317 +OTEL_SERVICE_NAME=my_subgraph_duckdb +DUCKDB_URL=/etc/connector/chinook.db ``` -12:10PM INF Building SupergraphManifest "base"... -+---------------+--------------------------------------------------------------------------------------------------+ -| Build Version | 7e01762194 | -+---------------+--------------------------------------------------------------------------------------------------+ -| API URL | https://pure-sparrow-2887-7e01762194.ddn.hasura.app/graphql | -+---------------+--------------------------------------------------------------------------------------------------+ -| Console URL | https://console.hasura.io/project/pure-sparrow-2887/environment/default/build/7e01762194/graphql | -+---------------+--------------------------------------------------------------------------------------------------+ -| Project Name | pure-sparrow-2887 | -+---------------+--------------------------------------------------------------------------------------------------+ -| Description | Dev build - Mon, 15 Apr 2024 12:10:17 CDT | -+---------------+--------------------------------------------------------------------------------------------------+ + +### Step 5: Introspect your data source + +With the connector configured, we can now use the CLI to introspect our database and create a source-specific configuration file for our connector. + +```bash +ddn connector introspect --connector my_subgraph/connector/duckdb/connector.yaml ``` -Navigate to the Console URL and you can issue a query. You can use the default DuckDB example tables to issue this query for example: - -```graphql -query QueryHackerNews { - duckdb_sampleDataHnHackerNews( - limit: 4 - where: {_or: [{text: {_like: "%Hasura%"}}, {text: {_like: "%DuckDB%"}}]} - ) { - text - id - } -} +### Step 6. Create the Hasura metadata + +Hasura DDN uses a concept called "connector linking" to take [NDC-compliant](https://github.com/hasura/ndc-spec) +configuration JSON files for a data connector and transform them into an `hml` (Hasura Metadata Language) file as a +[`DataConnectorLink` metadata object](https://hasura.io/docs/3.0/supergraph-modeling/data-connectors#dataconnectorlink-dataconnectorlink). + +Basically, metadata objects in `hml` files define our API. + +First we need to create this `hml` file with the `connector-link add` command and then convert our configuration files +into `hml` syntax and add it to this file with the `connector-link update` command. + +Let's name the `hml` file the same as our connector, `duckdb`: + +```bash +ddn connector-link add duckdb --subgraph my_subgraph ``` -And get the response: +The new file is scaffolded out at `my_subgraph/metadata/duckdb/duckdb.hml`. + +### Step 7. Update the environment variables + +The generated file has two environment variables — one for reads and one for writes — that you'll need to add to your subgraph's `.env.my_subgraph` file. +Each key is prefixed by the subgraph name, an underscore, and the name of the connector. Ensure the port value matches what is published in your connector's docker compose file. + +As an example: +```env +MY_SUBGRAPH_DUCKDB_READ_URL=http://local.hasura.dev: +MY_SUBGRAPH_DUCKDB_WRITE_URL=http://local.hasura.dev: ``` -{ - "data": { - "duckdb_sampleDataHnHackerNews": [ - { - "text": "Experimental Product? Hasura + React

Product with market fit and scale needs? Elixir/Phoenix", - "id": 31480758 - }, - { - "text": "I just skimmed through the article tbh, but I saw 14 GB and what looked like a predicate pushdown optimization. I think DuckDB could handle that on my 16GB mac.", - "id": 32182618 - }, - { - "text": "To what extent does this headache go away if autogenerating graphql from a relational db, using tools like Postgraphile or Hasura? I never considered making my "own" graphql service but those tools sure make it look easy to create a nice API controller through db migrations.", - "id": 31285405 - }, - { - "text": "Under what circumtances should I prefer to use DuckDB over SQLite or PostgreSQL? It is not clear.", - "id": 33285150 - } - ] - } -} + +These values are for the connector itself and utilize `local.hasura.dev` to ensure proper resolution within the docker container. + +### Step 8. Start the connector's Docker Compose + +Let's start our connector's Docker Compose file by running the following from inside the connector's subgraph: + +```bash +docker compose -f docker-compose.duckdb.yaml up ``` -### Setting up the DuckDB connector locally (Coming Soon) +### Step 9. Update the new `DataConnectorLink` object + +Finally, now that our `DataConnectorLink` has the correct environment variables configured for the connector, +we can run the update command to have the CLI look at the configuration JSON and transform it to reflect our database's +schema in `hml` format. In a new terminal tab, run: + +```bash +ddn connector-link update duckdb --subgraph my_subgraph +``` + +After this command runs, you can open your `my_subgraph/metadata/duckdb.hml` file and see your metadata completely +scaffolded out for you 🎉 + +## Documentation + +View the full documentation for the DuckDB connector [here](./docs/index.md). + +## Contributing + +Check out our [contributing guide](./docs/contributing.md) for more details. + +## License -Please keep an eye out for instructions on running things locally which will be coming soon. \ No newline at end of file +The DuckDB connector is available under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). \ No newline at end of file diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..9cf7910 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,109 @@ +# General Architecture of the DuckDB Connector + +## Query Engine +The query engine's job is to take a `QueryRequest`, which contains information about the query a user would like to run, translate it to DuckDB SQL (using the SQLite dialect), execute it against the database, and return the results as a `QueryResponse`. + +One place in particular that uses the Query Engine is the `/query` endpoint (defined in the `ndc-hub` repository). + +`/query` endpoints receives a `QueryRequest`, and calls the `plan_queries` function from the Query Engine in order to create a QueryPlan which includes the information needed to execute the query. It then calls the `perform_query` function using the QueryPlan (which is run against DuckDB) and gets back a `QueryResponse` which it can then return to the caller. + +API: + +```typescript +export async function plan_queries( + configuration: Configuration, + query: QueryRequest +): Promise +``` + +```typescript +async function perform_query( + state: State, + query_plans: SQLQuery[] +): Promise +``` + +Note that the response from this function should be in the format of an ndc-spec [QueryResponse](https://hasura.github.io/ndc-spec/reference/types.html#queryresponse) represented as JSON. + +### Query Planning +The query plan is essentially side-effect free - we use information from the request as well as the information about the metadata to translate the query request into a SQL statement to run against the database. + +This process is currently found in the [src/handlers](/src/handlers/query.ts) directory in the query.ts file. The API is the following function: + +```typescript +export async function plan_queries( + configuration: Configuration, + query: QueryRequest +): Promise +``` + +The `plan_queries` function returns a `SQLQuery[]` which functions as an execution plan. + +```typescript +export type SQLQuery = { + runSql: boolean; + runAgg: boolean; + sql: string; + args: any[]; + aggSql: string; + aggArgs: any[]; +}; +``` + +The incoming `QueryRequest` is used to construct a SQL statement by performing a recursive post-order walk over the QueryRequest to construct the SQL statement to be executed. The different pieces of the query are computed and then the query is constructed. + +```typescript + sql = wrap_rows(` + SELECT + JSON_OBJECT(${collect_rows.join(",")}) as r + FROM ${from_sql} + ${where_conditions.join(" AND ")} + ${order_by_sql} + ${limit_sql} + ${offset_sql} + `); +``` + +### Query Execution +The query execution creates a connection to the database and executes the query plan against the DuckDB or MotherDuck hosted database. It then returns the results from the query back to the caller of the function. + +```typescript +async function do_all(con: any, query: SQLQuery): Promise { + return new Promise((resolve, reject) => { + con.all(query.sql, ...query.args, function (err: any, res: any) { + if (err) { + reject(err); + } else { + resolve(res); + } + }); + }); +} + +async function perform_query( + state: State, + query_plans: SQLQuery[] +): Promise { + const con = state.client.connect(); + const response: RowSet[] = []; + for (let query_plan of query_plans) { + const res = await do_all(con, query_plan); + const row_set = JSON.parse(res[0]["data"] as string) as RowSet; + response.push(row_set); + } + return response; +} +``` + +## Patterns and guiding principles + +Here are a few ideas I have about working with this connector. + +### KISS (Keep it simple stupid!) +Robust and full-featured connector implementations should preferably be written in Rust for performance purposes. For Community Connectors it is preferred to try to keep things simple where possible, all we are doing is constructing a SQL query from the QueryRequest, which ultimately is manipulating a string. + + +### Consider the generated SQL +When working on a feature or fixing a bug, consider the generated SQL first. What does it currently look like? What should it look like? + +Construct what the desired query should be and make sure you can run it, and then work towards altering the SQL construction to match the hand-crafted query. diff --git a/docs/code-of-conduct.md b/docs/code-of-conduct.md new file mode 100644 index 0000000..4d6ab54 --- /dev/null +++ b/docs/code-of-conduct.md @@ -0,0 +1,74 @@ +# Hasura Community Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as +contributors and maintainers pledge to make participation in our project and +our community a harassment-free experience for everyone, regardless of age, body +size, disability, ethnicity, sex characteristics, gender identity and expression, +level of experience, education, socio-economic status, nationality, personal +appearance, race, religion, or sexual identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment +include: + +- Using welcoming, inclusive and gender-neutral language (example: instead of "Hey guys", + you could use "Hey folks" or "Hey all") +- Being respectful of differing viewpoints and experiences +- Gracefully accepting constructive criticism +- Focusing on what is best for the community +- Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +- The use of sexualized language or imagery and unwelcome sexual attention or + advances +- Trolling, insulting/derogatory comments, and personal or political attacks +- Public or private harassment +- Publishing others' private information, such as a physical or electronic + address, without explicit permission +- Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable +behavior and are expected to take appropriate and fair corrective action in +response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or +reject comments, commits, code, wiki edits, issues, and other contributions +that are not aligned to this Code of Conduct, or to ban temporarily or +permanently any contributor for other behaviors that they deem inappropriate, +threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces +when an individual is representing the project or its community. Examples of +representing a project or community include using an official project e-mail +address, posting via an official social media account, or acting as an appointed +representative at an online or offline event. Representation of a project may be +further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported by contacting the project team at community@hasura.io. All +complaints will be reviewed and investigated and will result in a response that +is deemed necessary and appropriate to the circumstances. The project team is +obligated to maintain confidentiality with regard to the reporter of an incident. +Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good +faith may face temporary or permanent repercussions as determined by other +members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, +available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html + +[homepage]: https://www.contributor-covenant.org \ No newline at end of file diff --git a/docs/contributing.md b/docs/contributing.md new file mode 100644 index 0000000..bcfd9cb --- /dev/null +++ b/docs/contributing.md @@ -0,0 +1,39 @@ +# Contributing + +_First_: if you feel insecure about how to start contributing, feel free to ask us on our +[Discord channel](https://discordapp.com/invite/hasura) in the #contrib channel. You can also just go ahead with your contribution and we'll give you feedback. Don't worry - the worst that can happen is that you'll be politely asked to change something. We appreciate any contributions, and we don't want a wall of rules to stand in the way of that. + +However, for those individuals who want a bit more guidance on the best way to contribute to the project, read on. This document will cover what we're looking for. By addressing the points below, the chances that we can quickly merge or address your contributions will increase. + +## 1. Code of conduct + +Please follow our [Code of conduct](./code-of-conduct.md) in the context of any contributions made to Hasura. + +## 2. CLA + +For all contributions, a CLA (Contributor License Agreement) needs to be signed +[here](https://cla-assistant.io/hasura/ndc-duckdb) before (or after) the pull request has been submitted. A bot will prompt contributors to sign the CLA via a pull request comment, if necessary. + +## 3. Ways of contributing + +### Reporting an Issue + +- Make sure you test against the latest released cloud version. It is possible that we may have already fixed the bug you're experiencing. +- Provide steps to reproduce the issue, including Database (e.g. Postgres) version and Hasura DDN version. +- Please include logs, if relevant. +- Create a [issue](https://github.com/hasura/ndc-duckdb/issues/new/choose). + +### Working on an issue + +- We use the [fork-and-branch git workflow](https://blog.scottlowe.org/2015/01/27/using-fork-branch-git-workflow/). +- Please make sure there is an issue associated with the work that you're doing. +- If you're working on an issue, please comment that you are doing so to prevent duplicate work by others also. +- Squash your commits and refer to the issue using `fix #` or `close #` in the commit message, at the end. For example: `resolve answers to everything (fix #42)` or `resolve answers to everything, fix #42` +- Rebase master with your branch before submitting a pull request. + +## 6. Commit messages + +- The first line should be a summary of the changes, not exceeding 50 characters, followed by an optional body which has more details about the changes. Refer to [this link](https://github.com/erlang/otp/wiki/writing-good-commit-messages) for more information on writing good commit messages. +- Use the imperative present tense: "add/fix/change", not "added/fixed/changed" nor "adds/fixes/changes". +- Don't capitalize the first letter of the summary line. +- Don't add a period/dot (.) at the end of the summary line. \ No newline at end of file diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..a90b43f --- /dev/null +++ b/docs/development.md @@ -0,0 +1,70 @@ +# DuckDB Connector Development + +### Prerequisites +1. Follow the steps in the [README](../README.md) +2. Install [NodeJS & npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) + +### Clone the repo + +In a new directory, clone the repo using: + +```git clone https://github.com/hasura/ndc-duckdb``` + +### Run the Introspection + +If you are working with a local database file, place the file in the root of the ndc-duckdb directory. + +Assuming the file is named: `chinook.db` you can run the introspection using: + +```DUCKDB_URL=chinook.db ts-node generate-config``` + +If you are working with a MotherDuck hosted database, you can run the introspection using: + +```DUCKDB_URL=md:?motherduck_token=eyJhbGc... ts-node generate-config``` + +This will generate a `config.json` file. + +### Run the Connector + +To start the connector on port 9094, for a local file named `chinook.db` run: +```HASURA_CONNECTOR_PORT=9094 DUCKDB_URL=chinook.db ts-node ./src/index.ts serve --configuration=.``` + +To start the connector on port 9094, for a MotherDuck hosted DuckDB instance run: +```HASURA_CONNECTOR_PORT=9094 DUCKDB_URL=md:?motherduck_token=eyJhbGc... ts-node ./src/index.ts serve --configuration=.``` + +### Attach the connector to the locally running engine + +There should a file located at `my_subgraph/.env.my_subgraph` that contains + +```env +MY_SUBGRAPH_DUCKDB_READ_URL=http://local.hasura.dev: +MY_SUBGRAPH_DUCKDB_WRITE_URL=http://local.hasura.dev: +``` + +Create a new .env file called `.env.my_subgraph.dev` and place the following values into it: + +```env +MY_SUBGRAPH_DUCKDB_READ_URL=http://local.hasura.dev:9094 +MY_SUBGRAPH_DUCKDB_WRITE_URL=http://local.hasura.dev:9094 +``` + +In your `supergraph.yaml` file change the env file to point to the dev file. + +``` + subgraphs: + my_subgraph: + generator: + rootPath: my_subgraph + # envFile: my_subgraph/.env.my_subgraph # Change the env file + envFile: my_subgraph/.env.my_subgraph.dev + includePaths: + - my_subgraph/metadata +``` + +Do a local supergraph build: + +```ddn supergraph build local --output-dir ./engine``` + +Mutations and Queries will now be issued against your locally running connector instance. + +Depending on the changes made to the connector, you may need to re-generate the config. The best way to do this is to regenerate the config locally then move that config into the supergraph. \ No newline at end of file diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..96768a7 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,13 @@ +# DuckDB Documentation + +The DuckDB connector is a [Hasura](https://hasura.io/) Native Data Connector. + +- [Usage](./usage/index.md) + - [Querying (for example)](./usage/querying.md) +- [Architecture](./architecture.md) +- [Development](./development.md) +- [Code of Conduct](./code-of-conduct.md) +- [Contributing](./contributing.md) +- [Production](./production.md) +- [Support](./support.md) +- [Security](./security.md) \ No newline at end of file diff --git a/docs/logo.svg b/docs/logo.svg new file mode 100644 index 0000000..c6d5d2f --- /dev/null +++ b/docs/logo.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/production.md b/docs/production.md new file mode 100644 index 0000000..5115ab0 --- /dev/null +++ b/docs/production.md @@ -0,0 +1,30 @@ +# Hasura DuckDB Connector in Production +We ship the DuckDB connectors as Docker images. + +You can look at the Dockerfile in the root of the repository to see how the image is built. + +The connector can be run via a docker-compose file: + +``` +services: + duckdb_duckdb: + build: + context: . + dockerfile_inline: |- + FROM ghcr.io/hasura/ndc-duckdb:v0.0.9 + COPY ./ /etc/connector + develop: + watch: + - path: ./ + action: sync+restart + target: /etc/connector + env_file: + - .env.local + extra_hosts: + - local.hasura.dev=host-gateway + ports: + - mode: ingress + target: 8080 + published: "8081" + protocol: tcp +``` \ No newline at end of file diff --git a/docs/security.md b/docs/security.md new file mode 100644 index 0000000..e6d4dd9 --- /dev/null +++ b/docs/security.md @@ -0,0 +1,30 @@ +## Reporting Vulnerabilities + +We’re extremely grateful for security researchers and users that report vulnerabilities to the Hasura Community. All reports are thoroughly investigated by a set of community volunteers and the Hasura team. + +To report a security issue, please email us at [security@hasura.io](mailto:security@hasura.io) with all the details, attaching all necessary information. + +### When Should I Report a Vulnerability? + +- You think you have discovered a potential security vulnerability in the Hasura GraphQL Engine or related components. +- You are unsure how a vulnerability affects the Hasura GraphQL Engine. +- You think you discovered a vulnerability in another project that Hasura GraphQL Engine depends on (e.g. Heroku, Docker, etc). +- You want to report any other security risk that could potentially harm Hasura GraphQL Engine users. + +### When Should I NOT Report a Vulnerability? + +- You need help tuning Hasura GraphQL Engine components for security. +- You need help applying security related updates. +- Your issue is not security related. + +## Security Vulnerability Response + +Each report is acknowledged and analyzed by the project's maintainers and the security team within 3 working days. + +The reporter will be kept updated at every stage of the issue's analysis and resolution (triage -> fix -> release). + +## Public Disclosure Timing + +A public disclosure date is negotiated by the Hasura product security team and the bug submitter. We prefer to fully disclose the bug as soon as possible once a user mitigation is available. It is reasonable to delay disclosure when the bug or the fix is not yet fully understood, the solution is not well-tested, or for vendor coordination. The timeframe for disclosure is from immediate (especially if it's already publicly known) to a few weeks. We expect the time-frame between a report to a public disclosure to typically be in the order of 7 days. The Hasura GraphQL Engine maintainers and the security team will take the final call on setting a disclosure date. + +(Some sections have been inspired and adapted from [https://github.com/kubernetes/website/blob/master/content/en/docs/reference/issues-security/security.md](https://github.com/kubernetes/website/blob/master/content/en/docs/reference/issues-security/security.md)). \ No newline at end of file diff --git a/docs/support.md b/docs/support.md new file mode 100644 index 0000000..f814927 --- /dev/null +++ b/docs/support.md @@ -0,0 +1,11 @@ +# Support & Troubleshooting + +The documentation and community will help you troubleshoot most issues. If you have encountered a bug or need to get in touch with us, you can contact us using one of the following channels: +* Support & feedback: [Discord](https://discord.gg/hasura) +* Issue & bug tracking: [GitHub issues](https://github.com/hasura/ndc=[connectorName]/issues) +* Follow product updates: [@HasuraHQ](https://twitter.com/hasurahq) +* Talk to us on our [website chat](https://hasura.io) + +We are committed to fostering an open and welcoming environment in the community. Please see the [Code of Conduct](code-of-conduct.md). + +If you want to report a security issue, please [read this](security.md). \ No newline at end of file diff --git a/docs/usage/index.md b/docs/usage/index.md new file mode 100644 index 0000000..7f9775a --- /dev/null +++ b/docs/usage/index.md @@ -0,0 +1,5 @@ +# DuckDB Usage Documentation + +Information on how to use the DuckDB connector. + +- [Querying](./querying.md) \ No newline at end of file diff --git a/docs/usage/querying.md b/docs/usage/querying.md new file mode 100644 index 0000000..2f095ec --- /dev/null +++ b/docs/usage/querying.md @@ -0,0 +1,128 @@ +# Querying Example + +Hasura instrospects the database and allows for Querying a pre-existing DuckDB database. + +If you want to try these queries out yourself, please see [this supergraph](https://github.com/hasura/super_supergraph/tree/main). + +These examples use [this DuckDB file](https://github.com/hasura/super_supergraph/blob/main/duckdb/connector/duckdb/chinook.db) which contains data from the Chinook Music database. + +Fetch all Albums + +```graphql +query Query { + duckdb_chinookMainAlbum { + albumId + artistId + title + } +} +``` + +Fetch all Albums where Album ID = 1 + +```graphql +query Query { + duckdb_chinookMainAlbum(where: {albumId: {_eq: 1}}) { + albumId + artistId + title + } +} +``` + +Fetch all Albums order by Title + +```graphql +query Query { + duckdb_chinookMainAlbum(order_by: {title: Asc}) { + albumId + artistId + title + } +} +``` + +The DuckDB connector takes advantage of duck-typing to add relationships. When you add a relationship in your Hasura metadata, Hasura will assume that the relationships exist and generate SQL that includes JOIN's and also enables filtering across joins. + +By adding some relationships in the HML: + +``` +--- +kind: Relationship +version: v1 +definition: + name: artist + source: ChinookMainAlbum + target: + model: + name: ChinookMainArtist + relationshipType: Object + mapping: + - source: + fieldPath: + - fieldName: artistId + target: + modelField: + - fieldName: artistId + +--- +kind: Relationship +version: v1 +definition: + name: tracks + source: ChinookMainAlbum + target: + model: + name: ChinookMainTrack + relationshipType: Array + mapping: + - source: + fieldPath: + - fieldName: albumId + target: + modelField: + - fieldName: albumId + +``` + +We can now query with relationships: + +```graphql +query Query { + duckdb_chinookMainAlbum { + albumId + artistId + title + artist { + artistId + name + } + tracks { + albumId + trackId + name + } + } +} +``` + +You can also filter across relationships as well: + +```graphql +query Query { + duckdb_chinookMainAlbum(where: {artist: {artistId: {_eq: 10}}}) { + albumId + artistId + title + artist { + artistId + name + } + } +} +``` + +### Additional Querying Capabilities that will be coming in the future + +* Aggregations +* GroupBy \ No newline at end of file