Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add sql component #193

Merged
merged 13 commits into from
Jul 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
190 changes: 190 additions & 0 deletions data/sql/v0/README.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
---
title: "SQL"
lang: "en-US"
draft: false
description: "Learn about how to set up a VDP SQL component https://github.com/instill-ai/instill-core"
---

The SQL component is a data component that allows users to access the SQL database of your choice.
It can carry out the following tasks:

- [Insert](#insert)
- [Update](#update)
- [Select](#select)
- [Delete](#delete)
- [Create Table](#create-table)
- [Drop Table](#drop-table)



## Release Stage

`Alpha`



## Configuration

The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/data/sql/v0/config/definition.json).




## Setup


| Field | Field ID | Type | Note |
| :--- | :--- | :--- | :--- |
| Username (required) | `user` | string | Fill in your account username |
| Password (required) | `password` | string | Fill in your account password |
| Database Name (required) | `database-name` | string | Fill in the name of your database |
| Host (required) | `host` | string | Fill in the host of your database |
| Port (required) | `port` | number | Fill in the port of your database |




## Supported Tasks

### Insert

Perform an insert operation based on specified filter


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_INSERT` |
| Engine (required) | `engine` | string | Choose the engine of your database |
| Table Name (required) | `table-name` | string | The table name in the database to insert data into |
| Data (required) | `data` | semi-structured/json | The data to be inserted |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Insert status |






### Update

Perform an update operation based on specified filter


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_UPDATE` |
| Engine (required) | `engine` | string | Choose the engine of your database |
| Table Name (required) | `table-name` | string | The table name in the database to update data into |
| Filter (required) | `filter` | string | The filter to be applied to the data with SQL syntax, which starts with WHERE clause |
| Update (required) | `update-data` | semi-structured/json | The new data to be updated to |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Update status |






### Select

Perform a select operation based on specified filter


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_SELECT` |
| Engine (required) | `engine` | string | Choose the engine of your database |
| Table Name (required) | `table-name` | string | The table name in the database to be selected |
| Filter | `filter` | string | The filter to be applied to the data with SQL syntax, which starts with WHERE clause, empty for all rows |
| Limit | `limit` | integer | The limit of rows to be selected, empty for all rows |
| Columns | `columns` | array[string] | The columns to return in the rows. If empty then all columns will be returned |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Rows | `rows` | array | The rows returned from the select operation |
| Status | `status` | string | Select status |






### Delete

Perform a delete operation based on specified filter


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DELETE` |
| Engine (required) | `engine` | string | Choose the engine of your database |
| Table Name (required) | `table-name` | string | The table name in the database to be deleted |
| Filter (required) | `filter` | string | The filter to be applied to the data with SQL syntax, which starts with WHERE clause |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Delete status |






### Create Table

Create a table in the database


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_CREATE_TABLE` |
| Engine (required) | `engine` | string | Choose the engine of your database |
| Table Name (required) | `table-name` | string | The table name in the database to be created |
| Columns (required) | `columns-structure` | semi-structured/json | The columns structure to be created in the table, json with value string, e.g \{"name": "VARCHAR(255)", "age": "INT not null"\} |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Create table status |






### Drop Table

Drop a table in the database


| Input | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Task ID (required) | `task` | string | `TASK_DROP_TABLE` |
| Engine (required) | `engine` | string | Choose the engine of your database |
| Table Name (required) | `table-name` | string | The table name in the database to be dropped |



| Output | ID | Type | Description |
| :--- | :--- | :--- | :--- |
| Status | `status` | string | Drop table status |







6 changes: 6 additions & 0 deletions data/sql/v0/assets/sql.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
91 changes: 91 additions & 0 deletions data/sql/v0/client.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
package sql

import (
"fmt"
"strconv"

"github.com/jmoiron/sqlx"
"google.golang.org/protobuf/types/known/structpb"

// Import all the SQL drivers
_ "github.com/denisenkom/go-mssqldb" // SQL Server
_ "github.com/go-sql-driver/mysql" // MySQL and MariaDB
_ "github.com/lib/pq" // PostgreSQL
_ "github.com/nakagami/firebirdsql" // Firebird
_ "github.com/sijms/go-ora/v2" // Oracle
)

var engines = map[string]string{
"PostgreSQL": "postgresql://%s:%s@%s/%s", // PostgreSQL
"SQL Server": "sqlserver://%s:%s@%s?database=%s", // SQL Server
"Oracle": "oracle://%s:%s@%s/%s", // Oracle
"MySQL": "%s:%s@tcp(%s)/%s", // MySQL and MariaDB
"Firebird": "firebirdsql://%s:%s@%s/%s", // Firebird
}

var enginesType = map[string]string{
"PostgreSQL": "postgres", // PostgreSQL
"SQL Server": "sqlserver", // SQL Server
"Oracle": "oracle", // Oracle
"MySQL": "mysql", // MySQL and MariaDB
"Firebird": "firebirdsql", // Firebird
}

type Config struct {
DBUser string
DBPassword string
DBName string
DBHost string
DBPort string
}

func LoadConfig(setup *structpb.Struct) *Config {
return &Config{
DBUser: getUser(setup),
DBPassword: getPassword(setup),
DBName: getDatabaseName(setup),
DBHost: getHost(setup),
DBPort: getPort(setup),
}
}

func newClient(setup *structpb.Struct, inputSetup *Engine) SQLClient {
cfg := LoadConfig(setup)

DBEndpoint := fmt.Sprintf("%v:%v", cfg.DBHost, cfg.DBPort)

// Test every engines to find the correct one
var db *sqlx.DB
var err error

// Get the correct engine
engine := engines[inputSetup.DBEngine]
engineType := enginesType[inputSetup.DBEngine]

dsn := fmt.Sprintf(engine,
cfg.DBUser, cfg.DBPassword, DBEndpoint, cfg.DBName,
)

db, err = sqlx.Open(engineType, dsn)
if err != nil {
return nil
}

return db
}

func getUser(setup *structpb.Struct) string {
return setup.GetFields()["user"].GetStringValue()
}
func getPassword(setup *structpb.Struct) string {
return setup.GetFields()["password"].GetStringValue()
}
func getDatabaseName(setup *structpb.Struct) string {
return setup.GetFields()["database-name"].GetStringValue()
}
func getHost(setup *structpb.Struct) string {
return setup.GetFields()["host"].GetStringValue()
}
func getPort(setup *structpb.Struct) string {
return strconv.Itoa(int(setup.GetFields()["port"].GetNumberValue()))
}
Loading
Loading