Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Skeleton project for destination MSSQL V2 #49460

Merged
merged 9 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions airbyte-integrations/connectors/destination-mssql-v2/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Microsoft SQL Server V2 (Bulk CDK) Destination

## Build

### airbyte-ci

To build the connector via the [Airbyte CI CLI tool](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md), navigate to the root of the [Airbyte repository](https://github.com/airbytehq/airbyte) and execute the following command:

```shell
> airbyte-ci connectors --name=destination-mssql-v2 build
```

### Gradle

To build the connector via [Gradle](https://gradle.org/), navigate to the root of the [Airbyte repository](https://github.com/airbytehq/airbyte) and execute the following command:

```shell
> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:build
```
## Execute

### Local Execution via Docker

In order to run the connector image locally, first either build the connector's [Docker](https://www.docker.com/) image using the commands found
in this section of this document OR build the image using the following command:

```shell
> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:buildConnectorImage
```

The built image will automatically be tagged with the `dev` label. To run the connector image, use the following commands:

```shell
docker run --rm airbyte/destination-mssql-v2:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-mssql-v2:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-mssql-v2:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-mssql-v2:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```

## Test

The connector contains both unit and acceptance tests which can each be executed from the local environment.

### Unit Tests

The connector uses a combination of [Kotlin](https://kotlinlang.org/), [JUnit 5](https://junit.org/junit5/) and [MockK](https://mockk.io/)
to implement unit tests. Existing tests can be found within the destination-mssql-v2 module in the conventional `src/test/kotlin` source folder. New tests should also be added to this location.

The unit tests can be executed either via the [Airbyte CI CLI tool](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) or [Gradle](https://gradle.org/):

###### Airbyte CI CLI
```shell
> airbyte-ci connectors --name=destination-mssql-v2 test
```

###### Gradle
```shell
> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:test
```

### Acceptance Tests

The [Airbyte project](https://github.com/airbytehq/airbyte) a standard test suite that all destination connectors must pass. The tests require specific implementations of a few components in order to connect the acceptance test suite with the connector's specific logic. The existing acceptance test scaffolding can be found in the conventional `src/test-integration/kotlin` source folder.

The acceptance tests can be executed either via the [Airbyte CI CLI tool](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) or [Gradle](https://gradle.org/):

###### Airbyte CI CLI
```shell
> airbyte-ci connectors --name=destination-mssql-v2 test
```

###### Gradle
```shell
> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:integrationTest
```

## Release

### Publishing a new version of the connector

You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?

1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=destination-mssql-v2 test`
2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors).
3. Make sure the `metadata.yaml` content is up to date.
4. Make the connector documentation and its changelog is up to date (`docs/integrations/destinations/mssql-v2.md`).
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention).
6. Pat yourself on the back for being an awesome contributor.
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.


Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
/*
* Copyright (c) 2024 Airbyte, Inc., all rights reserved.
*/

plugins {
id("application")
id("airbyte-bulk-connector")
}

airbyteBulkConnector {
core = "load"
toolkits = listOf()
cdk = "local"
}

application {
mainClass = "io.airbyte.integrations.destination.mssql.v2.MSSQLDestination"

applicationDefaultJvmArgs = listOf("-XX:+ExitOnOutOfMemoryError", "-XX:MaxRAMPercentage=75.0")

// Uncomment and replace to run locally
//applicationDefaultJvmArgs = listOf("-XX:+ExitOnOutOfMemoryError", "-XX:MaxRAMPercentage=75.0", "--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED", "--add-opens", "java.base/sun.security.action=ALL-UNNAMED", "--add-opens", "java.base/java.lang=ALL-UNNAMED")
}

val junitVersion = "5.11.3"

configurations.configureEach {
// Exclude additional SLF4J providers from all classpaths
exclude(mapOf("group" to "org.slf4j", "module" to "slf4j-reload4j"))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nonblocking, did we ever find where this dep is coming from? and/or we could add this exclude to the plugin (airbyte-bulk-connector.gradle)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@edgao I would have to check. It may not be an issue in this destination (copy pasta here). If I recall correctly, it may have been one of the hadoop dependencies that we got via Iceberg and maybe even via the CDK because of parquet.

}

// Uncomment to run locally
//tasks.run.configure {
// standardInput = System.`in`
//}

dependencies {
implementation("com.microsoft.sqlserver:mssql-jdbc:12.8.1.jre11")
implementation("io.github.oshai:kotlin-logging-jvm:7.0.0")
implementation("jakarta.inject:jakarta.inject-api:2.0.1")
implementation("com.github.spotbugs:spotbugs-annotations:4.8.6")
implementation("io.micronaut:micronaut-inject:4.6.1")

testImplementation("io.mockk:mockk:1.13.13")
testImplementation("org.junit.jupiter:junit-jupiter-api:$junitVersion")
testImplementation("org.junit.jupiter:junit-jupiter-params:$junitVersion")
testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine:$junitVersion")
}

tasks.named<Test>("test") {
systemProperties(mapOf("mockk.junit.extension.keepmocks" to "true", "mockk.junit.extension.requireParallelTesting" to "true"))
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
testExecutionConcurrency=-1
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
34 changes: 34 additions & 0 deletions airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
data:
connectorSubtype: database
connectorType: destination
definitionId: 37a928c1-2d5c-431a-a97d-ae236bd1ea0c
dockerImageTag: 0.1.0
dockerRepository: airbyte/destination-mssql-v2
githubIssueLabel: destination-mssql-v2
icon: icon.svg
license: ELv2
name: MSSQL V2 Destination
registryOverrides:
cloud:
enabled: false
oss:
enabled: false
releaseStage: alpha
documentationUrl: https://docs.airbyte.com/integrations/destinations/mssql-v2
tags:
- language:java
ab_internal:
sl: 100
ql: 100
supportLevel: community
supportsRefreshes: true
connectorTestSuitesOptions:
- suite: unitTests
- suite: integrationTests
testSecrets:
- name: SECRET_DESTINATION-S3-V2-MINIMAL-REQUIRED-CONFIG
fileName: s3_dest_v2_minimal_required_config.json
secretStore:
type: GSM
alias: airbyte-connector-testing-secret-store
metadataSpecVersion: "1.0"
Loading
Loading