From 32017be8dad60e67a7e5cb33c4c5200be39b81cc Mon Sep 17 00:00:00 2001 From: Jonathan Pearlin Date: Fri, 13 Dec 2024 09:20:51 -0500 Subject: [PATCH 1/6] chore: Skeleton project for destination MSSQL V2 --- .../connectors/destination-mssql-v2/README.md | 91 +++++++++++++++++++ .../destination-mssql-v2/build.gradle.kts | 52 +++++++++++ .../destination-mssql-v2/gradle.properties | 1 + .../connectors/destination-mssql-v2/icon.svg | 1 + .../destination-mssql-v2/metadata.yaml | 34 +++++++ .../destination/mssql/v2/MSSQLDestination.kt | 14 +++ docs/integrations/destinations/mssql-v2.md | 12 +++ settings.gradle | 2 +- 8 files changed, 206 insertions(+), 1 deletion(-) create mode 100644 airbyte-integrations/connectors/destination-mssql-v2/README.md create mode 100644 airbyte-integrations/connectors/destination-mssql-v2/build.gradle.kts create mode 100644 airbyte-integrations/connectors/destination-mssql-v2/gradle.properties create mode 100644 airbyte-integrations/connectors/destination-mssql-v2/icon.svg create mode 100644 airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml create mode 100644 airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/MSSQLDestination.kt create mode 100644 docs/integrations/destinations/mssql-v2.md diff --git a/airbyte-integrations/connectors/destination-mssql-v2/README.md b/airbyte-integrations/connectors/destination-mssql-v2/README.md new file mode 100644 index 000000000000..4f2fe1572dfa --- /dev/null +++ b/airbyte-integrations/connectors/destination-mssql-v2/README.md @@ -0,0 +1,91 @@ +# Microsoft SQL Server V2 (Bulk CDK) Destination + +## Build + +### airbyte-ci + +To build the connector via the [Airbyte CI CLI tool](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md), navigate to the root of the [Airbyte repository](https://github.com/airbytehq/airbyte) and execute the following command: + +```shell +> airbyte-ci connectors --name=destination-mssql-v2 build +``` + +### Gradle + +To build the connector via [Gradle](https://gradle.org/), navigate to the root of the [Airbyte repository](https://github.com/airbytehq/airbyte) and execute the following command: + +```shell +> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:build +``` +## Execute + +### Local Execution via Docker + +In order to run the connector image locally, first either build the connector's [Docker](https://www.docker.com/) image using the commands found +in this section of this document OR build the image using the following command: + +```shell +> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:buildConnectorImage +``` + +The built image will automatically be tagged with the `dev` label. To run the connector image, use the following commands: + +```shell +docker run --rm airbyte/destination-mssql-v2:dev spec +docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-mssql-v2:dev check --config /secrets/config.json +docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-mssql-v2:dev discover --config /secrets/config.json +docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-mssql-v2:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json +``` + +## Test + +The connector contains both unit and acceptance tests which can each be executed from the local environment. + +### Unit Tests + +The connector uses a combination of [Kotlin](https://kotlinlang.org/), [JUnit 5](https://junit.org/junit5/) and [MockK](https://mockk.io/) +to implement unit tests. Existing tests can be found within the destination-mssql-v2 module in the conventional `src/test/kotlin` source folder. New tests should also be added to this location. + +The unit tests can be executed either via the [Airbyte CI CLI tool](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) or [Gradle](https://gradle.org/): + +###### Airbyte CI CLI +```shell +> airbyte-ci connectors --name=destination-mssql-v2 test +``` + +###### Gradle +```shell +> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:test +``` + +### Acceptance Tests + +The [Airbyte project](https://github.com/airbytehq/airbyte) a standard test suite that all destination connectors must pass. The tests require specific implementations of a few components in order to connect the acceptance test suite with the connector's specific logic. The existing acceptance test scaffolding can be found in the conventional `src/test-integration/kotlin` source folder. + +The acceptance tests can be executed either via the [Airbyte CI CLI tool](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) or [Gradle](https://gradle.org/): + +###### Airbyte CI CLI +```shell +> airbyte-ci connectors --name=destination-mssql-v2 test +``` + +###### Gradle +```shell +> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:integrationTest +``` + +## Release + +### Publishing a new version of the connector + +You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what? + +1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=destination-mssql-v2 test` +2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors). +3. Make sure the `metadata.yaml` content is up to date. +4. Make the connector documentation and its changelog is up to date (`docs/integrations/destinations/mssql-v2.md`). +5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention). +6. Pat yourself on the back for being an awesome contributor. +7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master. + + diff --git a/airbyte-integrations/connectors/destination-mssql-v2/build.gradle.kts b/airbyte-integrations/connectors/destination-mssql-v2/build.gradle.kts new file mode 100644 index 000000000000..4cc0467ff7bd --- /dev/null +++ b/airbyte-integrations/connectors/destination-mssql-v2/build.gradle.kts @@ -0,0 +1,52 @@ +/* + * Copyright (c) 2024 Airbyte, Inc., all rights reserved. + */ + +plugins { + id("application") + id("airbyte-bulk-connector") +} + +airbyteBulkConnector { + core = "load" + toolkits = listOf() + cdk = "local" +} + +application { + mainClass = "io.airbyte.integrations.destination.mssql.v2.MSSQLDestination" + + applicationDefaultJvmArgs = listOf("-XX:+ExitOnOutOfMemoryError", "-XX:MaxRAMPercentage=75.0") + + // Uncomment and replace to run locally + //applicationDefaultJvmArgs = listOf("-XX:+ExitOnOutOfMemoryError", "-XX:MaxRAMPercentage=75.0", "--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED", "--add-opens", "java.base/sun.security.action=ALL-UNNAMED", "--add-opens", "java.base/java.lang=ALL-UNNAMED") +} + +val junitVersion = "5.11.3" + +configurations.configureEach { + // Exclude additional SLF4J providers from all classpaths + exclude(mapOf("group" to "org.slf4j", "module" to "slf4j-reload4j")) +} + +// Uncomment to run locally +//tasks.run.configure { +// standardInput = System.`in` +//} + +dependencies { + implementation("com.microsoft.sqlserver:mssql-jdbc:12.8.1.jre11") + implementation("io.github.oshai:kotlin-logging-jvm:7.0.0") + implementation("jakarta.inject:jakarta.inject-api:2.0.1") + implementation("com.github.spotbugs:spotbugs-annotations:4.8.6") + implementation("io.micronaut:micronaut-inject:4.6.1") + + testImplementation("io.mockk:mockk:1.13.13") + testImplementation("org.junit.jupiter:junit-jupiter-api:$junitVersion") + testImplementation("org.junit.jupiter:junit-jupiter-params:$junitVersion") + testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine:$junitVersion") +} + +tasks.named("test") { + systemProperties(mapOf("mockk.junit.extension.keepmocks" to "true", "mockk.junit.extension.requireParallelTesting" to "true")) +} diff --git a/airbyte-integrations/connectors/destination-mssql-v2/gradle.properties b/airbyte-integrations/connectors/destination-mssql-v2/gradle.properties new file mode 100644 index 000000000000..4dbe8b8729df --- /dev/null +++ b/airbyte-integrations/connectors/destination-mssql-v2/gradle.properties @@ -0,0 +1 @@ +testExecutionConcurrency=-1 diff --git a/airbyte-integrations/connectors/destination-mssql-v2/icon.svg b/airbyte-integrations/connectors/destination-mssql-v2/icon.svg new file mode 100644 index 000000000000..edcaeb77c8f2 --- /dev/null +++ b/airbyte-integrations/connectors/destination-mssql-v2/icon.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml b/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml new file mode 100644 index 000000000000..1b09be9a1ad8 --- /dev/null +++ b/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml @@ -0,0 +1,34 @@ +data: + connectorSubtype: database + connectorType: destination + definitionId: 37a928c1-2d5c-431a-a97d-ae236bd1ea0c + dockerImageTag: 0.1.0 + dockerRepository: airbyte/destination-mssql-v2 + githubIssueLabel: destination-mssql-v2 + icon: icon.svg + license: ELv2 + name: MSSQL V2 Destination + registryOverrides: + cloud: + enabled: false + oss: + enabled: false + releaseStage: alpha + documentationUrl: https://docs.airbyte.com/integrations/destinations/mssql-v2 + tags: + - language:java + ab_internal: + sl: 100 + ql: 100 + supportLevel: community + supportsRefreshes: true + connectorTestSuitesOptions: + - suite: unitTests + - suite: integrationTests + testSecrets: + - name: SECRET_DESTINATION-S3-V2-MINIMAL-REQUIRED-CONFIG + fileName: s3_dest_v2_minimal_required_config.json + secretStore: + type: GSM + alias: airbyte-connector-testing-secret-store +metadataSpecVersion: "1.0" diff --git a/airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/MSSQLDestination.kt b/airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/MSSQLDestination.kt new file mode 100644 index 000000000000..cb1d288390aa --- /dev/null +++ b/airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/MSSQLDestination.kt @@ -0,0 +1,14 @@ +/* + * Copyright (c) 2024 Airbyte, Inc., all rights reserved. + */ + +package io.airbyte.integrations.destination.mssql.v2 + +import io.airbyte.cdk.AirbyteDestinationRunner + +object MSSQLDestination { + @JvmStatic + fun main(args: Array) { + AirbyteDestinationRunner.run(*args) + } +} diff --git a/docs/integrations/destinations/mssql-v2.md b/docs/integrations/destinations/mssql-v2.md new file mode 100644 index 000000000000..2f793828ec3e --- /dev/null +++ b/docs/integrations/destinations/mssql-v2.md @@ -0,0 +1,12 @@ +# MSSQL (V2) + +## Changelog + +
+ Expand to review + +| Version | Date | Pull Request | Subject | +|:--------|:-----------| :--------------------------------------------------------- | :-------------------------------------------------------------------------------------------------- | +| 0.1.0 | 2024-12-13 | | | + +
\ No newline at end of file diff --git a/settings.gradle b/settings.gradle index f89f86c38e7d..2568ec4a481f 100644 --- a/settings.gradle +++ b/settings.gradle @@ -176,7 +176,7 @@ if (bulkCdkToolkitPath.toFile().exists()) { // Include all java connector projects. def integrationsPath = rootDir.toPath().resolve('airbyte-integrations/connectors') integrationsPath.eachDir { dir -> - def buildFiles = file(dir).list { file, name -> name == "build.gradle" } + def buildFiles = file(dir).list { file, name -> name.matches("build\\.gradle(?:\\.kts)?") } if (buildFiles.length != 1) { // Ignore python and other non-gradle connectors. return From dda61f62475c0df2fe00396d8b4f3445ba81ebf6 Mon Sep 17 00:00:00 2001 From: Jonathan Pearlin Date: Fri, 13 Dec 2024 15:06:18 -0500 Subject: [PATCH 2/6] Remove doc for now --- docs/integrations/destinations/mssql-v2.md | 12 ------------ 1 file changed, 12 deletions(-) delete mode 100644 docs/integrations/destinations/mssql-v2.md diff --git a/docs/integrations/destinations/mssql-v2.md b/docs/integrations/destinations/mssql-v2.md deleted file mode 100644 index 2f793828ec3e..000000000000 --- a/docs/integrations/destinations/mssql-v2.md +++ /dev/null @@ -1,12 +0,0 @@ -# MSSQL (V2) - -## Changelog - -
- Expand to review - -| Version | Date | Pull Request | Subject | -|:--------|:-----------| :--------------------------------------------------------- | :-------------------------------------------------------------------------------------------------- | -| 0.1.0 | 2024-12-13 | | | - -
\ No newline at end of file From 05d37faa742dbe25129a4af418ed6635468f0836 Mon Sep 17 00:00:00 2001 From: Jonathan Pearlin Date: Mon, 16 Dec 2024 10:45:33 -0500 Subject: [PATCH 3/6] Add stub configuration classes so build passes --- .../mssql/v2/config/MSSQLConfiguration.kt | 28 +++++++++++++++++++ .../mssql/v2/config/MSSQLSpecification.kt | 26 +++++++++++++++++ 2 files changed, 54 insertions(+) create mode 100644 airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/config/MSSQLConfiguration.kt create mode 100644 airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/config/MSSQLSpecification.kt diff --git a/airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/config/MSSQLConfiguration.kt b/airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/config/MSSQLConfiguration.kt new file mode 100644 index 000000000000..ca468c08caa7 --- /dev/null +++ b/airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/config/MSSQLConfiguration.kt @@ -0,0 +1,28 @@ +/* + * Copyright (c) 2024 Airbyte, Inc., all rights reserved. + */ + +package io.airbyte.integrations.destination.mssql.v2.config + +import dagger.Component.Factory +import io.airbyte.cdk.load.command.DestinationConfiguration +import io.airbyte.cdk.load.command.DestinationConfigurationFactory +import jakarta.inject.Singleton + +data class MSSQLConfiguration(val placeholder: String) : DestinationConfiguration() + +@Singleton +class MSSQLConfigurationFactory : + DestinationConfigurationFactory { + override fun makeWithoutExceptionHandling(pojo: MSSQLSpecification): MSSQLConfiguration { + TODO("Not yet implemented") + } +} + +@Factory +class MSSQLConfigurationProvider(private val config: DestinationConfiguration) { + @Singleton + fun get(): MSSQLConfiguration { + return config as MSSQLConfiguration + } +} diff --git a/airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/config/MSSQLSpecification.kt b/airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/config/MSSQLSpecification.kt new file mode 100644 index 000000000000..30a7f9769c63 --- /dev/null +++ b/airbyte-integrations/connectors/destination-mssql-v2/src/main/kotlin/io/airbyte/integrations/destination/mssql/v2/config/MSSQLSpecification.kt @@ -0,0 +1,26 @@ +/* + * Copyright (c) 2024 Airbyte, Inc., all rights reserved. + */ + +package io.airbyte.integrations.destination.mssql.v2.config + +import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle +import io.airbyte.cdk.command.ConfigurationSpecification +import io.airbyte.cdk.load.spec.DestinationSpecificationExtension +import io.airbyte.protocol.models.v0.DestinationSyncMode +import jakarta.inject.Singleton + +@Singleton +@JsonSchemaTitle("MSSQL V2 Destination Specification") +class MSSQLSpecification : ConfigurationSpecification() {} + +@Singleton +class MSSQLSpecificationExtension : DestinationSpecificationExtension { + override val supportedSyncModes = + listOf( + DestinationSyncMode.OVERWRITE, + DestinationSyncMode.APPEND, + DestinationSyncMode.APPEND_DEDUP + ) + override val supportsIncremental = true +} From 6ce321a348742780f3065390064e6d1c945ad899 Mon Sep 17 00:00:00 2001 From: Jonathan Pearlin Date: Mon, 16 Dec 2024 12:01:09 -0500 Subject: [PATCH 4/6] Fix metadata --- .../connectors/destination-mssql-v2/metadata.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml b/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml index 1b09be9a1ad8..9a9ab96d1c90 100644 --- a/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml +++ b/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml @@ -6,6 +6,7 @@ data: dockerRepository: airbyte/destination-mssql-v2 githubIssueLabel: destination-mssql-v2 icon: icon.svg + language: java license: ELv2 name: MSSQL V2 Destination registryOverrides: @@ -14,7 +15,7 @@ data: oss: enabled: false releaseStage: alpha - documentationUrl: https://docs.airbyte.com/integrations/destinations/mssql-v2 + documentationUrl: https://docs.airbyte.com/integrations/destinations/mssql tags: - language:java ab_internal: From 005f9a35510ccef2a552ab9a9a8f466ae57b4ae8 Mon Sep 17 00:00:00 2001 From: Jonathan Pearlin Date: Mon, 16 Dec 2024 12:21:24 -0500 Subject: [PATCH 5/6] Remove language element --- .../connectors/destination-mssql-v2/metadata.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml b/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml index 9a9ab96d1c90..6b5a1df0b22f 100644 --- a/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml +++ b/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml @@ -6,7 +6,6 @@ data: dockerRepository: airbyte/destination-mssql-v2 githubIssueLabel: destination-mssql-v2 icon: icon.svg - language: java license: ELv2 name: MSSQL V2 Destination registryOverrides: From 06804c76d009f411054c0113e36fc4e3ca6ab5e5 Mon Sep 17 00:00:00 2001 From: Jonathan Pearlin Date: Mon, 16 Dec 2024 12:54:38 -0500 Subject: [PATCH 6/6] Add public docs --- .../connectors/destination-mssql-v2/metadata.yaml | 2 +- docs/integrations/destinations/mssql-v2.md | 12 ++++++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) create mode 100644 docs/integrations/destinations/mssql-v2.md diff --git a/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml b/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml index 6b5a1df0b22f..1b09be9a1ad8 100644 --- a/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml +++ b/airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml @@ -14,7 +14,7 @@ data: oss: enabled: false releaseStage: alpha - documentationUrl: https://docs.airbyte.com/integrations/destinations/mssql + documentationUrl: https://docs.airbyte.com/integrations/destinations/mssql-v2 tags: - language:java ab_internal: diff --git a/docs/integrations/destinations/mssql-v2.md b/docs/integrations/destinations/mssql-v2.md new file mode 100644 index 000000000000..0e168a4ea5b7 --- /dev/null +++ b/docs/integrations/destinations/mssql-v2.md @@ -0,0 +1,12 @@ +# MSSQL (V2) + +## Changelog + +
+ Expand to review + +| Version | Date | Pull Request | Subject | +|:--------|:-----------| :--------------------------------------------------------- |:---------------| +| 0.1.0 | 2024-12-16 | [\#49460](https://github.com/airbytehq/airbyte/pull/49460) | Initial commit | + +
\ No newline at end of file