-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: Skeleton project for destination MSSQL V2 #49460
Merged
Merged
Changes from 3 commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
32017be
chore: Skeleton project for destination MSSQL V2
jdpgrailsdev 1389d9f
Merge branch 'master' into jonathan/destination-mssql-v2-skeleton
jdpgrailsdev dda61f6
Remove doc for now
jdpgrailsdev 05d37fa
Add stub configuration classes so build passes
jdpgrailsdev 76586dd
Merge branch 'master' into jonathan/destination-mssql-v2-skeleton
jdpgrailsdev 6ce321a
Fix metadata
jdpgrailsdev 005f9a3
Remove language element
jdpgrailsdev 06804c7
Add public docs
jdpgrailsdev 7a0bcfc
Merge branch 'master' into jonathan/destination-mssql-v2-skeleton
jdpgrailsdev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
91 changes: 91 additions & 0 deletions
91
airbyte-integrations/connectors/destination-mssql-v2/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
# Microsoft SQL Server V2 (Bulk CDK) Destination | ||
|
||
## Build | ||
|
||
### airbyte-ci | ||
|
||
To build the connector via the [Airbyte CI CLI tool](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md), navigate to the root of the [Airbyte repository](https://github.com/airbytehq/airbyte) and execute the following command: | ||
|
||
```shell | ||
> airbyte-ci connectors --name=destination-mssql-v2 build | ||
``` | ||
|
||
### Gradle | ||
|
||
To build the connector via [Gradle](https://gradle.org/), navigate to the root of the [Airbyte repository](https://github.com/airbytehq/airbyte) and execute the following command: | ||
|
||
```shell | ||
> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:build | ||
``` | ||
## Execute | ||
|
||
### Local Execution via Docker | ||
|
||
In order to run the connector image locally, first either build the connector's [Docker](https://www.docker.com/) image using the commands found | ||
in this section of this document OR build the image using the following command: | ||
|
||
```shell | ||
> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:buildConnectorImage | ||
``` | ||
|
||
The built image will automatically be tagged with the `dev` label. To run the connector image, use the following commands: | ||
|
||
```shell | ||
docker run --rm airbyte/destination-mssql-v2:dev spec | ||
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-mssql-v2:dev check --config /secrets/config.json | ||
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-mssql-v2:dev discover --config /secrets/config.json | ||
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-mssql-v2:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json | ||
``` | ||
|
||
## Test | ||
|
||
The connector contains both unit and acceptance tests which can each be executed from the local environment. | ||
|
||
### Unit Tests | ||
|
||
The connector uses a combination of [Kotlin](https://kotlinlang.org/), [JUnit 5](https://junit.org/junit5/) and [MockK](https://mockk.io/) | ||
to implement unit tests. Existing tests can be found within the destination-mssql-v2 module in the conventional `src/test/kotlin` source folder. New tests should also be added to this location. | ||
|
||
The unit tests can be executed either via the [Airbyte CI CLI tool](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) or [Gradle](https://gradle.org/): | ||
|
||
###### Airbyte CI CLI | ||
```shell | ||
> airbyte-ci connectors --name=destination-mssql-v2 test | ||
``` | ||
|
||
###### Gradle | ||
```shell | ||
> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:test | ||
``` | ||
|
||
### Acceptance Tests | ||
|
||
The [Airbyte project](https://github.com/airbytehq/airbyte) a standard test suite that all destination connectors must pass. The tests require specific implementations of a few components in order to connect the acceptance test suite with the connector's specific logic. The existing acceptance test scaffolding can be found in the conventional `src/test-integration/kotlin` source folder. | ||
|
||
The acceptance tests can be executed either via the [Airbyte CI CLI tool](https://github.com/airbytehq/airbyte/blob/master/airbyte-ci/connectors/pipelines/README.md) or [Gradle](https://gradle.org/): | ||
|
||
###### Airbyte CI CLI | ||
```shell | ||
> airbyte-ci connectors --name=destination-mssql-v2 test | ||
``` | ||
|
||
###### Gradle | ||
```shell | ||
> ./gradlew :airbyte-integrations:connectors:destination-mssql-v2:integrationTest | ||
``` | ||
|
||
## Release | ||
|
||
### Publishing a new version of the connector | ||
|
||
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what? | ||
|
||
1. Make sure your changes are passing our test suite: `airbyte-ci connectors --name=destination-mssql-v2 test` | ||
2. Bump the connector version in `metadata.yaml`: increment the `dockerImageTag` value. Please follow [semantic versioning for connectors](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#semantic-versioning-for-connectors). | ||
3. Make sure the `metadata.yaml` content is up to date. | ||
4. Make the connector documentation and its changelog is up to date (`docs/integrations/destinations/mssql-v2.md`). | ||
5. Create a Pull Request: use [our PR naming conventions](https://docs.airbyte.com/contributing-to-airbyte/resources/pull-requests-handbook/#pull-request-title-convention). | ||
6. Pat yourself on the back for being an awesome contributor. | ||
7. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master. | ||
|
||
|
52 changes: 52 additions & 0 deletions
52
airbyte-integrations/connectors/destination-mssql-v2/build.gradle.kts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
/* | ||
* Copyright (c) 2024 Airbyte, Inc., all rights reserved. | ||
*/ | ||
|
||
plugins { | ||
id("application") | ||
id("airbyte-bulk-connector") | ||
} | ||
|
||
airbyteBulkConnector { | ||
core = "load" | ||
toolkits = listOf() | ||
cdk = "local" | ||
} | ||
|
||
application { | ||
mainClass = "io.airbyte.integrations.destination.mssql.v2.MSSQLDestination" | ||
|
||
applicationDefaultJvmArgs = listOf("-XX:+ExitOnOutOfMemoryError", "-XX:MaxRAMPercentage=75.0") | ||
|
||
// Uncomment and replace to run locally | ||
//applicationDefaultJvmArgs = listOf("-XX:+ExitOnOutOfMemoryError", "-XX:MaxRAMPercentage=75.0", "--add-opens", "java.base/sun.nio.ch=ALL-UNNAMED", "--add-opens", "java.base/sun.security.action=ALL-UNNAMED", "--add-opens", "java.base/java.lang=ALL-UNNAMED") | ||
} | ||
|
||
val junitVersion = "5.11.3" | ||
|
||
configurations.configureEach { | ||
// Exclude additional SLF4J providers from all classpaths | ||
exclude(mapOf("group" to "org.slf4j", "module" to "slf4j-reload4j")) | ||
} | ||
|
||
// Uncomment to run locally | ||
//tasks.run.configure { | ||
// standardInput = System.`in` | ||
//} | ||
|
||
dependencies { | ||
implementation("com.microsoft.sqlserver:mssql-jdbc:12.8.1.jre11") | ||
implementation("io.github.oshai:kotlin-logging-jvm:7.0.0") | ||
implementation("jakarta.inject:jakarta.inject-api:2.0.1") | ||
implementation("com.github.spotbugs:spotbugs-annotations:4.8.6") | ||
implementation("io.micronaut:micronaut-inject:4.6.1") | ||
|
||
testImplementation("io.mockk:mockk:1.13.13") | ||
testImplementation("org.junit.jupiter:junit-jupiter-api:$junitVersion") | ||
testImplementation("org.junit.jupiter:junit-jupiter-params:$junitVersion") | ||
testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine:$junitVersion") | ||
} | ||
|
||
tasks.named<Test>("test") { | ||
systemProperties(mapOf("mockk.junit.extension.keepmocks" to "true", "mockk.junit.extension.requireParallelTesting" to "true")) | ||
} |
1 change: 1 addition & 0 deletions
1
airbyte-integrations/connectors/destination-mssql-v2/gradle.properties
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
testExecutionConcurrency=-1 |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
34 changes: 34 additions & 0 deletions
34
airbyte-integrations/connectors/destination-mssql-v2/metadata.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
data: | ||
connectorSubtype: database | ||
connectorType: destination | ||
definitionId: 37a928c1-2d5c-431a-a97d-ae236bd1ea0c | ||
dockerImageTag: 0.1.0 | ||
dockerRepository: airbyte/destination-mssql-v2 | ||
githubIssueLabel: destination-mssql-v2 | ||
icon: icon.svg | ||
license: ELv2 | ||
name: MSSQL V2 Destination | ||
registryOverrides: | ||
cloud: | ||
enabled: false | ||
oss: | ||
enabled: false | ||
releaseStage: alpha | ||
documentationUrl: https://docs.airbyte.com/integrations/destinations/mssql-v2 | ||
tags: | ||
- language:java | ||
ab_internal: | ||
sl: 100 | ||
ql: 100 | ||
supportLevel: community | ||
supportsRefreshes: true | ||
connectorTestSuitesOptions: | ||
- suite: unitTests | ||
- suite: integrationTests | ||
testSecrets: | ||
- name: SECRET_DESTINATION-S3-V2-MINIMAL-REQUIRED-CONFIG | ||
fileName: s3_dest_v2_minimal_required_config.json | ||
secretStore: | ||
type: GSM | ||
alias: airbyte-connector-testing-secret-store | ||
metadataSpecVersion: "1.0" |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nonblocking, did we ever find where this dep is coming from? and/or we could add this exclude to the plugin (
airbyte-bulk-connector.gradle
)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@edgao I would have to check. It may not be an issue in this destination (copy pasta here). If I recall correctly, it may have been one of the hadoop dependencies that we got via Iceberg and maybe even via the CDK because of parquet.