Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Destination S3-V2: Bug: Honor path variables in bucket prefix #51039

Merged
merged 1 commit into from
Jan 10, 2025
Merged

Destination S3-V2: Bug: Honor path variables in bucket prefix #51039

merged 1 commit into from
Jan 10, 2025

Conversation

johnny-schmidt
Copy link
Contributor

What

The old CDK honored variables in both the bucket prefix and the path. The documentation is a little ambiguous about this, but it's a reasonable expectation that they should work in both places. This gets parity with the old code.

I'll test against this connection before releasing

I also modified the ambiguous filepath config secret to use this pattern, so there's an IT test case as well as a UTs:

{
...
  "s3_bucket_path": "s3-v2-test/${NAMESPACE}",
...
  "s3_path_format": "${STREAM_NAME}/",
  "file_name_pattern": "very_bad_practice"
}

@johnny-schmidt johnny-schmidt requested a review from a team as a code owner January 10, 2025 18:52
@vercel Vercel
Copy link

vercel bot commented Jan 10, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jan 10, 2025 6:52pm

@octavia-squidington-iii octavia-squidington-iii added the CDK Connector Development Kit label Jan 10, 2025
tryangul
tryangul reviewed Jan 10, 2025
View reviewed changes
...-storage/src/main/kotlin/io/airbyte/cdk/load/file/object_storage/ObjectStoragePathFactory.kt
): String {
val pattern = pathPatternResolved
val selectedPrefix = if (isStaging) stagingPrefix else finalPrefix
val pattern = resolveRetainingTerminalSlash(selectedPrefix, pathPatternResolved)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be simpler to pass this pattern in instead of passing down a boolean to replicate the branching?

@johnny-schmidt johnny-schmidt merged commit 05b6d0b into master Jan 10, 2025
30 checks passed
@johnny-schmidt johnny-schmidt deleted the jschmidt/s3v2/bug-regex-maybe2 branch January 10, 2025 19:44
@johnny-schmidt johnny-schmidt mentioned this pull request Jan 10, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tryangul tryangul tryangul approved these changes

Assignees
No one assigned
Labels
CDK Connector Development Kit
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@johnny-schmidt @tryangul @octavia-squidington-iii