Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Empty file name when append_date_to_filename=False #19

Open
srpwnd opened this issue Jun 22, 2023 · 1 comment · May be fixed by #43
Open

Empty file name when append_date_to_filename=False #19

srpwnd opened this issue Jun 22, 2023 · 1 comment · May be fixed by #43
Labels
bug Something isn't working

Comments

@srpwnd
Copy link

srpwnd commented Jun 22, 2023

Hello,

I'm using target-s3 within Meltano and want to overwrite the Parquet files I'm creating with every run.

I set the TARGET_S3_APPEND_DATE_TO_FILENAME and TARGET_S3_APPEND_DATE_TO_PREFIX to False but now I end up with files in S3 like so: bucket_name/prefix/stream_name/.parquet

It is still possible to query this file but feels a bit odd to me.

As you can see here the filename is only set when append_date_to_filename is True.

I would expect that when both append_date_to_filename and append_date_to_prefix is set to false the filename would be equal to the stream_name resulting in this: bucket_name/prefix/stream_name.parquet

Is there any reason behind it?

I could make a PR adjusting this if it is welcome.

@crowemi
Copy link
Owner

crowemi commented Jun 22, 2023

There is no reason behind this, similar to #18 -- go ahead an make the PR and we can get this corrected, thank you 😄

@crowemi crowemi added the bug Something isn't working label Jun 29, 2023
dror-g added a commit to dror-g/target-s3 that referenced this issue May 1, 2024
Fixed using: file_name = stream_name

(:man_shrugging: )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants