You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm using target-s3 within Meltano and want to overwrite the Parquet files I'm creating with every run.
I set the TARGET_S3_APPEND_DATE_TO_FILENAME and TARGET_S3_APPEND_DATE_TO_PREFIX to False but now I end up with files in S3 like so: bucket_name/prefix/stream_name/.parquet
It is still possible to query this file but feels a bit odd to me.
As you can see here the filename is only set when append_date_to_filename is True.
I would expect that when both append_date_to_filename and append_date_to_prefix is set to false the filename would be equal to the stream_name resulting in this: bucket_name/prefix/stream_name.parquet
Is there any reason behind it?
I could make a PR adjusting this if it is welcome.
The text was updated successfully, but these errors were encountered:
Hello,
I'm using
target-s3
within Meltano and want to overwrite the Parquet files I'm creating with every run.I set the
TARGET_S3_APPEND_DATE_TO_FILENAME
andTARGET_S3_APPEND_DATE_TO_PREFIX
toFalse
but now I end up with files in S3 like so:bucket_name/prefix/stream_name/.parquet
It is still possible to query this file but feels a bit odd to me.
As you can see here the filename is only set when
append_date_to_filename
isTrue
.I would expect that when both
append_date_to_filename
andappend_date_to_prefix
is set to false the filename would be equal to the stream_name resulting in this:bucket_name/prefix/stream_name.parquet
Is there any reason behind it?
I could make a PR adjusting this if it is welcome.
The text was updated successfully, but these errors were encountered: