Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What should be the config for output to be in parquet format? #258

Open
Rstar1998 opened this issue Nov 1, 2022 · 0 comments
Open

What should be the config for output to be in parquet format? #258

Rstar1998 opened this issue Nov 1, 2022 · 0 comments
Labels

Comments

@Rstar1998
Copy link

Rstar1998 commented Nov 1, 2022

I have the following file for gcs sink connector for events to be stored in parquet format?
But this thing is not working. Is there anything more needed for parquet conversion ?

{
  "name": "GCS_CONN_REG",
  "config": {
    "connector.class": "io.aiven.kafka.connect.gcs.GcsSinkConnector",
    "gcs.bucket.name": "name",
    "file.name.prefix": "test/",
    "format.output.type": "parquet",
    "name": "GCS_CONN_REG",
    "value.converter.schemas.enable": "false",
    "format.output.fields": "key,value,offset,timestamp,headers",
    "gcs.credentials.json": "",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "topics.regex": "abc.*",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "file.compression.type": "gzip",
    "file.name.template": "{{topic}}/{{timestamp:unit=yyyy}}/{{timestamp:unit=MM}}/{{timestamp:unit=dd}}/{{partition}}-{{start_offset}}.parquet",
    "errors.tolerance": "all",
    "consumer.override.auto.offset.reset": "latest",
    "errors.log.enable": "true",
    "errors.deadletterqueue.topic.name": "gcs_external_connector",
    "errors.log.include.messages": "true"
  }
}
@Rstar1998 Rstar1998 changed the title What should be in config What should be the config for output to be in parquet format? Nov 1, 2022
@ahmedsobeh ahmedsobeh transferred this issue from Aiven-Open/gcs-connector-for-apache-kafka Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants