Releases: snowplow-incubator/snowplow-bigquery-loader
Version 1.5.1
- Bump scala version to 2.13.9 (#311)
Version 1.5.0
Adding flow control to streaming loader for greater control over memory usage
Version 1.4.3
Version 1.4.2
This release fixes two bugs that had crept in since the 1.4.0 release:
Fix for critical exceptions like UnsupportedOperationException empty.min (see #302). In previous versions this exception would cause the loader application to crash, and it happened whenever a batch insert into BigQuery resulted in zero successful inserts.
Fix for loader shutdown so it does not hang on exit (See #303). Previous versions got stuck when shutting down, with a final message in the logs like Source of events was cancelled. It prevented the loader from exiting upon failures, or during any other controlled shutdown.
Version 1.4.1
The main goal of this patch release is to handle security vulnerabilities for some packages in the base docker image (see openssl and libfreetype6).
It also brings small improvements to metrics reporting.
Changelog
Version 1.4.0
The biggest new feature allowing files to be named on the command line. You no longer need to provide base-64 encoded config on the command line, but instead you can pass a file path for each of the --config and --resolver arguments. Here is an example of a docker command, in which the config files are mounted into the container:
docker run \
-v /path/to/configs:/snowplow \
snowplow/snowplow-bigquery-streamloader:1.4.0 \
--config=/snowplow/config.hocon \
--resolver=/snowplow/resolver.json
We also bumped the cats effect library from version 2 to version 3. This is a real under-the-hood type change, but it's a big step up onto a newer major version of our main runtime library.
Changelog
Version 1.3.2
A small patch release to fix possible race conditions in the Mutator. The bug was not critical, but it could be a nuisance.
Changelog
Mutator: remove concurrency on table updates (#287)
Version 1.3.1
Release that aims at mitigating security vulnerabilities and addresses a bug that effects cross-project usage of BigQuery tables.
Changelog
Version 1.3.0
Starting from this release, we are publishing alternative lightweight docker images based on Google's "distroless" base image. The distroless images provide some security advantages, because they carry only the minimal files and executables needed for the loader to run.
You can pull the 1.3.0-distroless tag to get started with the new images:
docker pull snowplow/snowplow-bigquery-streamloader:1.3.0-distroless
docker pull snowplow/snowplow-bigquery-mutator:1.3.0-distroless
docker pull snowplow/snowplow-bigquery-repeater:1.3.0-distroless
This release also bumps many underlying libraries to newer versions, and fixes the graceful shutdown feature so it shuts down more quickly, still without dropping any events.
Changelog
- Bump google-oauth-client to 1.33.3 (#281)
- Fix graceful shutdown (#260)
- Publish distroless docker images (#279)
- Bump sbt-native-packager to 1.9.9 (#278)
- Bump jackson-databind to 2.13.2.2 (#275)
- Remove explicit version overrides of gson and fast-json (#280)
- Bump netty codec to 4.1.76.Final (#271)
- Bump scio to 0.11.7 (#270)
- Bump google-cloud-bigquery for beam to 2.9.4 (#269)
- Bump beam to 2.38.0 (#268)
- Bump dropwizard metrics to 4.1.31 (#267)
- Bump http4s-async-http-client to 0.22.12 (#266)
- Bump google-cloud-storage to 2.6.1 (#265)
- Bump google-cloud-pubsub to 1.116.4 (#264)
- Bump google-cloud-bigquery to 2.10.10 (#263)
Version 1.2.0
This release adds graceful shutdown and retry setting for BigQueryException errors in Streamloader.
New functionality
- Streamloader: add retrySettings to BigQuery client (#248)
- Streamloader: add graceful shutdown (#249)
Dependency management
- Bump scio-core to 0.11.4 (#252)
- Bump specs2-core to 4.13.2 (#254)
- Bump fs2-google-pubsub-grpc to 0.18.1 (#255)
- Bump log4cats-slf4j to 1.5.1 (#256)
- Bump schema-ddl to 0.14.4 (#253)
Under the hood
- Change docker base image to eclipse/temurin:11-jre-focal (#251)