Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.32.0 release #733

Merged
merged 6 commits into from
Oct 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 18 additions & 10 deletions AWS_FLB_CHERRY_PICKS
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ https://github.com/PettitWesley/fluent-bit.git mk_event_add_corruption-one-commi
# Useful debug messages for input events
# input chunk append message with input name context and number of records
https://github.com/PettitWesley/fluent-bit.git chunk-append-context b671ed38e7ae87ab955083e2324ef1972298c5d7

# in_tail file name context for inotify events
https://github.com/PettitWesley/fluent-bit.git tail-modify-debug-context-immutable-cherry-pick ab11d1d7438f5254d04eba8dfc2f10b78cc2244d
https://github.com/PettitWesley/fluent-bit.git tail-modify-debug-context-immutable-cherry-pick 1f3bdeec49ac42b04b921303610ea0f53110ab2c
Expand Down Expand Up @@ -83,15 +84,6 @@ https://github.com/fluent/fluent-bit.git 1.9 81cdf7eced4e420043277237fba092157b1
# upstream engine retry clean up fix https://github.com/fluent/fluent-bit/pull/6862
https://github.com/fluent/fluent-bit.git 1.9 712e5fbe10bee44269d5dfed214c4e087ea1ec2a

# TCP input: user friendly message for skipping logs https://github.com/fluent/fluent-bit/pull/6061
https://github.com/fluent/fluent-bit.git 1.9 c5d57ccf4f85af77e2a2bbb1121df95887e2962f

# ElasticSearch: fix buffer bulk over run https://github.com/fluent/fluent-bit/pull/5770
https://github.com/fluent/fluent-bit.git 1.9 d3a0697c3c4a876250cc6c2c4ff44cd77908ea3c

# windows: fix time zone offset https://github.com/fluent/fluent-bit/pull/6368
https://github.com/fluent/fluent-bit.git 1.9 cae9a8f9cb6eab7449f14d914f9ba1d8d4e93cf0

Comment on lines -86 to -94
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matthewfala why are these deleted?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, thanks

# quick fix for S3 key $INDEX bug https://github.com/aws/aws-for-fluent-bit/issues/653
https://github.com/PettitWesley/fluent-bit.git index_s3_key_format_quick_fix 5e48218670681aef152aeedcf90a4593ac623470

Expand Down Expand Up @@ -127,11 +119,27 @@ https://github.com/PettitWesley/fluent-bit.git filter-modify-fix-aws-distro 1a72
# AWS core code self-review issue fixes
# upstream version of this: https://github.com/fluent/fluent-bit/pull/7512/files
# use calloc in all credential code to prevent freeing of garbage pointers
https://github.com/PettitWesley/fluent-bit.git aws-distro-crypto-self-review eb48b79b34f91d36d28434390b976e882d553681
https://github.com/PettitWesley/fluent-bit.git aws-distro-crypto-self-review eb48b79b34f91d36d28434390b976e882d553681
# fix brittle XML parsing
https://github.com/PettitWesley/fluent-bit.git aws-distro-crypto-self-review e6401ad3811b42dee0b7f92aba726cca4bee74ec
# add pthread_mutex + trylock to protect cred providers
https://github.com/PettitWesley/fluent-bit.git aws-distro-crypto-self-review a1d7469da62a4b3ca869b10732f41562d668cfce
# signv4: always use calloc
https://github.com/PettitWesley/fluent-bit.git aws-distro-crypto-self-review 3381c388ee956e7d0e7c0d5fc44683da75095a6a

# Cloudwatch_logs sequence token deprecation
https://github.com/matthewfala/fluent-bit.git immutable-sequence-token-deprecation 8ee560e388bbbf850069c81bbca06275f330baeb

# prometheus: sigv4 aws-for-fluent-bit
https://github.com/matthewfala/fluent-bit.git immutable-2.32.0-prometheus-sigv4 ca93bd1d43ebedeb8e81b46b800ea229fde66fa5

# multiline: remove incorrect flush
https://github.com/matthewfala/fluent-bit.git immutable-multiline-incorrect-flush 6431a4e584d52170dbe873d93ba532659921740a
https://github.com/matthewfala/fluent-bit.git immutable-multiline-incorrect-flush 35f23875ca356ea30e9aac19854b810cf8ecad8f

# core: network event drop shutdown fix
https://github.com/matthewfala/fluent-bit.git 2.32.0-premature-connection-destruction 9e2e5d1bffca92bbcc5001fcfc34c1d9ae2716db
https://github.com/matthewfala/fluent-bit.git 2.32.0-premature-connection-destruction b2e8ff1ae738c1db7bf50942ef619609436ffe02

# throttle: print_status configuration issue resolution
https://github.com/matthewfala/fluent-bit.git throttle-filter-print-status-fix 7b05b7ebfe55261ed12d5006c8b682572b6abf4c
2 changes: 1 addition & 1 deletion AWS_FOR_FLUENT_BIT_VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2.31.12.20231011
2.32.0
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,20 @@
# Changelog

### 2.32.0
This release includes:
* Fluent Bit [1.9.10](https://github.com/fluent/fluent-bit/tree/v1.9.10)
* Amazon CloudWatch Logs for Fluent Bit 1.9.4
* Amazon Kinesis Streams for Fluent Bit 1.10.2
* Amazon Kinesis Firehose for Fluent Bit 1.7.2

Compared to `2.31.12` this release adds:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated version comparison to 2.31.12

* Feature - Customers can send metrics to Amazon Managed Prometheus via added sigv4 authentication on `prometheus_remote_write`. Refer to [amazon-ecs-firelens-examples](https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/amazon-managed-service-for-prometheus/README.md) for information on how to export metrics to AMP on ECS [aws-for-fluent-bit:256](https://github.com/aws/aws-for-fluent-bit/issues/256)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added link to prometheus example

* Feature - Support multiline parsers with the init tag. Multiline parsers can be specified same as a standard parser file [aws-for-fluent-bit:537](https://github.com/aws/aws-for-fluent-bit/issues/537)
* Enhancement - Customers can route logs to CloudWatch Logs at higher throughputs by increasing number of output workers as `cloudwatch_logs` output plugin removed sequence tokens from API requests [aws-for-fluent-bit:526](https://github.com/aws/aws-for-fluent-bit/issues/526)
* Fix - Fix multiline input behavior when multiple streams are parsed (stderr, stdout) together. Multiline logs are no longer terminated when streams are switched between [fluent-bit:7469](https://github.com/fluent/fluent-bit/pull/7469)
* Fix - Fix networking edgecase causing data loss and OOM related issues via net timeout event injection resolution [fluent-bit:7728](https://github.com/fluent/fluent-bit/pull/7728/files)
* Fix - Allow reduced filter_throttle log output with `print_status` `false` configuration option fix [fluent-bit:7469](https://github.com/fluent/fluent-bit/pull/6500/files)

Copy link
Contributor

@PettitWesley PettitWesley Oct 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@matthewfala we should add on3e more things from here, when I tried to stage 2.32: https://github.com/aws/aws-for-fluent-bit/pull/658/files

  1. AWS Profile support

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documenting our decision:

  1. Will further review this feature item in the future in the 2.33.0 release. Requires further review and testing. link
  2. Proceed with release

### 2.31.12.20231011 Linux re-build

*This release has the same Fluent Bit contents as 2.31.12, and is simply a linux-only re-build for recent patches in dependencies installed in the image. There are no windows images for this release.*
Expand Down
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ all: release
export DOCKER_BUILD_FLAGS=--no-cache

.PHONY: release
release: build build-init
release: build build-init linux-plugins
docker system prune -f
docker build $(DOCKER_BUILD_FLAGS) -t amazon/aws-for-fluent-bit:main-release -f ./scripts/dockerfiles/Dockerfile.main-release .
docker tag amazon/aws-for-fluent-bit:main-release amazon/aws-for-fluent-bit:latest
Expand All @@ -28,7 +28,7 @@ release: build build-init
debug: main-debug init-debug

.PHONY: build
build: linux-plugins
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build should still build linux-plugins?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build dockerfile/image does not depend on the linux-plugins image. Actually only the release and main-release depends on the "linux plugins"

It was slowing down my fluent-bit build tests.

build:
docker system prune -f
docker build $(DOCKER_BUILD_FLAGS) -t amazon/aws-for-fluent-bit:build -f ./scripts/dockerfiles/Dockerfile.build .

Expand Down Expand Up @@ -118,7 +118,7 @@ init-debug-s3: main-debug-base build-init
docker build $(DOCKER_BUILD_FLAGS) -t amazon/aws-for-fluent-bit:init-debug-s3 -f ./scripts/dockerfiles/Dockerfile.init-debug-s3 .

.PHONY: main-debug-base
main-debug-base: build
main-debug-base: build linux-plugins
docker build $(DOCKER_BUILD_FLAGS) -t amazon/aws-for-fluent-bit:main-debug-base -f ./scripts/dockerfiles/Dockerfile.main-debug-base .

.PHONY: validate-version-file-format
Expand Down
2 changes: 1 addition & 1 deletion linux.version
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"linux": {
"version": "2.31.12.20231011",
"version": "2.32.0",
"latest": "true",
"build": "1",
"fluent-bit": "1.9.10",
Expand Down
9 changes: 5 additions & 4 deletions scripts/dockerfiles/Dockerfile.build
Original file line number Diff line number Diff line change
Expand Up @@ -88,14 +88,15 @@ COPY AWS_FLB_CHERRY_PICKS \
RUN git config --global user.email "[email protected]" \
&& git config --global user.name "FireLens Team"

RUN AWS_FLB_CHERRY_PICKS_COUNT=`cat /AWS_FLB_CHERRY_PICKS | sed '/^#/d' | sed '/^\s*$/d' | wc -l | awk '{ print $1 }'`; \
RUN AWS_FLB_CHERRY_PICKS_COUNT=`awk '{print $0 }' /AWS_FLB_CHERRY_PICKS | sed '/^#/d' | sed '/^\s*$/d' | wc -l | awk '{ print $1 }'`; echo $AWS_FLB_CHERRY_PICKS_COUNT; \
if [ $AWS_FLB_CHERRY_PICKS_COUNT -gt 0 ]; \
then \
cat /AWS_FLB_CHERRY_PICKS | sed '/^#/d' \
| xargs -l bash -c 'git fetch $0 $1 && git cherry-pick $2'; \
| xargs -l bash -c 'git fetch $0 $1 && git cherry-pick $2 || exit 255' && \
\
echo "Cherry Pick Patch Summary:"; \
(echo "Cherry Pick Patch Summary:"; \
echo -n "Base "; \
git log --oneline \
-$((AWS_FLB_CHERRY_PICKS_COUNT+1)) \
| tac | awk '{ print "Commit",NR-1,"--",$0 }'; sleep 2; \
| tac | awk '{ print "Commit",NR-1,"--",$0 }'; sleep 2;)\
fi
14 changes: 13 additions & 1 deletion windows.versions
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"windows": [
{
"version": "2.31.12",
"version": "2.32.0",
"build": "1",
"fluent-bit": "1.9.10",
"kinesis-plugin": "v1.10.2",
Expand All @@ -10,6 +10,18 @@
"openssl": "3.0.7",
"flexBison": "2.5.22",
"latest": true,
"stable": false
},
{
"version": "2.31.12",
"build": "1",
"fluent-bit": "1.9.10",
"kinesis-plugin": "v1.10.2",
"firehose-plugin": "v1.7.2",
"cloudwatch-plugin": "v1.9.4",
"openssl": "3.0.7",
"flexBison": "2.5.22",
"latest": false,
"stable": true
},
{
Expand Down