windows: build docker image with out_kafka and gzip #6214

alt-dima · 2022-10-16T15:25:49Z

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change
Debug log output from testing the change

Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Attached local packaging test output showing all targets (including any new ones) build.

Documentation

Documentation required for this feature

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

alt-dima · 2022-10-16T15:31:53Z

Probably this PR will be rejected, no problem.
just trying to help another devops to build fluent-bit with out_kafka plugin for windows

My changes

I was not luck to enable cmake -DFLB_OUT_KAFKA=On (cmake just ignored it), changed option in cmake/windows-setup.cmake and it worked
ENTRYPOINT and CMD was fixed in dockerfiles/Dockerfile.windows, because \ must be escaped. Without this fix container was not able to start with error "can not find the file specified" (in Kubernetes)
lib/librdkafka-1.9.2/src/rdkafka_ssl.c was fixed for proper ssl libraries from version 1.1.1
plugins/out_kafka/CMakeLists.txt fixed to remove lpthread if building with MSVC

alt-dima · 2022-10-16T15:38:12Z

@patrick-stephens can you, please, take a look?
Is windows docker image works completely as expected without all those changes for you?

alt-dima · 2022-10-16T15:53:31Z

Just checked my image in Kubernetes (with fluentbit helm chart)

Fluent Bit v2.0.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/10/16 15:51:39] [ warn] [config] I cannot open C:\fluent-bit\etc\/C:\fluent-bit\etc\parsers.conf file
[2022/10/16 15:51:39] [ info] [fluent bit] version=2.0.0, commit=, pid=252
[2022/10/16 15:51:39] [ info] [storage] ver=1.3.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2022/10/16 15:51:39] [ info] [cmetrics] version=0.5.2
[2022/10/16 15:51:39] [ info] [input:tail:tail.0] initializing
[2022/10/16 15:51:39] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2022/10/16 15:51:39] [ info] [input:tail:tail.0] multiline core started
[2022/10/16 15:51:39] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc.cluster.local port=443
[2022/10/16 15:51:39] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/16 15:51:39] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2022/10/16 15:51:40] [ info] [filter:kubernetes:kubernetes.0] host: kubernetes.default.svc.cluster.local Wait 30 secs until DNS starts up (1/6)
[2022/10/16 15:52:10] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2022/10/16 15:52:10] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2022/10/16 15:52:10] [ info] [input:emitter:emitter_for_rewrite_tag.4] initializing
[2022/10/16 15:52:10] [ info] [input:emitter:emitter_for_rewrite_tag.4] storage_strategy='memory' (memory only)
[2022/10/16 15:52:10] [error] [flb_kafka] cannot configure 'compression.type' property
[2022/10/16 15:52:10] [ info] [output:kafka:kafka.0] brokers='b-1.stagis.com:9092' topics='app-logs'
[2022/10/16 15:52:10] [error] [flb_kafka] cannot configure 'compression.type' property
[2022/10/16 15:52:10] [ info] [output:kafka:kafka.1] brokers='b-1.staging1kafkaops.com:9092' topics='app-metrics'
[2022/10/16 15:52:10] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2022/10/16 15:52:10] [ info] [sp] stream processor started

Started successfully with kafka_out plugin!

Current problems

[warn] [config] I cannot open C:\fluent-bit\etc\/C:\fluent-bit\etc\parsers.conf file = because of volumeMounts specific implementation in Windows containers, I mount completely etc folder and loose all the files.
Workaround = copy content of parsers.conf to customParsers in helm config.yaml

Error: Error response from daemon: invalid volume specification: 'c:\var\lib\kubelet\pods\88ae9a4a-df0d-41c6-af72-fd44047753db\volumes\kubernetes.io~configmap\config\fluent-b │
│ it.conf:c:/fluent-bit/etc/fluent-bit.conf:ro': invalid mount config for type "bind": source path must be a directory

Liveness probe shoul be with delay 2-5 min because of network delay in windows container = fixed with initialDelaySeconds: 180 in values.yaml in helm chart
[error] [flb_kafka] cannot configure 'compression.type' property = added Zlib installation and compilation to the dockerfile
actually logs are not being ready by fluent-bit because of Windows file permissions/location. No access to the docker log files

PS C:\var\log\containers> cat win-fluent-bit-9h8dw_logging_fluent-bit-e17294a03115c2b2bebc1624c7e850d80fe643f8b52d22d432720d7acbbd3270.log                                                                                                                  
cat : Could not find a part of the path 'C:\var\log\containers\win-fluent-bit-9h8dw_logging_fluent-bit-e17294a03115c2b2bebc1624c7e850d80fe643f8b52d22d432720d7acbbd3270.log'.

Fixed with proper/additional volume mounts

daemonSetVolumeMounts:
  - name: var
    mountPath: /var
    readOnly: true
  - name: docker
    mountPath: /ProgramData/docker/containers
    readOnly: true

Logs are coming without any kubernetes fields because of wrong pod name search in API (proper pod name is rms-adhd-web-7f797b678f-mdsjc)

 [2022/10/16 18:22:41] [debug] [filter:kubernetes:kubernetes.0] Request (ns=default, pod=s.rms-adhd-web-7f797b678f-mdsjc) http_do=0, HTTP Status: 404                                                                                                     │
│ [2022/10/16 18:22:41] [debug] [filter:kubernetes:kubernetes.0] HTTP response                                                                                                                                                                             │
│ {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"s.rms-adhd-web-7f797b678f-mdsjc\" not found","reason":"NotFound","details":{"name":"s.rms-adhd-web-7f797b678f-mdsjc","kind":"pods"},"code":404}

Fixed with correct Kube_Tag_Prefix application.c.var.log.containers. (before it was like for Linux Kube_Tag_Prefix application.var.log.containers.)

patrick-stephens · 2022-10-17T09:37:56Z

The Windows containers are not officially released or supported by the OSS project, they are provided primarily for reference but good to ensure they work. I do have a separate PR to provide OpenSSL as well but this was only around getting it to build so happy to have a better fix that verifies the container runs.

Note your commits do not follow the contribution guidance and are not signed off for DCO so those will need fixing: https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md#commit-changes

patrick-stephens · 2022-10-17T14:52:55Z

@cosmo0920 may have some input here.

@alt-dima could you update your commit messages and sign them off? I'm reluctant to kick off the CI without it as we'll just have to do it again when the commits are updated.

alt-dima · 2022-10-17T15:33:33Z

To fix an issue
[error] [flb_kafka] cannot configure 'compression.type' property = removed from configuration, probably gzip support was missing when build

I added to the dockerfile.windows:

# Install ZLIB: https://github.com/horta/zlib.install
ADD https://github.com/alt-dima/zlib.install/raw/master/install.bat /local/install_zlib.bat
RUN /local/install_zlib.bat
RUN Copy-Item -Path '/Program Files/zlib/bin/zlib.dll' -Destination /fluent-bit/bin/;

But I'm not sure it is a good choice.

alt-dima · 2022-10-17T15:44:29Z

Nice! Finally, all the problems were solved, will test some days.

Fluent Bit v2.0.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/10/17 15:40:45] [ info] [fluent bit] version=2.0.0, commit=, pid=3836
[2022/10/17 15:40:45] [ info] [storage] ver=1.3.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2022/10/17 15:40:45] [ info] [cmetrics] version=0.5.2
[2022/10/17 15:40:45] [ info] [input:tail:tail.0] initializing
[2022/10/17 15:40:45] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2022/10/17 15:40:45] [ info] [input:tail:tail.0] multiline core started
[2022/10/17 15:40:45] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc.cluster.local port=443
[2022/10/17 15:40:45] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2022/10/17 15:40:45] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2022/10/17 15:40:45] [ info] [filter:kubernetes:kubernetes.0] host: kubernetes.default.svc.cluster.local Wait 30 secs until DNS starts up (1/6)
[2022/10/17 15:41:15] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2022/10/17 15:41:15] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2022/10/17 15:41:15] [ info] [input:emitter:emitter_for_rewrite_tag.4] initializing
[2022/10/17 15:41:15] [ info] [input:emitter:emitter_for_rewrite_tag.4] storage_strategy='memory' (memory only)
[2022/10/17 15:41:15] [ info] [output:kafka:kafka.0] brokers='b-1.staging1konaws.com:9092' topics='app-logs'
[2022/10/17 15:41:15] [ info] [output:kafka:kafka.1] brokers='b-1.stagi.amazonaws.com:9092' topics='app-metrics'
[2022/10/17 15:41:15] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2022/10/17 15:41:15] [ info] [sp] stream processor started

alt-dima · 2022-10-17T16:02:30Z

Build for 64bit was fine, but for 32bit failed, probably because of

#pragma comment(lib, "libcrypto64MT.lib")
#pragma comment(lib, "libssl64MT.lib")

alt-dima · 2022-10-18T07:21:31Z

I have a problem with in_tail. Seems like rotation handling working correctly, but error appears.
Moved to PR #6230

Signed-off-by: alt-dima <[email protected]>

cosmo0920 · 2022-10-24T07:57:11Z

cmake/windows-setup.cmake

@@ -75,7 +75,7 @@ if(FLB_WINDOWS_DEFAULTS)
  set(FLB_OUT_LIB               Yes)
  set(FLB_OUT_NULL              Yes)
  set(FLB_OUT_FLOWCOUNTER       Yes)
-  set(FLB_OUT_KAFKA              No)


This is just for my curiosity.
Should we disable for out_kafka plugin building on Windows and specify enabling parameter on configure at Windows container building?

This is because to build out_kafka plugin on Windows, we have to introduce dependent zlib library for our CI tasks.

Did we do this anyway now @cosmo0920? Ah no, I think that was just for our commercial offering...

cosmo0920 · 2022-10-24T07:59:34Z

lib/librdkafka-1.9.2/src/rdkafka_ssl.c

+#pragma comment(lib, "libcrypto64MT.lib")
+#pragma comment(lib, "libssl64MT.lib")


Could you determine whether the using 64MT or 32MT suffixed libraries with CMakeLists.txt?
This change should be working only for 64bit of Windows.

patrick-stephens · 2022-10-31T12:09:56Z

dockerfiles/Dockerfile.windows

@@ -38,6 +38,11 @@ RUN Start-Process /local/vc_redist.x64.exe -ArgumentList '/install', '/quiet', '
    Copy-Item -Path /Windows/System32/vccorlib140.dll -Destination /fluent-bit/bin/; `
    Copy-Item -Path /Windows/System32/vcruntime140.dll -Destination /fluent-bit/bin/;

+# Install ZLIB: https://github.com/horta/zlib.install
+ADD https://github.com/alt-dima/zlib.install/raw/master/install.bat /local/install_zlib.bat


We probably should not depend on master of another repo we have no control over - potentially this is updated and breaks the build best case (worst case it's a malware attack vector). At the very least this makes a build non-reproducible.

We should either put the commands inline in the Dockerfile or move the script into this repo as part of the PR - then it can be versioned alongside.

github-actions · 2023-01-30T02:00:30Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

github-actions · 2023-05-31T02:07:04Z

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

alt-dima requested review from niedbalski, patrick-stephens, fujimotos, edsiper, leonardo-albertovich and koleini as code owners October 16, 2022 15:25

github-actions bot added the docs-required label Oct 16, 2022

alt-dima force-pushed the fix-win-build branch from 23f2478 to 530b8b0 Compare October 17, 2022 15:28

alt-dima changed the title ~~Fix windows docker build and enable out_kafka~~ windows: build docker image with out_kafka and gzip Oct 17, 2022

alt-dima had a problem deploying to pr October 17, 2022 15:40 Failure

alt-dima temporarily deployed to pr October 17, 2022 15:40 Inactive

Dmitrii Altukhov added 10 commits October 18, 2022 11:27

enable kafka fix docker

189171f

Signed-off-by: alt-dima <[email protected]>

fix cmake kafka

dd9e70b

Signed-off-by: alt-dima <[email protected]>

kafka ssl

1f50853

Signed-off-by: alt-dima <[email protected]>

added zlib

20d8df0

Signed-off-by: alt-dima <[email protected]>

fix

6e6cea0

Signed-off-by: alt-dima <[email protected]>

enable CMP0074

6aced7c

Signed-off-by: alt-dima <[email protected]>

fix zlibsrc path

dfbb1c9

Signed-off-by: alt-dima <[email protected]>

fiz zlib install

d18b557

Signed-off-by: alt-dima <[email protected]>

revert cmake 0074

28843a5

Signed-off-by: alt-dima <[email protected]>

add zlib dll copy

4292bb4

Signed-off-by: alt-dima <[email protected]>

fix in_tail

b52ba2e

Signed-off-by: alt-dima <[email protected]>

alt-dima force-pushed the fix-win-build branch from e6c53e3 to b52ba2e Compare October 18, 2022 08:27

alt-dima added 2 commits October 18, 2022 11:57

Merge branch 'master' into fix-win-build

1ba1695

revert changes in in_tail

d3a3085

Signed-off-by: alt-dima <[email protected]>

cosmo0920 reviewed Oct 24, 2022

View reviewed changes

alt-dima had a problem deploying to pr October 31, 2022 12:07 Failure

alt-dima temporarily deployed to pr October 31, 2022 12:07 Inactive

patrick-stephens reviewed Oct 31, 2022

View reviewed changes

github-actions bot added the Stale label Jan 30, 2023

Merge branch 'fluent:master' into fix-win-build

d86f39f

github-actions bot removed the Stale label Mar 1, 2023

github-actions bot added the Stale label May 31, 2023

Merge branch 'master' into fix-win-build

299ed6c

alt-dima requested a review from celalettin1286 as a code owner August 5, 2023 09:51

github-actions bot removed the Stale label Aug 6, 2023

Merge branch 'fluent:master' into fix-win-build

6bfa55b

alt-dima had a problem deploying to pr October 4, 2023 10:25 — with GitHub Actions Failure

alt-dima closed this Oct 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

windows: build docker image with out_kafka and gzip #6214

windows: build docker image with out_kafka and gzip #6214

alt-dima commented Oct 16, 2022

alt-dima commented Oct 16, 2022 •

edited

Loading

alt-dima commented Oct 16, 2022

alt-dima commented Oct 16, 2022 •

edited

Loading

patrick-stephens commented Oct 17, 2022

patrick-stephens commented Oct 17, 2022

alt-dima commented Oct 17, 2022

alt-dima commented Oct 17, 2022

alt-dima commented Oct 17, 2022

alt-dima commented Oct 18, 2022 •

edited

Loading

cosmo0920 Oct 24, 2022

patrick-stephens Oct 4, 2023

cosmo0920 Oct 24, 2022

patrick-stephens Oct 31, 2022 •

edited

Loading

github-actions bot commented Jan 30, 2023

github-actions bot commented May 31, 2023

		#pragma comment(lib, "libcrypto64MT.lib")
		#pragma comment(lib, "libssl64MT.lib")

windows: build docker image with out_kafka and gzip #6214

windows: build docker image with out_kafka and gzip #6214

Conversation

alt-dima commented Oct 16, 2022

alt-dima commented Oct 16, 2022 • edited Loading

alt-dima commented Oct 16, 2022

alt-dima commented Oct 16, 2022 • edited Loading

patrick-stephens commented Oct 17, 2022

patrick-stephens commented Oct 17, 2022

alt-dima commented Oct 17, 2022

alt-dima commented Oct 17, 2022

alt-dima commented Oct 17, 2022

alt-dima commented Oct 18, 2022 • edited Loading

cosmo0920 Oct 24, 2022

Choose a reason for hiding this comment

patrick-stephens Oct 4, 2023

Choose a reason for hiding this comment

cosmo0920 Oct 24, 2022

Choose a reason for hiding this comment

patrick-stephens Oct 31, 2022 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Jan 30, 2023

github-actions bot commented May 31, 2023

alt-dima commented Oct 16, 2022 •

edited

Loading

alt-dima commented Oct 16, 2022 •

edited

Loading

alt-dima commented Oct 18, 2022 •

edited

Loading

patrick-stephens Oct 31, 2022 •

edited

Loading