Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP output to Betterstack crashes with a BusError on FreeBSD 14.2 amd64 #9722

Open
arsatiki opened this issue Dec 13, 2024 · 10 comments
Open

Comments

@arsatiki
Copy link

Bug Report

Describe the bug

My goal is to send logs from my FreeBSD host and any jails running on it to Betterstack. When I enable the HTTP output plugin, fluent-bit crashes with a Bus Error when it tries to send the message forward.

To Reproduce

  • Steps to reproduce the problem:

The configuration file looks like this:

[SERVICE]
    flush        1
    log_level    info
    parsers_file parsers.conf
    plugins_file plugins.conf
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
    storage.metrics on

[INPUT]
    tag  syslog
    name tail
    path /var/log/messages

[INPUT]
    tag siansaksa
    name random

[OUTPUT]
    match *
    name stdout
    format json_lines
        
[OUTPUT]
    name    http
    match   *
    tls     On
    host    in.logs.betterstack.com
    port    443
    uri     /fluentbit
    header  Authorization Bearer XXXXXX # Token omitted for privacy
    header  Content-Type application/msgpack
    format  msgpack
    retry_limit 5

I execute Fluent Bit with doas -u nobody /usr/local/bin/fluent-bit -c /usr/local/etc/fluent-bit/fluent-bit.conf.

The execution crashes with a Bus Error after the first random entry is generated:

[2024/12/13 12:18:45] [ info] [config] changing coro_stack_size from 3072 to 4096 bytes
Fluent Bit v3.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io/

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2024/12/13 12:18:45] [ info] [fluent bit] version=3.2.2, commit=, pid=23342
[2024/12/13 12:18:45] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/12/13 12:18:45] [ info] [simd    ] disabled
[2024/12/13 12:18:45] [ info] [cmetrics] version=0.9.9
[2024/12/13 12:18:45] [ info] [ctraces ] version=0.5.7
[2024/12/13 12:18:45] [ info] [input:tail:tail.0] initializing
[2024/12/13 12:18:45] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/12/13 12:18:45] [ info] [input:random:random.1] initializing
[2024/12/13 12:18:45] [ info] [input:random:random.1] storage_strategy='memory' (memory only)
[2024/12/13 12:18:45] [ info] [output:stdout:stdout.0] worker #0 started
[2024/12/13 12:18:45] [ info] [sp] stream processor started
[2024/12/13 12:18:45] [ info] [output:http:http.1] worker #0 started
[2024/12/13 12:18:45] [ info] [output:http:http.1] worker #1 started
{"date":1734085126.234562,"rand_value":6488732564125523264}
Bus error

Expected behavior

  • Fluent will print out random log entries on console
  • Same entries are visible in Betterstack

Your Environment

  • Version used: 3.1.9 and 3.2.2
  • Configuration: See above
  • Operating System and version: FreeBSD 14.2 on amd64

Additional context

Since the shipper doesn't work for me, I've been forced to install Fluentd and it makes me unhappy.

@arsatiki
Copy link
Author

Some wrangling with the core dump says that the problem is in the ares__slist_node_first function. The address for head seems to be rax = 0xd234bc1b34275b44, which should be aligned with 8 on amd64.

@arsatiki
Copy link
Author

We managed to get it working if there's only one active source and only one active output, ranom values in this case. Once I write something to the syslog, the next random entry brings down the process again.

@arsatiki
Copy link
Author

Mitigated the problem by making the coro stack 80k

@patrick-stephens
Copy link
Contributor

Was going to say, we don't technically support it directly as a platform: https://docs.fluentbit.io/manual/installation/supported-platforms

I was assuming you were compiling it directly so we would need a lot more information about how/what you configured to do that but sounds like you sorted.

@arsatiki
Copy link
Author

Initially I got it from the package system, i.e. as a prebuilt binary. See https://www.freshports.org/sysutils/fluent-bit/ for example. I did my own build for debugging purposes to get debugging symbols and address sanitizer. That build also used the setup from the ports system, so both cases built it the same way.

@patrick-stephens
Copy link
Contributor

That build is unrelated to this project so we cannot support it.

Did you get it going then with the coro stack size change?

@arsatiki
Copy link
Author

It's been running 15 hours now without crashing, keeping my fingers crossed :D

@patrick-stephens
Copy link
Contributor

It might be worth adding to the general Raspbian builds then here:

RUN cmake -DCMAKE_INSTALL_PREFIX="$CMAKE_INSTALL_PREFIX" \
-DCMAKE_INSTALL_SYSCONFDIR="$CMAKE_INSTALL_SYSCONFDIR" \
-DFLB_RELEASE="$FLB_RELEASE" \
-DFLB_TRACE="$FLB_TRACE" \
-DFLB_SQLDB="$FLB_SQLDB" \
-DFLB_HTTP_SERVER="$FLB_HTTP_SERVER" \
-DFLB_OUT_KAFKA="$FLB_OUT_KAFKA" \
-DFLB_OUT_PGSQL="$FLB_OUT_PGSQL" \
-DFLB_NIGHTLY_BUILD="$FLB_NIGHTLY_BUILD" \
-DFLB_JEMALLOC="${FLB_JEMALLOC}" \
-DFLB_CHUNK_TRACE="${FLB_CHUNK_TRACE}" \
../

@girgen
Copy link
Contributor

girgen commented Dec 22, 2024

Hi!

I'm the "porter" for fluent-bit to FreeBSD. I guess, since you don't support the platform, I'm trying to do that for you. FreeBSD users are kind of used to this scenario. No problem.

I tried switching clang for gcc just to rule out problems related to clang, and the error persists with gcc as well.

Mitigated the problem by making the coro stack 80k

How do you do that. Thorugh configuration or in the build?

@arsatiki
Copy link
Author

Configuration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants