Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_pgsql: pass connection options to postgresql client #6837

Closed
wants to merge 1 commit into from

Conversation

bkayranci
Copy link
Contributor

@bkayranci bkayranci commented Feb 12, 2023

libpq supports connection options on the postgresql plugin. Passing connection options to client are basic functionality but the plugin cannot pass connection options. I patched it.

#6836


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@bkayranci
Copy link
Contributor Author

bkayranci commented Feb 12, 2023

Example Configuration

fluent/fluent-bit-docs#1047

[SERVICE]
    Log_Level       debug

[INPUT]
    Name cpu
    Tag  my_cpu

[OUTPUT]
    Name                pgsql
    Match               *
    Host                172.17.0.2
    Port                5432
    User                fluentbit
    Password            YourCrazySecurePassword
    Database            fluentbit
    Table               fluentbit
    Connection_Options  project=fluentbit
    Timestamp_Key       ts

Terminal Output

Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/02/12 23:06:30] [ info] Configuration:
[2023/02/12 23:06:30] [ info]  flush time     | 1.000000 seconds
[2023/02/12 23:06:30] [ info]  grace          | 5 seconds
[2023/02/12 23:06:30] [ info]  daemon         | 0
[2023/02/12 23:06:30] [ info] ___________
[2023/02/12 23:06:30] [ info]  inputs:
[2023/02/12 23:06:30] [ info]      cpu
[2023/02/12 23:06:30] [ info] ___________
[2023/02/12 23:06:30] [ info]  filters:
[2023/02/12 23:06:30] [ info] ___________
[2023/02/12 23:06:30] [ info]  outputs:
[2023/02/12 23:06:30] [ info]      pgsql.0
[2023/02/12 23:06:30] [ info] ___________
[2023/02/12 23:06:30] [ info]  collectors:
[2023/02/12 23:06:30] [ info] [fluent bit] version=2.1.0, commit=d9d817d5d0, pid=1
[2023/02/12 23:06:30] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/02/12 23:06:30] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/02/12 23:06:30] [ info] [cmetrics] version=0.5.8
[2023/02/12 23:06:30] [ info] [ctraces ] version=0.3.0
[2023/02/12 23:06:30] [ info] [input:cpu:cpu.0] initializing
[2023/02/12 23:06:30] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2023/02/12 23:06:30] [debug] [cpu:cpu.0] created event channels: read=21 write=22
[2023/02/12 23:06:30] [debug] [pgsql:pgsql.0] created event channels: read=23 write=24
[2023/02/12 23:06:30] [ info] [output:pgsql:pgsql.0] Opening connection: #0
[2023/02/12 23:06:30] [debug] [output:pgsql:pgsql.0] connection_options: project=ep-white-sun-208279
[2023/02/12 23:06:40] [ info] [output:pgsql:pgsql.0] switching postgresql connection to non-blocking mode
[2023/02/12 23:06:40] [ info] [output:pgsql:pgsql.0] host=ep-white-sun-208279.us-east-2.aws.neon.tech port=5432 dbname=neondb OK
[2023/02/12 23:06:40] [ info] [output:pgsql:pgsql.0] we check that the table "logs" exists, if not we create it
NOTICE:  relation "logs" already exists, skipping
[2023/02/12 23:06:40] [ info] [sp] stream processor started
[2023/02/12 23:06:41] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/12 23:06:42] [debug] [task] created task=0x7f4818f4f310 id=0 OK
[2023/02/12 23:06:42] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/12 23:06:42] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/12 23:06:42] [debug] [out flush] cb_destroy coro_id=0
[2023/02/12 23:06:42] [debug] [task] destroy task=0x7f4818f4f310 (task_id=0)
[2023/02/12 23:06:43] [debug] [task] created task=0x7f4818f4f310 id=0 OK
[2023/02/12 23:06:43] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/12 23:06:43] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/12 23:06:43] [debug] [out flush] cb_destroy coro_id=1
[2023/02/12 23:06:43] [debug] [task] destroy task=0x7f4818f4f310 (task_id=0)
[2023/02/12 23:06:44] [debug] [task] created task=0x7f4818f4f310 id=0 OK
[2023/02/12 23:06:44] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/12 23:06:44] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/12 23:06:44] [debug] [out flush] cb_destroy coro_id=2
[2023/02/12 23:06:44] [debug] [task] destroy task=0x7f4818f4f310 (task_id=0)
^C[2023/02/12 23:06:44] [engine] caught signal (SIGINT)
[2023/02/12 23:06:44] [debug] [task] created task=0x7f4818f4f310 id=0 OK
[2023/02/12 23:06:44] [ warn] [engine] service will shutdown in max 5 seconds
[2023/02/12 23:06:44] [ info] [input] pausing cpu.0
[2023/02/12 23:06:44] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/12 23:06:44] [debug] [out flush] cb_destroy coro_id=3
[2023/02/12 23:06:44] [debug] [task] destroy task=0x7f4818f4f310 (task_id=0)
[2023/02/12 23:06:45] [ info] [engine] service has stopped (0 pending tasks)
[2023/02/12 23:06:45] [ info] [input] pausing cpu.0

Valgrind Output

==24716== Memcheck, a memory error detector
==24716== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==24716== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==24716== Command: ./bin/fluent-bit --config ../example.conf
==24716== 
Fluent Bit v2.1.0
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/02/13 16:36:09] [ info] Configuration:
[2023/02/13 16:36:09] [ info]  flush time     | 1.000000 seconds
[2023/02/13 16:36:09] [ info]  grace          | 5 seconds
[2023/02/13 16:36:09] [ info]  daemon         | 0
[2023/02/13 16:36:09] [ info] ___________
[2023/02/13 16:36:09] [ info]  inputs:
[2023/02/13 16:36:09] [ info]      cpu
[2023/02/13 16:36:09] [ info] ___________
[2023/02/13 16:36:09] [ info]  filters:
[2023/02/13 16:36:09] [ info] ___________
[2023/02/13 16:36:09] [ info]  outputs:
[2023/02/13 16:36:09] [ info]      pgsql.0
[2023/02/13 16:36:09] [ info] ___________
[2023/02/13 16:36:09] [ info]  collectors:
[2023/02/13 16:36:09] [ info] [fluent bit] version=2.1.0, commit=58548bd66d, pid=24716
[2023/02/13 16:36:09] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/02/13 16:36:09] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/02/13 16:36:09] [ info] [cmetrics] version=0.5.8
[2023/02/13 16:36:09] [ info] [ctraces ] version=0.3.0
[2023/02/13 16:36:09] [ info] [input:cpu:cpu.0] initializing
[2023/02/13 16:36:09] [ info] [input:cpu:cpu.0] storage_strategy='memory' (memory only)
[2023/02/13 16:36:09] [debug] [cpu:cpu.0] created event channels: read=22 write=23
[2023/02/13 16:36:09] [debug] [pgsql:pgsql.0] created event channels: read=24 write=25
[2023/02/13 16:36:09] [ info] [output:pgsql:pgsql.0] Opening connection: #0
[2023/02/13 16:36:09] [debug] [output:pgsql:pgsql.0] connection_options: project=ep-white-sun-208279
[2023/02/13 16:36:16] [ info] [output:pgsql:pgsql.0] switching postgresql connection to non-blocking mode
[2023/02/13 16:36:16] [ info] [output:pgsql:pgsql.0] host=ep-white-sun-208279.us-east-2.aws.neon.tech port=5432 dbname=neondb OK
[2023/02/13 16:36:16] [ info] [output:pgsql:pgsql.0] we check that the table "logs" exists, if not we create it
NOTICE:  relation "logs" already exists, skipping
[2023/02/13 16:36:17] [ info] [sp] stream processor started
[2023/02/13 16:36:17] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:18] [debug] [task] created task=0x738bd80 id=0 OK
[2023/02/13 16:36:18] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:18] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:18] [debug] [out flush] cb_destroy coro_id=0
[2023/02/13 16:36:18] [debug] [task] destroy task=0x738bd80 (task_id=0)
[2023/02/13 16:36:19] [debug] [task] created task=0x73e2250 id=0 OK
[2023/02/13 16:36:19] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:19] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:19] [debug] [out flush] cb_destroy coro_id=1
[2023/02/13 16:36:19] [debug] [task] destroy task=0x73e2250 (task_id=0)
[2023/02/13 16:36:20] [debug] [task] created task=0x74386a0 id=0 OK
[2023/02/13 16:36:20] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:20] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:20] [debug] [out flush] cb_destroy coro_id=2
[2023/02/13 16:36:20] [debug] [task] destroy task=0x74386a0 (task_id=0)
[2023/02/13 16:36:21] [debug] [task] created task=0x744bdf0 id=0 OK
[2023/02/13 16:36:21] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:21] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:21] [debug] [out flush] cb_destroy coro_id=3
[2023/02/13 16:36:21] [debug] [task] destroy task=0x744bdf0 (task_id=0)
[2023/02/13 16:36:22] [debug] [task] created task=0x7460090 id=0 OK
[2023/02/13 16:36:22] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:22] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:22] [debug] [out flush] cb_destroy coro_id=4
[2023/02/13 16:36:22] [debug] [task] destroy task=0x7460090 (task_id=0)
[2023/02/13 16:36:23] [debug] [task] created task=0x74764a0 id=0 OK
[2023/02/13 16:36:23] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:23] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:23] [debug] [out flush] cb_destroy coro_id=5
[2023/02/13 16:36:23] [debug] [task] destroy task=0x74764a0 (task_id=0)
[2023/02/13 16:36:24] [debug] [task] created task=0x7593860 id=0 OK
[2023/02/13 16:36:24] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:24] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:24] [debug] [out flush] cb_destroy coro_id=6
[2023/02/13 16:36:24] [debug] [task] destroy task=0x7593860 (task_id=0)
[2023/02/13 16:36:25] [debug] [task] created task=0x75e9cd0 id=0 OK
[2023/02/13 16:36:25] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:25] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:25] [debug] [out flush] cb_destroy coro_id=7
[2023/02/13 16:36:25] [debug] [task] destroy task=0x75e9cd0 (task_id=0)
[2023/02/13 16:36:26] [debug] [task] created task=0x7642140 id=0 OK
[2023/02/13 16:36:26] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:26] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:26] [debug] [out flush] cb_destroy coro_id=8
[2023/02/13 16:36:26] [debug] [task] destroy task=0x7642140 (task_id=0)
[2023/02/13 16:36:27] [debug] [task] created task=0x7698560 id=0 OK
[2023/02/13 16:36:27] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:27] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:27] [debug] [out flush] cb_destroy coro_id=9
[2023/02/13 16:36:27] [debug] [task] destroy task=0x7698560 (task_id=0)
[2023/02/13 16:36:28] [debug] [task] created task=0x76ee9b0 id=0 OK
[2023/02/13 16:36:28] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:28] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:28] [debug] [out flush] cb_destroy coro_id=10
[2023/02/13 16:36:28] [debug] [task] destroy task=0x76ee9b0 (task_id=0)
[2023/02/13 16:36:29] [debug] [task] created task=0x7744e20 id=0 OK
[2023/02/13 16:36:29] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:29] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:29] [debug] [out flush] cb_destroy coro_id=11
[2023/02/13 16:36:29] [debug] [task] destroy task=0x7744e20 (task_id=0)
[2023/02/13 16:36:30] [debug] [task] created task=0x779b250 id=0 OK
[2023/02/13 16:36:30] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:30] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:30] [debug] [out flush] cb_destroy coro_id=12
[2023/02/13 16:36:30] [debug] [task] destroy task=0x779b250 (task_id=0)
[2023/02/13 16:36:31] [debug] [task] created task=0x77f36c0 id=0 OK
[2023/02/13 16:36:31] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:31] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:31] [debug] [out flush] cb_destroy coro_id=13
[2023/02/13 16:36:31] [debug] [task] destroy task=0x77f36c0 (task_id=0)
[2023/02/13 16:36:32] [debug] [task] created task=0x7849ae0 id=0 OK
[2023/02/13 16:36:32] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:32] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:32] [debug] [out flush] cb_destroy coro_id=14
[2023/02/13 16:36:32] [debug] [task] destroy task=0x7849ae0 (task_id=0)
[2023/02/13 16:36:33] [debug] [task] created task=0x785fef0 id=0 OK
[2023/02/13 16:36:33] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:33] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:33] [debug] [out flush] cb_destroy coro_id=15
[2023/02/13 16:36:33] [debug] [task] destroy task=0x785fef0 (task_id=0)
[2023/02/13 16:36:34] [debug] [task] created task=0x7876300 id=0 OK
[2023/02/13 16:36:34] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:34] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:34] [debug] [out flush] cb_destroy coro_id=16
[2023/02/13 16:36:34] [debug] [task] destroy task=0x7876300 (task_id=0)
[2023/02/13 16:36:35] [debug] [task] created task=0x794c7f0 id=0 OK
[2023/02/13 16:36:35] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:35] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:35] [debug] [out flush] cb_destroy coro_id=17
[2023/02/13 16:36:35] [debug] [task] destroy task=0x794c7f0 (task_id=0)

[2023/02/13 16:36:36] [debug] [task] created task=0x79a4c80 id=0 OK
[2023/02/13 16:36:36] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:36] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:36] [debug] [out flush] cb_destroy coro_id=18
[2023/02/13 16:36:36] [debug] [task] destroy task=0x79a4c80 (task_id=0)
[2023/02/13 16:36:37] [debug] [task] created task=0x79fb0d0 id=0 OK
[2023/02/13 16:36:37] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:37] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:37] [debug] [out flush] cb_destroy coro_id=19
[2023/02/13 16:36:37] [debug] [task] destroy task=0x79fb0d0 (task_id=0)
[2023/02/13 16:36:38] [debug] [task] created task=0x7a51540 id=0 OK
[2023/02/13 16:36:38] [debug] [input chunk] update output instances with new chunk size diff=575
[2023/02/13 16:36:38] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:38] [debug] [out flush] cb_destroy coro_id=20
[2023/02/13 16:36:38] [debug] [task] destroy task=0x7a51540 (task_id=0)
^C[2023/02/13 16:36:38] [engine] caught signal (SIGINT)
[2023/02/13 16:36:38] [debug] [task] created task=0x7aa79e0 id=0 OK
[2023/02/13 16:36:38] [ warn] [engine] service will shutdown in max 5 seconds
[2023/02/13 16:36:38] [ info] [input] pausing cpu.0
[2023/02/13 16:36:38] [debug] [output:pgsql:pgsql.0] Connection number 0
[2023/02/13 16:36:38] [debug] [out flush] cb_destroy coro_id=21
[2023/02/13 16:36:38] [debug] [task] destroy task=0x7aa79e0 (task_id=0)
[2023/02/13 16:36:39] [ info] [engine] service has stopped (0 pending tasks)
[2023/02/13 16:36:39] [ info] [input] pausing cpu.0
==24716== 
==24716== HEAP SUMMARY:
==24716==     in use at exit: 859,601 bytes in 6,547 blocks
==24716==   total heap usage: 43,930 allocs, 37,383 frees, 11,919,557 bytes allocated
==24716== 
==24716== LEAK SUMMARY:
==24716==    definitely lost: 0 bytes in 0 blocks
==24716==    indirectly lost: 0 bytes in 0 blocks
==24716==      possibly lost: 0 bytes in 0 blocks
==24716==    still reachable: 859,601 bytes in 6,547 blocks
==24716==         suppressed: 0 bytes in 0 blocks
==24716== Rerun with --leak-check=full to see details of leaked memory
==24716== 
==24716== For lists of detected and suppressed errors, rerun with: -s
==24716== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Documentation

fluent/fluent-bit-docs#1047

The PR is available.

Back Compatibility

The plugin pass NULL value by default. So, it will not affect future versions.

@sxd
Copy link
Member

sxd commented Feb 13, 2023

@bkayranci hi! thanks for this! can you add the missing documentation for this?

@bkayranci
Copy link
Contributor Author

@bkayranci hi! thanks for this! can you add the missing documentation for this?

I attached the valgrind output to above comment. Is there any checks for approving?

@bkayranci bkayranci temporarily deployed to pr February 14, 2023 20:21 — with GitHub Actions Inactive
@bkayranci bkayranci temporarily deployed to pr February 14, 2023 20:21 — with GitHub Actions Inactive
@bkayranci bkayranci temporarily deployed to pr February 14, 2023 20:39 — with GitHub Actions Inactive
@patrick-stephens patrick-stephens changed the title pass connection options to postgresql client out_pgsql: pass connection options to postgresql client Feb 15, 2023
@patrick-stephens
Copy link
Contributor

patrick-stephens commented Feb 15, 2023

I wonder if the commit checker regex is failing due to non Ascii characters in your sign off.

It looks fine to me and using your actual name should be acceptable.

Edit: debug logging shows it is getting the old commit message still
https://github.com/fluent/fluent-bit/actions/runs/4165722341/jobs/7247943185#step:2:45

Double check your commit and force push a new one to verify @bkayranci

@bkayranci
Copy link
Contributor Author

I wonder if the commit checker regex is failing due to non Ascii characters in your sign off.

It looks fine to me and using your actual name should be acceptable.

Edit: debug logging shows it is getting the old commit message still https://github.com/fluent/fluent-bit/actions/runs/4165722341/jobs/7247943185#step:2:45

Double check your commit and force push a new one to verify @bkayranci

Actually, it seems on Commits Tab. When I tried to force push, I got "Everything up-to-date". I will push a dummy commit.

@bkayranci
Copy link
Contributor Author

pipeline is broken :(
I cleared non-ascii char in my name but pipeline could not detect the changes.
lets continue on #6875

@sxd

@bkayranci bkayranci temporarily deployed to pr February 17, 2023 17:51 — with GitHub Actions Inactive
@bkayranci bkayranci temporarily deployed to pr February 17, 2023 17:51 — with GitHub Actions Inactive
@bkayranci bkayranci temporarily deployed to pr February 17, 2023 17:51 — with GitHub Actions Inactive
@bkayranci bkayranci temporarily deployed to pr February 17, 2023 18:09 — with GitHub Actions Inactive
@sxd sxd temporarily deployed to pr February 28, 2023 16:10 — with GitHub Actions Inactive
@sxd sxd temporarily deployed to pr February 28, 2023 16:10 — with GitHub Actions Inactive
@sxd sxd temporarily deployed to pr February 28, 2023 16:11 — with GitHub Actions Inactive
@sxd sxd temporarily deployed to pr February 28, 2023 16:35 — with GitHub Actions Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants