Filtering messages to a different cloudwatch log group

[oc-christopher-billett]

Hi,

Im struggling to filter messages that container a certain string into a different Cloudwatch log group form EKS. I need the other log group to have a shorter retention policy.

LUA

               function check_log(tag, timestamp, record)
                    local message = record["log"] or ""

                    if string.find(message, "11111111") or 
                      string.find(message, "22222222") or 
                      string.find(message, "aaaaaaaa") or 
                      string.find(message, "bbbbbbbbb") then
                        print("Tagging log as 'my.logs'")
                        return 2, timestamp, record, "my.logs"
                    end

                    return 1, timestamp, record
                end

FILTERS

                [FILTER]
                    Name kubernetes
                    Match kube.*
                    Merge_Log On
                    Keep_Log Off
                    K8S-Logging.Parser On
                    K8S-Logging.Exclude On
                [FILTER]
                    Name modify
                    Match *
                    Remove  _p
                    Remove  stream
                    Remove  time
                [FILTER]
                    Name    lua
                    Match   *
                    script  /fluent-bit/scripts/my.lua
                    call    check_log

OUTPUTS

                [OUTPUT]
                    Name cloudwatch_logs
                    Match my.logs
                    region eu-west-2
                    log_retention_days 30
                    log_group_name /aws/eks/fluentbit-cloudwatch/logs
                    log_group_template /aws/eks/fluentbit-cloudwatch/workload/mylogs/$kubernetes['namespace_name']
                    log_stream_prefix fluent-bit-
                    log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
                    auto_create_group On
                [OUTPUT]
                    Name cloudwatch_logs
                    Match   *
                    region eu-west-2
                    log_retention_days 731
                    log_group_name /aws/eks/fluentbit-cloudwatch/logs
                    log_group_template /aws/eks/fluentbit-cloudwatch/workload/$kubernetes['namespace_name']
                    log_stream_prefix fluent-bit-
                    log_stream_template $kubernetes['pod_name'].$kubernetes['container_name']
                    auto_create_group On

I could for a period get the 'Tagging log as 'my.logs' output into the fluent-bit logs, which stopped.

[patrick-stephens]

Do you still want those logs in both? The wildcard match will do that.

[oc-christopher-billett]

No I gave up and used a lambda to split it out and push to its final distination.