The problem with Logstash in Malcolm k8s

[alleniverson33]

version: 23.08.1

Logstash always shows ERROR logs during runtime，Is this a normal phenomenon？
Wait a few minutes and the data will also be inserted into OpenSearch

[mmguero]

There is always some amount of error messages during Logstash's initial startup while waiting for the services it connects to be available. For example, here is the output from a working instance of logstash starting up I'm running right now, which I got with ./scripts/logs -s logstash

logstash-1  | logstash
logstash-1  | uid=1001(logstash) gid=1001(logstash) groups=1001(logstash),5(tty)
logstash-1  | 2025-02-10 14:34:01,851 INFO RPC interface 'supervisor' initialized
logstash-1  | 2025-02-10 14:34:01,851 CRIT Server 'inet_http_server' running without any HTTP authentication checking
logstash-1  | 2025-02-10 14:34:01,851 INFO supervisord started with pid 1116
logstash-1  | 2025-02-10 14:34:02,853 INFO spawned: 'logstash' with pid 1122
logstash-1  | 2025-02-10 14:34:03,854 INFO success: logstash entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
logstash-1  |
logstash-1  | Using bundled JDK: /usr/share/logstash/jdk
logstash-1  | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash-1  |
logstash-1  | [2025-02-10T14:34:23,598][INFO ][org.logstash.secret.store.backend.JavaKeyStore] Created Logstash keystore at /usr/share/logstash/config/logstash.keystore
logstash-1  | Created Logstash keystore at /usr/share/logstash/config/logstash.keystore
logstash-1  | opensearch-local is up and healthy at http://opensearch:9200
logstash-1  | Waiting until opensearch-local has index template "malcolm_template"...
logstash-1  | opensearch-local index template "malcolm_template" exists
logstash-1  | 2025/02/10 14:36:14 Setting 'pipeline.batch.delay' from environment.
logstash-1  | 2025/02/10 14:36:14 Setting 'pipeline.batch.size' from environment.
logstash-1  | 2025/02/10 14:36:14 Setting 'pipeline.workers' from environment.
logstash-1  | Using bundled JDK: /usr/share/logstash/jdk
logstash-1  | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash-1  | [2025-02-10T14:36:24,013][INFO ][logstash.runner          ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
logstash-1  | [2025-02-10T14:36:24,014][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.17.0", "jruby.version"=>"jruby 9.4.9.0 (3.1.4) 2024-11-04 547c6b150e OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +indy +jit [x86_64-linux]"}
logstash-1  | [2025-02-10T14:36:24,016][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xmx3g, -Xms3g, -Xss2048k, -XX:-HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/./urandom, -Dlog4j.formatMsgNoLookups=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
logstash-1  | [2025-02-10T14:36:24,039][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
logstash-1  | [2025-02-10T14:36:24,039][INFO ][org.logstash.jackson.StreamReadConstraintsUtil] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
logstash-1  | [2025-02-10T14:36:24,049][INFO ][logstash.settings        ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash-1  | [2025-02-10T14:36:24,050][INFO ][logstash.settings        ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash-1  | [2025-02-10T14:36:24,176][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"d4d30efa-3641-465b-8582-66bf80954bb5", :path=>"/usr/share/logstash/data/uuid"}
logstash-1  | [2025-02-10T14:36:24,399][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
logstash-1  | [2025-02-10T14:36:25,065][INFO ][org.reflections.Reflections] Reflections took 195 ms to scan 1 urls, producing 151 keys and 528 values
logstash-1  | [2025-02-10T14:36:25,805][INFO ][logstash.javapipeline    ] Pipeline `malcolm-output` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
logstash-1  | [2025-02-10T14:36:25,805][INFO ][logstash.javapipeline    ] Pipeline `malcolm-input` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
logstash-1  | [2025-02-10T14:36:25,819][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: send_to. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
logstash-1  | [2025-02-10T14:36:25,822][INFO ][logstash.outputs.opensearch] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["http://opensearch:9200"]}
logstash-1  | [2025-02-10T14:36:25,845][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"malcolm-input", "pipeline.workers"=>3, "pipeline.batch.size"=>75, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>225, "pipeline.sources"=>["/usr/share/logstash/malcolm-pipelines/input/01_beats_input.conf", "/usr/share/logstash/malcolm-pipelines/input/99_input_forward.conf"], :thread=>"#<Thread:0xe78bf2e /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}
logstash-1  | [2025-02-10T14:36:25,961][INFO ][logstash.outputs.opensearch] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://opensearch:9200/]}}
logstash-1  | [2025-02-10T14:36:26,058][WARN ][logstash.outputs.opensearch] Restored connection to OpenSearch instance {:url=>"http://opensearch:9200/"}
logstash-1  | [2025-02-10T14:36:26,122][INFO ][logstash.outputs.opensearch] Cluster version determined (2.18.0) {:version=>2}
logstash-1  | [2025-02-10T14:36:26,127][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"malcolm-output", "pipeline.workers"=>3, "pipeline.batch.size"=>75, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>225, "pipeline.sources"=>["/usr/share/logstash/malcolm-pipelines/output/01_input_internal_os.conf", "/usr/share/logstash/malcolm-pipelines/output/98_finalize.conf", "/usr/share/logstash/malcolm-pipelines/output/99_opensearch_output.conf"], :thread=>"#<Thread:0x3e90fb8e /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}
logstash-1  | [2025-02-10T14:36:26,724][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>0.88}
logstash-1  | [2025-02-10T14:36:26,729][INFO ][logstash.inputs.beats    ] Starting input listener {:address=>"0.0.0.0:5044"}
logstash-1  | [2025-02-10T14:36:26,754][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>0.63}
logstash-1  | [2025-02-10T14:36:26,761][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"malcolm-output"}
logstash-1  | [2025-02-10T14:36:27,146][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"malcolm-input"}
logstash-1  | [2025-02-10T14:36:27,215][INFO ][org.logstash.beats.Server] Starting server on port: 5044
logstash-1  | [2025-02-10T14:36:29,241][INFO ][logstash.javapipeline    ] Pipeline `malcolm-suricata` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
logstash-1  | [2025-02-10T14:36:29,295][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: send_to. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
logstash-1  | [2025-02-10T14:36:29,296][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: send_to. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
logstash-1  | [2025-02-10T14:36:29,439][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"malcolm-suricata", "pipeline.workers"=>3, "pipeline.batch.size"=>75, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>225, "pipeline.sources"=>["/usr/share/logstash/malcolm-pipelines/suricata/01_input_suricata.conf", "/usr/share/logstash/malcolm-pipelines/suricata/11_suricata_logs.conf", "/usr/share/logstash/malcolm-pipelines/suricata/12_suricata_normalize.conf", "/usr/share/logstash/malcolm-pipelines/suricata/13_suricata_convert.conf", "/usr/share/logstash/malcolm-pipelines/suricata/19_severity.conf", "/usr/share/logstash/malcolm-pipelines/suricata/99_suricata_forward.conf"], :thread=>"#<Thread:0x39d41b18 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}
logstash-1  | [2025-02-10T14:36:29,968][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:29,969][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:29,973][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:30,969][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:30,969][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:30,973][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:31,196][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>1.76}
logstash-1  | [2025-02-10T14:36:31,198][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"malcolm-suricata"}
logstash-1  | [2025-02-10T14:36:31,969][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:31,969][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:31,974][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:32,969][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:32,970][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:32,974][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:33,970][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:33,970][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:33,974][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:34,591][INFO ][logstash.javapipeline    ] Pipeline `malcolm-enrichment` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
logstash-1  | [2025-02-10T14:36:34,680][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: send_to. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
logstash-1  | [2025-02-10T14:36:34,970][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:34,970][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:34,974][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:35,158][INFO ][logstash.javapipeline    ] Pipeline `malcolm-beats` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
logstash-1  | [2025-02-10T14:36:35,199][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: send_to. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
logstash-1  | [2025-02-10T14:36:35,970][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:35,975][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:35,976][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:36,205][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"malcolm-beats", "pipeline.workers"=>3, "pipeline.batch.size"=>75, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>225, "pipeline.sources"=>["/usr/share/logstash/malcolm-pipelines/beats/01_input_beats.conf", "/usr/share/logstash/malcolm-pipelines/beats/11_beats_logs.conf", "/usr/share/logstash/malcolm-pipelines/beats/12_lookups.conf", "/usr/share/logstash/malcolm-pipelines/beats/13_normalize.conf", "/usr/share/logstash/malcolm-pipelines/beats/98_finalize.conf", "/usr/share/logstash/malcolm-pipelines/beats/99_beats_forward.conf"], :thread=>"#<Thread:0x6edf09ac /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}
logstash-1  | [2025-02-10T14:36:36,971][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:36,975][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:36,976][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'beats-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:37,403][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>1.2}
logstash-1  | [2025-02-10T14:36:37,407][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"malcolm-beats"}
logstash-1  | [2025-02-10T14:36:37,976][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:37,976][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:53,980][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:53,980][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:54,598][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"malcolm-enrichment", "pipeline.workers"=>3, "pipeline.batch.size"=>75, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>225, "pipeline.sources"=>["/usr/share/logstash/malcolm-pipelines/enrichment/01_input_log_enrichment.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/11_lookups.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/12_type_conv.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/20_enriched_to_ecs.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/21_netbox.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/22_segment_comparison.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/23_severity.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/96_make_unique.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/97_arkimize.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/98_finalize.conf", "/usr/share/logstash/malcolm-pipelines/enrichment/99_opensearch_forward.conf"], :thread=>"#<Thread:0x7c1d585f /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}
logstash-1  | [2025-02-10T14:36:54,980][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:54,980][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:54,980][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:55,980][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:55,980][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:55,987][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:56,580][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>1.98}
logstash-1  | [2025-02-10T14:36:56,584][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"malcolm-enrichment"}
logstash-1  | [2025-02-10T14:36:56,981][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:56,981][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:56,987][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:57,981][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:57,981][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:57,987][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:58,049][INFO ][logstash.javapipeline    ] Pipeline `malcolm-zeek` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
logstash-1  | [2025-02-10T14:36:58,147][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: send_to. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
logstash-1  | [2025-02-10T14:36:58,148][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: send_to. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
logstash-1  | [2025-02-10T14:36:58,675][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"malcolm-zeek", "pipeline.workers"=>3, "pipeline.batch.size"=>75, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>225, "pipeline.sources"=>["/usr/share/logstash/malcolm-pipelines/zeek/0100_input_zeek.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1000_zeek_prep.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1001_zeek_parse.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1011_zeek_bacnet.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1012_zeek_bestguess.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1013_zeek_bsap.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1014_zeek_conn.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1015_zeek_dce_rpc.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1016_zeek_dhcp.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1017_zeek_diagnostic.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1018_zeek_dnp3.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1019_zeek_dns.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1020_zeek_ecat.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1021_zeek_enip.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1022_zeek_files.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1023_zeek_ftp.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1024_zeek_genisys.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1025_zeek_ge_srtp.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1026_zeek_gquic.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1027_zeek_hart_ip.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1028_zeek_http.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1029_zeek_intel.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1030_zeek_ipsec.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1031_zeek_irc.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1032_zeek_kerberos.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1033_zeek_known.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1034_zeek_ldap.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1035_zeek_login.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1036_zeek_modbus.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1037_zeek_mqtt.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1038_zeek_mysql.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1039_zeek_notice.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1040_zeek_ntlm.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1041_zeek_ntp.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1042_zeek_ocsp.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1043_zeek_opcua_binary.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1044_zeek_ospf.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1045_zeek_pe.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1046_zeek_profinet.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1047_zeek_radius.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1048_zeek_rdp.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1049_zeek_rfb.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1050_zeek_s7comm.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1051_zeek_signatures.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1052_zeek_sip.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1053_zeek_smb.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1054_zeek_smtp.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1055_zeek_snmp.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1056_zeek_socks.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1057_zeek_software.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1058_zeek_ssh.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1059_zeek_ssl.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1060_zeek_stun.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1061_zeek_synchrophasor.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1062_zeek_syslog.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1063_zeek_tds.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1064_zeek_tftp.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1065_zeek_tunnel.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1066_zeek_weird.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1067_zeek_wireguard.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1068_zeek_x509.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1069_zeek_websocket.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1070_zeek_postgresql.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1171_zeek_omron_fins.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1199_zeek_unknown.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1200_zeek_mutate.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1300_zeek_normalize.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1400_zeek_convert.conf", "/usr/share/logstash/malcolm-pipelines/zeek/1900_severity.conf", "/usr/share/logstash/malcolm-pipelines/zeek/9900_zeek_forward.conf"], :thread=>"#<Thread:0x5a5e4cec /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:138 run>"}
logstash-1  | [2025-02-10T14:36:58,981][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:36:58,981][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:37:03,990][WARN ][org.logstash.plugins.pipeline.AbstractPipelineBus] Attempted to send event to 'zeek-parse' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry.
logstash-1  | [2025-02-10T14:37:04,610][INFO ][logstash.javapipeline    ] Pipeline Java execution initialization time {"seconds"=>5.94}
logstash-1  | [2025-02-10T14:37:04,614][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"malcolm-zeek"}
logstash-1  | [2025-02-10T14:37:04,627][INFO ][logstash.agent           ] Pipelines running {:count=>6, :running_pipelines=>[:"malcolm-output", :"malcolm-input", :"malcolm-suricata", :"malcolm-enrichment", :"malcolm-beats", :"malcolm-zeek"], :non_running_pipelines=>[]}

At the bottom of that text I get the Pipelines running message which means it's finally up and initialized. At that point the error messages mostly go away, although there may be some warnings during runtime which is normal.

Within a few minutes if you're getting data into logstash I'd say it's working fine and those errors are part of normal startup while logstash/filebeat/opensearch all get synced up and connected.

Version 23.08.1 is way out of date though, if possible I'd look for an opportunity to update.

[alleniverson33]

Um. What I want to make clear is that Logstash occasionally shows an ERROR log after running for a long time, which shows that OpenSearch cannot connect, but I executed curl in the Logstash pod http://localhost:9200 It can return OpenSearch information.

[mmguero]

Logstash will occasionally have its connection timeout and then reestablish it, as long as you continue to get events it should be nothing to worry about.