enhance support for syslog ingestion

[mmguero]

An INL-internal group using Malcolm was discussing Malcolm's ability to ingest host logs with us, and syslog in particular. As syslog is a common format for linux-and-related host logs, this might be worth looking at.

There are three parts to this I can see:

• From the aspect of collecting the syslogs, fluent bit has a syslog input that will monitor a TCP or UDP socket for syslogs sent to it.
  • In other words, on some other box you could set up a socket that rsyslog is sending to, then have fluent-bit monitor that socket and forward things along to Malcolm in the regular JSON-over-TCP/TLS method we do for everything else.
  • Our logstash pipeline already has support to parse these so in a major sense this is already working and supported
  • we don't have a prebuilt dashboard for this so we would want to build one (similar to the journald/systemd dashboard found under Malcolm and Third-Party Logs in the navigation panel)
• Now, what about Malcolm directly accepting syslogs itself, rather than having an external fluent-bit process forwarding them?
  • filebeat has a syslog processor, so we could leverage another filebeat process (filebeat-syslog) in the filebeat container (similar to how we added the one filebeat TCP somewhat recently); this is probably the way to go
  • logstash has a syslog input that would be another options
  • we'd want to normalize, as much as possible, so that the inputs from syslog directly and the inputs from fluent-bit have the same data/field format and are viewable in the same dashboard (use ECS as much as possible)
  • this would potentially have to expose another port, but I think that if we do it it merits a separate question in the configuration as it's unencrypted/unauthenticated and as such doesn't rise to the standard of the stuff we're doing for filebeat-tcp/5045 right now. I think we'd want to give them a more explicit "are you sure" kind of thing.

[mmguero]

Kamino closed and cloned this issue to cisagov/Malcolm