Mikrotik: Add parser for multiline DHCP events

[ehlo550]

What is the sc4s version?
3.32.0

Is there a pcap available? If so, would you prefer to attach it to this issue or send it to Splunk support?
Splunk support

What the vendor name?
Mikrotik

What's the product name?
routeros

Do you have syslog documentation or a manual for that device??
https://help.mikrotik.com/docs/spaces/ROS/pages/328094/Log

Feature Request description:
This routers are able to emit dhcp logs.
Unfortunately these logs are Multiline logs with indentation

<24>Oct 24 10:24:01 ab-cde2fgh01 AA-BB-PROD received request id 1234567891 from 10.10.10.10 '1:0:11:22:33:44:aa'
<24>Oct 24 10:24:01 ab-cde2fgh01     secs = 1
<24>Oct 24 10:24:01 ab-cde2fgh01     ciaddr = 10.10.10.10
<24>Oct 24 10:24:01 ab-cde2fgh01     chaddr = 00:11:22:33:44:AA
<24>Oct 24 10:24:01 ab-cde2fgh01     Host-Name = "serverling01"
<24>Oct 24 10:24:01 ab-cde2fgh01     Msg-Type = request
<24>Oct 24 10:24:01 ab-cde2fgh01     Parameter-List = Subnet-Mask,Unknown(2),Domain-Server,Host-Name,Domain-Name,Interface-MTU,Broadcast-Address,Classless-Route,Router,Static-Route,Unknown(40),Unknown(41),NTP-Server,Domain-Search,MS-Classless-Route,Auto-Proxy-Config,Unknown(17)
<24>Oct 24 10:24:01 ab-cde2fgh01     Max-DHCP-Message-Size = 65535
<24>Oct 24 10:24:01 ab-cde2fgh01     Client-Id = 01-00-11-22-33-44-AA
<24>Oct 24 10:24:01 ab-cde2fgh01 lease bound, extending
<24>Oct 24 10:24:01 ab-cde2fgh01 AA-BB-PROD on vlan123-ABC-AA-BB-PROD sending ack with id 1234567891 to 10.10.10.10
<24>Oct 24 10:24:01 ab-cde2fgh01     ciaddr = 10.10.10.10
<24>Oct 24 10:24:01 ab-cde2fgh01     yiaddr = 10.10.10.10
<24>Oct 24 10:24:01 ab-cde2fgh01     siaddr = 10.10.10.254
<24>Oct 24 10:24:01 ab-cde2fgh01     chaddr = 00:11:22:33:44:AA
<24>Oct 24 10:24:01 ab-cde2fgh01     Subnet-Mask = 255.255.255.0
<24>Oct 24 10:24:01 ab-cde2fgh01     NTP-Server = 10.10.10.253,10.10.11.10,10.10.12.11
<24>Oct 24 10:24:01 ab-cde2fgh01     Address-Time = 3600
<24>Oct 24 10:24:01 ab-cde2fgh01     Msg-Type = ack
<24>Oct 24 10:24:01 ab-cde2fgh01     Server-Id = 10.10.10.254

Do you want to have it for local usage or prepare a github PR?
I would take either

[cwadhwani-splunk]

Hello @ehlo550 ,
Upon reviewing this multiline issue, we've noted that grouping-by() is typically used to handle multiline events by detecting a specific identifier in the last line to signal the end of an event, allowing all preceding messages to be grouped as a single log. However, for MikroTik RouterOS logs, it appears that identifying a unique identifier in the last line may not be straightforward.

To develop a more generalized parser, could you provide additional log samples for further analysis? This would allow us to look for the patterns across various event types. You can create a support ticket and attach the PCAP file there.

[ehlo550]

Hi,
Sure I already have a support ticket open.
Can i provide the support ticket number via Slack ?

Regards
Stefan

[cwadhwani-splunk]

Yes that will work. You can also post the support ticket number here, this will help in better tracking.

[ehlo550]

Ok.
Case # 3603201

I will add the pcap to the case but I fear this mikrotik device only emits dhcp logs.

[cwadhwani-splunk]

We checked the pcap and observed multiple multiline logs, but we could not find enough information that can be used to combine the logs.

Here is a brief about how we group multine logs into one.

We usually use grouping-by() parser method, which allows multiple logs to be combined into a single log before forwarding them to Splunk. This method works by defining a specific triggering condition to decide when the logs should be grouped. Common trigger conditions include:

• A particular pattern in the last log, such as a prefix or unique identifier (reliable).
• Reaching a certain log length (unreliable in this case).
• A timeout condition (unreliable).

When the trigger condition is met, all logs, including the one that satisfies the condition, are grouped and sent to Splunk as a single entry. But here in case of the Mikrotik multiline logs, we've noticed there is no consistent pattern marking the end of each log entry. This lack of a clear trigger point makes it challenging to identify log boundaries and to develop an effective parser based on these criteria. Additionally, only the first log in each group contains the program value, whereas we aim to locate this value in all the logs of each group to assign the program to the grouped logs accurately (this is not necessary until we have something to identify that the log is supposed to be grouped).

Without these details it is difficult to make reliable parser. I am closing this ticket for now, please feel free to reopen if you have some insights that could help us write this parser.