Home

LogHub

Loghub is a pipeline log, close to logstash. But it's written in java for improved stability and performance.

It received events from external sources, process them and send them.

All components are organized in many pipeline that can be interconnected. A pipeline goes from one receiver source that generate events, send through processors and forward them to a sender.

Receiver source uses decoders that takes bytes messages and generate an event from that.

Sender source uses decoders that take event and produce bytes message that are then send to the configured destination.

All of these five kind of operator (Receivers, Senders, Processors, Coders and Decoders) are java classes that can be derived for custom usages.

For configuration, it uses a DSL generated using antlr. Its syntax is a strange mix of logstash configuration files, java and a small test of groovy. The exact grammar can be found at https://github.com/fbacchella/LogHub/blob/master/src/main/antlr4/loghub/Route.g4.

It looks like:

input {
    loghub.receivers.ZMQ {
        listen: "tcp://localhost:2120",
        decoder: loghub.decoders.Log4j
    }
} | $main
input {
    loghub.receivers.Udp {
        port: 2121
        decoder: loghub.decoders.Msgpack
    }
} | $apache

output $main | { loghub.senders.ElasticSearch }

pipeline[apache] { 
    loghub.processors.Geoip2 {
        destination:"location" 
    }
}
pipeline[main] {
    log("event", INFO)
    | event.logger_name == "jrds.starter.Timer" || event.info > 4 ? drop  : {
            loghub.processors.ParseJson | loghub.processors.Groovy { script: "println event['logger_name']" }
        }
}
geoip2data: "/user/local/share/GeoIP/GeoIP2-Country.mmdb"
plugins: ["/usr/share/loghub/plugins", "/usr/share/loghub/scripts"]

This configuration define two receivers, one that listen using øMQ for log4j events. The other listen for msgpack encoded events on an udp port, like some that can be generated by mod_log_net.

The events received on UDP are send to one pipeline called "apache". All the events are transferred to the default "main" pipeline after resolving location from visitors.

The log4j events are directly send to the main pipeline, that does some magic treatment on it. Pay attention to the test. It will be evaluated as a groovy scripts.

A property called "plugins" is defined. It allows to define custom extensions folders that will be used to resolve scripts and added to the class path.

In the configuration file, all the agent are defined using directly the class name.