Release 2.4.1-alpha1

Release Notes for OX Abuse Shield 2.4.1 (Alpha1)

New Features

• Dynamic management of siblings via Lua functions and REST API
• Optional per-sibling encryption keys
• Packaging for Amazon Linux in pdns-builder

Bug Fixes/Changes

• None

Dynamic Management of Siblings via Lua functions

Before this release, siblings could only be defined as part of the startup configuration;
there was no way to add or remove siblings dynamically while wforce was running. With this
release all sibling management functions in Lua can be used from the console to add/remove
siblings at runtime. In addition, per-sibling encryption keys can optionally be specified.

The complete set of sibling management functions is as follows:

• setSiblings()
• setSiblingsWithKey() (New)
• addSibling()
• addSiblingWithKey() (New)
• removeSibling() (New)

For full details, see the wforce.conf man page.

Dynamic Management of Siblings via REST API

New REST API endpoints enable siblings to be managed dynamically.

The new REST API endpoints are as follows:

• /?command=addSibling
• /?command=removeSibling
• /?command=setSibling

For more details see the wforce OpenAPI specification, which is available at https://powerdns.github.io/weakforced/

Note that the REST API does not currently support TLS natively, so use of a HTTPS reverse proxy on localhost
is strongly recommended when specifying per-sibling encryption keys.

Optional Per-Sibling Encryption Keys

All the methods of managing siblings (Lua or REST API) enable per-sibling encryption keys to be set.
Encryption keys are are 32-byte strings that are Base-64 encoded before passing to the sibling
management functions or REST API.

Release 2.4.0

Release Notes for OX Abuse Shield 2.4.0

New Features

• New wf_dump_entries tool to dump stats DBs to file
• Support for new "forwarding" type in replication messages
• Support for Prometheus via the new /metrics REST endpoint
• Add Date, Last-Modified and Cache-Control headers to all responses
• Session-ID now logged for allow/report commands
• Improved logging to show line numbers for Lua errors
• Enable TCP keepalive for redis connections
• Configurable timeout for R/W on redis connections

Bug Fixes/Changes

• Fix duplicate command stats under some circumstances

New wf_dump_entries Tool

New tool to dump the contents of Stats DBs for debugging purposes.
man wforce_dump_entries for more information.

Forwarding Type in Replication Messages

Replication messages now have a 'forwarding' flag, which is used to
indicate when a message has been forwarded. This can be used to
prevent forwarding loops.

Support for Prometheus Metrics

Both the wforce and trackalert daemons support native Prometheus
metrics via the new /metrics REST API endpoint. This endpoint follows
the format described
here.

The prometheus metrics deprecate the existing metrics functionality
including the following console commands:

• showPerfStats()
• showCommandStats()
• showCustomStats()

as well as the following REST API endpoints:

• /?command=stats

The prometheus metrics include metrics for many components that were
not previously instrumented, including:

• Redis statistics
• DNS Queries
• Whitelists and blacklists

New HTTP Response Headers

All HTTP responses now include the following headers:

• Last-Modified
• Date
• Cache-Control

Last-Modified and Date headers will always reflect the current
date/time as seen by the wforce server.

Session-ID Logging

The allow and report logs will now contain session_id information.

Improved Logging for Lua Errors

The Lua wrapper code has been updated to provide better traceback
information, including line numbers, for Lua errors. This helps when
writing Lua policy that triggers a Lua exception.

Enable TCP keepalive for redis connections

Redis connections could be timed-out by middleboxes, which would not
be detected because keepalive was not enabled for Redis
connections. Now it is enabled (always).

Configurable timeout for R/W on redis connections

Previously reads from and writes to Redis were subject to the
underlying socket timeout defaults. Now the timeout defaults to 100000
microseconds, and is configurable with new Lua functions:
blacklistPersistRWTimeout() and whitelistPersistRWTimeout(). See
wforce.conf for more details.

Fix Duplicate Command Stats

Under certain circumstances, relating to EOF handling when sockets are
closed, REST API command statistics would be double counted. This has
been fixed by refactoring the EOF handling code.

Release 2.4.0-rc1

Release Notes for OX Abuse Shield 2.4.0

New Features

• New wf_dump_entries tool to dump stats DBs to file
• Support for new "forwarding" type in replication messages
• Support for Prometheus via the new /metrics REST endpoint
• Add Date, Last-Modified and Cache-Control headers to all responses
• Session-ID now logged for allow/report commands
• Improved logging to show line numbers for Lua errors
• Enable TCP keepalive for redis connections
• Configurable timeout for R/W on redis connections

Bug Fixes/Changes

• Fix duplicate command stats under some circumstances

New wf_dump_entries Tool

New tool to dump the contents of Stats DBs for debugging purposes.
man wforce_dump_entries for more information.

Forwarding Type in Replication Messages

Replication messages now have a 'forwarding' flag, which is used to
indicate when a message has been forwarded. This can be used to
prevent forwarding loops.

Support for Prometheus Metrics

Both the wforce and trackalert daemons support native Prometheus
metrics via the new /metrics REST API endpoint. This endpoint follows
the format described
here.

The prometheus metrics deprecate the existing metrics functionality
including the following console commands:

• showPerfStats()
• showCommandStats()
• showCustomStats()

as well as the following REST API endpoints:

• /?command=stats

The prometheus metrics include metrics for many components that were
not previously instrumented, including:

• Redis statistics
• DNS Queries
• Whitelists and blacklists

New HTTP Response Headers

All HTTP responses now include the following headers:

• Last-Modified
• Date
• Cache-Control

Last-Modified and Date headers will always reflect the current
date/time as seen by the wforce server.

Session-ID Logging

The allow and report logs will now contain session_id information.

Improved Logging for Lua Errors

The Lua wrapper code has been updated to provide better traceback
information, including line numbers, for Lua errors. This helps when
writing Lua policy that triggers a Lua exception.

Enable TCP keepalive for redis connections

Redis connections could be timed-out by middleboxes, which would not
be detected because keepalive was not enabled for Redis
connections. Now it is enabled (always).

Configurable timeout for R/W on redis connections

Previously reads from and writes to Redis were subject to the
underlying socket timeout defaults. Now the timeout defaults to 100000
microseconds, and is configurable with new Lua functions:
blacklistPersistRWTimeout() and whitelistPersistRWTimeout(). See
wforce.conf for more details.

Fix Duplicate Command Stats

Under certain circumstances, relating to EOF handling when sockets are
closed, REST API command statistics would be double counted. This has
been fixed by refactoring the EOF handling code.

Release 2.2.2

Release Notes for OX Abuse Shield 2.2.2

Bug Fixes/Changes

• Fix control socket leak when client closes connection immediately

New Features

• Support building debian buster packages
• StatsDBs can now be sharded for better performance (man wforce.conf
  for more info)
• StatsDB expiry thread now runs more often by default
• StatsDB expiry thread sleep time is now configurable (man
  wforce.conf for more info)
• Support for Kafka REST Proxy in webhooks (man wforce_webhook for
  more info)

Release 2.2.1

Release Notes for OX Abuse Shield 2.2.1

Bug Fixes/Changes

• Fix wforce crash in Sibling send thread triggered by syncDB operation

Release 2.2.0

Release Notes for OX Abuse Shield 2.0.0

New Features

• Lookup individual GeoIP2 values (Unsigned Integer, String,
  Double/Float and Boolean)
• Add session_id to wforce LoginTuple
• The Report API is now deployable via proper packaging, systemd and gunicorn
• Custom GET Endpoints supporting non-JSON return values
• New Kibana Reports and Dashboard to view effectiveness of wforce
  policy
• New "type" field added to built-in webhooks
• Built-in whitelists added in addition to built-in blacklists
• New checkBlacklist and checkWhitelist functions in Lua
• Built-in black/whitelisting can be disabled and checked from Lua
  instead
• Thread names support
• Built-In Blacklist and Whitelist return messages are configurable
• Support TCP Keepalive

Bug Fixes/Changes

• Fix typo in wforce.conf.example blackistLogin->blacklistLogin
• Python3 support throughout wforce (regression tests and deployment)
• Uses "bytes" instead of "string" in replication, which avoids
  protobuf errors for non-UTF-8 StatsDB keys or values.
• Remove ctpl from wforce
• Sibling threads now use explicit std::queue instead of ctpl
• Only connect to TCP siblings when necessary (not on startup) to
  prevent delays on startup.
• Updated logstash.conf file for ELK integration

Lookup Individual GeoIP2 Values

Previously the GeoIP2 support was limited to retrieving City and
County DB information, with a fixed set of fields returned. However
this meant that other DBs (e.g. ISPs, Anonymizers etc.) could not be
queried, as well as additional fields in City or Country DBs.

Now there are four new Lua functions to query arbitrary fields in the
Maxmind DBs:

• lookupStringValue
• lookupDoubleValue
• lookupUIntValue
• lookupBoolValue

For example:

local citydb = getGeoIP2DB("City")
local city_name = citydb:lookupStringValue(newCA(ip_address), {"city", "names", "en"})
local city_lat = citydb:lookupDoubleValue(newCA(ip_address), {"location", "latitude"})
local city_long = citydb:lookupDoubleValue(newCA(ip_address), {"location", "longitude"})
local accuracy = citydb:lookupUIntValue(newCA(ip_address), {"location", "accuracy_radius"})
local eu = citydb:lookupBoolValue(newCA(ip_address), {"country", "is_in_european_union"})

Add session_id to wforce LoginTuple

It can be useful for wforce to recognise when multiple logins belong
to the same session. Therefore a new field "session_id" is added to
the LoginTuple table, which allows a session id to be passed to
wforce. This field is optional, so if empty it should be ignored. For
example:

local session_id = lt.session_id

Deployable Report API

Previously the Report API was supplied as an experimental feature,
which was supplied as a Flask application, but without support for running
in production, packaging etc.

Now the Report API is fully deployable; it is packaged in a new
package "wforce-report-api", and can be started/stopped via
systemd. For example:

systemctl enable wforce-report-api
systemctl start wforce-report-api

The Report API is still a Flash application, which runs under
gunicorn, and the configuration and deployment settings are
configurable.

Report API configuration is via:

/etc/wforce-report-api/wforce-report-api-instance.conf

Gunicorn configuration is via:

/etc/wforce-report-api/wforce-report-api-web.conf

It is recommended to run Gunicorn behind an nginx proxy in a
production deployment.

Custom GET Endpoints

The existing Custom Endpoint functionality is based only on POST
commands, and requires return values to be json-encoded. This presents
problems with more limited clients such as firewalls or other network
equipment, which only support HTTP GET, and cannot parse json-encoded
return values.

The Custom GET endpoints solve this issue by allowing endpoints to be
created which are based on HTTP GET, and which return data using
the text/plain content type.

Custom GET endpoints do not take any parameters, so there is no way to
pass data to the endpoints.

For example the following function uses the new getIPBlacklist()
functionality to return a text version of the IP blacklist:

function returnTextBlacklist()
    local ipbl = getIPBlacklist()
    local ret_table = {}
    for i,j in pairs(ipbl)
    do
        for k,v in pairs(j)
        do
            if k == "ip"
            then
                table.insert(ret_table, v)
            end
        end
    end
    local s =  table.concat(ret_table, "\n") .. "\n"
    unblacklistIP(newCA("1.2.3.4"))
    return s
end

setCustomGetEndpoint("textBlacklist", returnTextBlacklist)

New Kibana Reports and Dashboard

New Kibana Reports and Dashboard have been added to the
kibana_saved_objects.json file. These reports are based on "allow"
webhooks sent to elasticsearch.

New "type" field added to built-in webhooks

The built-in webhooks add a "type" field to the webhook json to make
it easier to search for specific webhook types in elasticsearch. The
type field can have the following values:

• wforce_report
• wforce_allow
• wforce_reset
• wforce_expireblwl
• wforce_addblwl
• wforce_delblwl

Built-in Whitelists

In addition to the built-in blacklists there are now built-in
whitelists. Entries can be added and deleted from the built-in
whitelists using either the REST API or using Lua commands.

For example using the REST API:

curl -XPOST --data '{ "ip":"1.2.3.4" }' -u user:pass http://localhost:8084/?command=addWLEntry

For example using Lua:

whitelistIP(lt.remote, 3600, "This is the reason why it is blacklisted")

New checkBlacklist and checkWhitelist functions in Lua

New functions to check the built-in black and whitelists are available
from Lua:

• checkBlacklistIP
• checkBlacklistLogin
• checkBlacklistIPLogin
• checkWhitelistIP
• checkWhitelistLogin
• checkWhitelistIPLogin

See the wforce.conf man page for more details.

Built-in Black/Whitelisting can be disabled

The built-in black/whitelisting functionality can be disabled so that
the checks can instead be performed from Lua. This is achieved with
the following Lua commands:

disableBuiltinBlacklists()
disableBuiltinWhitelists()

Thread Names Support

Every thread type in wforce is now named, so that top -H for example
will show individual thread names. This can be very useful for
diagnosing when particular threads are CPU-bound and could benefit
from increasing the size of the thread pool for example.

Built-In Blacklist return messages are configurable

The following new functions enable the return messages for built-in blacklists to be configured:

• setBlacklistIPRetMsg
• setBlacklistLoginRetMsg
• setBlackistIPLoginRetMsg

See the wforce.conf man page for more details.

Support TCP Keepalive

The wforce daemon previously did not enable TCP keepalive on
accepted sockets. The TCP keepalive socketoption is now enabled for
all sockets.

Release 2.2.0-beta1

Release Notes for OX Abuse Shield 2.2.0-beta1

New Features

• Lookup individual GeoIP2 values (Unsigned Integer, String,
  Double/Float and Boolean)
• Add session_id to wforce LoginTuple
• The Report API is now deployable via proper packaging, systemd and gunicorn
• Custom GET Endpoints supporting non-JSON return values
• New Kibana Reports and Dashboard to view effectiveness of wforce
  policy
• New "type" field added to built-in webhooks
• Built-in whitelists added in addition to built-in blacklists
• New checkBlacklist and checkWhitelist functions in Lua
• Built-in black/whitelisting can be disabled and checked from Lua
  instead
• Thread names support
• Built-In Blacklist and Whitelist return messages are configurable
• Support TCP Keepalive

Bug Fixes/Changes

• Fix typo in wforce.conf.example blackistLogin->blacklistLogin
• Python3 support throughout wforce (regression tests and deployment)
• Uses "bytes" instead of "string" in replication, which avoids
  protobuf errors for non-UTF-8 StatsDB keys or values.
• Remove ctpl from wforce
• Sibling threads now use explicit std::queue instead of ctpl
• Only connect to TCP siblings when necessary (not on startup) to
  prevent delays on startup.
• Updated logstash.conf file for ELK integration

Lookup Individual GeoIP2 Values

Previously the GeoIP2 support was limited to retrieving City and
County DB information, with a fixed set of fields returned. However
this meant that other DBs (e.g. ISPs, Anonymizers etc.) could not be
queried, as well as additional fields in City or Country DBs.

Now there are four new Lua functions to query arbitrary fields in the
Maxmind DBs:

• lookupStringValue
• lookupDoubleValue
• lookupUIntValue
• lookupBoolValue

For example:

local citydb = getGeoIP2DB("City")
local city_name = citydb:lookupStringValue(newCA(ip_address), {"city", "names", "en"})
local city_lat = citydb:lookupDoubleValue(newCA(ip_address), {"location", "latitude"})
local city_long = citydb:lookupDoubleValue(newCA(ip_address), {"location", "longitude"})
local accuracy = citydb:lookupUIntValue(newCA(ip_address), {"location", "accuracy_radius"})
local eu = citydb:lookupBoolValue(newCA(ip_address), {"country", "is_in_european_union"})

Add session_id to wforce LoginTuple

It can be useful for wforce to recognise when multiple logins belong
to the same session. Therefore a new field "session_id" is added to
the LoginTuple table, which allows a session id to be passed to
wforce. This field is optional, so if empty it should be ignored. For
example:

local session_id = lt.session_id

Deployable Report API

Previously the Report API was supplied as an experimental feature,
which was supplied as a Flask application, but without support for running
in production, packaging etc.

Now the Report API is fully deployable; it is packaged in a new
package "wforce-report-api", and can be started/stopped via
systemd. For example:

systemctl enable wforce-report-api
systemctl start wforce-report-api

The Report API is still a Flash application, which runs under
gunicorn, and the configuration and deployment settings are
configurable.

Report API configuration is via:

/etc/wforce-report-api/wforce-report-api-instance.conf

Gunicorn configuration is via:

/etc/wforce-report-api/wforce-report-api-web.conf

It is recommended to run Gunicorn behind an nginx proxy in a
production deployment.

Custom GET Endpoints

The existing Custom Endpoint functionality is based only on POST
commands, and requires return values to be json-encoded. This presents
problems with more limited clients such as firewalls or other network
equipment, which only support HTTP GET, and cannot parse json-encoded
return values.

The Custom GET endpoints solve this issue by allowing endpoints to be
created which are based on HTTP GET, and which return data using
the text/plain content type.

Custom GET endpoints do not take any parameters, so there is no way to
pass data to the endpoints.

For example the following function uses the new getIPBlacklist()
functionality to return a text version of the IP blacklist:

function returnTextBlacklist()
    local ipbl = getIPBlacklist()
    local ret_table = {}
    for i,j in pairs(ipbl)
    do
        for k,v in pairs(j)
        do
            if k == "ip"
            then
                table.insert(ret_table, v)
            end
        end
    end
    local s =  table.concat(ret_table, "\n") .. "\n"
    unblacklistIP(newCA("1.2.3.4"))
    return s
end

setCustomGetEndpoint("textBlacklist", returnTextBlacklist)

New Kibana Reports and Dashboard

New Kibana Reports and Dashboard have been added to the
kibana_saved_objects.json file. These reports are based on "allow"
webhooks sent to elasticsearch.

New "type" field added to built-in webhooks

The built-in webhooks add a "type" field to the webhook json to make
it easier to search for specific webhook types in elasticsearch. The
type field can have the following values:

• wforce_report
• wforce_allow
• wforce_reset
• wforce_expireblwl
• wforce_addblwl
• wforce_delblwl

Built-in Whitelists

In addition to the built-in blacklists there are now built-in
whitelists. Entries can be added and deleted from the built-in
whitelists using either the REST API or using Lua commands.

For example using the REST API:

curl -XPOST --data '{ "ip":"1.2.3.4" }' -u user:pass http://localhost:8084/?command=addWLEntry

For example using Lua:

whitelistIP(lt.remote, 3600, "This is the reason why it is blacklisted")

New checkBlacklist and checkWhitelist functions in Lua

New functions to check the built-in black and whitelists are available
from Lua:

• checkBlacklistIP
• checkBlacklistLogin
• checkBlacklistIPLogin
• checkWhitelistIP
• checkWhitelistLogin
• checkWhitelistIPLogin

See the wforce.conf man page for more details.

Built-in Black/Whitelisting can be disabled

The built-in black/whitelisting functionality can be disabled so that
the checks can instead be performed from Lua. This is achieved with
the following Lua commands:

disableBuiltinBlacklists()
disableBuiltinWhitelists()

Thread Names Support

Every thread type in wforce is now named, so that top -H for example
will show individual thread names. This can be very useful for
diagnosing when particular threads are CPU-bound and could benefit
from increasing the size of the thread pool for example.

Built-In Blacklist return messages are configurable

The following new functions enable the return messages for built-in blacklists to be configured:

• setBlacklistIPRetMsg
• setBlacklistLoginRetMsg
• setBlackistIPLoginRetMsg

See the wforce.conf man page for more details.

Support TCP Keepalive

The wforce daemon previously did not enable TCP keepalive on
accepted sockets. The TCP keepalive socketoption is now enabled for
all sockets.

Release 2.0.1

Release Notes for 2.0.1

New Features/Bug Fixes

• Fix issue where blacklisted is always true in getDBStats output
• Fix issue where threadnames were not displayed correctly
• Fix issue where siblings defined as tcp were connected to on startup
• Fix issue where non-UTF-8 login names would cause protobuf erros

Fix blacklisted always true in getDBStats output

Fix unitialized variable causing getDBStats command to always return
blacklisted: true.

Fix Threadnames Display Issue

Threadn support was implemented, but wasn't working due to compile
dependency issues. This is now fixed meaning that top will show thread
names when called with the -H option for example.

Fix Sibling TCP Connect Issue on Startup

Support for sibling using tcp was added in 2.0.0, however when
siblings are defined as TCP, wforce was attempting to connect to them
on startup. If several wforce servers were started at the same time,
this would cause a delay while each tries to connect to the other over
TCP but fails. This fix delays TCP connection until the first
replication attempt. Note that any static blacklist entries in the
config will cause a replication attempt on startup and this will
trigger the same startup delay behaviour. Thus it is not recommended to
create static blacklist entries in the config.

Fix non-UTF-8 Login Name Issue

Replication of data used the protobuf "string" type, which gets
validated on parsing to ensure it is a UTF-8 string. However, certain
fields such as login can contain non-UTF-8 characters, so this fix
changes to use the "bytes" type instead. This fix is backwards
compatible because string and bytes types are identical on the wire.

Release v2.0.0

Release Notes for OX Abuse Shield 2.0.0

New Features/Bug Fixes

• Fix extra : at the end of custom log lines when kv table is empty
• Return additional info about blacklist in allow and getDBStats RESTAPI functions
• New Lua functions to remove blacklist entries
• Add configuration setting "setNumWebHookConnsPerThread"
• Add configuration setting "setWebHookTimeoutSecs"
• Add support for querying replication status in showStringStatsDB()
• Add sibling received success/fail stats to siblings() command
• New custom stats framework
• New stats for all commands, including custom commands
• GeoIP2 support (MMDB-style DBs)
• New twResetField() function for statsDBs
• Configurable accuracy for HLL and CountMin types
• DB Synchronization for newly started wforce instances
• Support for replication over TCP
• Customizable log facility via a command line option
• New trackalert daemon
• Logstash Configuration and Elasticsearch Templates
• Kibana Reports and Dashboards
• Report API

Fix Extra : at end of Custom Log Lines

The Lua infoLog, errorLog etc. functions would previously, when called
as 'errorLog("foo", {})' log:

foo :

Now the same call will log only:

foo

Return additional info about blacklist in allow and getDBStats REST API functions

The allow command will now return additional information when an
IP/Login is blacklisted. The 'r_attrs' object will contain four new
fields:

• expiration - A string showing the date/time when the blacklist will
  expire
• reason - A string stating why the blacklist was created
• key - What was blacllisted, i.e. either ip, login or iplogin
• blacklisted - This will be set to 1

The getDBStats command will return additional information about
blacklisted objects:

• bl_expire - A string showing the date/time when the blacklist will
  expire
• lb_reason - A string stating why the blacklist was created

New Lua functions to remove blacklist entries

The following new Lua functions are available:

• unblacklistNetmask
• unblacklistIP
• unblacklistLogin
• unblacklistIPLogin

See the wforce.conf manpage for more details.

New Configuration Setting setNumWebHookConnsPerThread

The webhook support has been completely refactored in order to achieve
much higher performance with fewer resources. Previously a very high
number of webhook threads was required to achieve good performance,
whereas now a much smaller number of threads can achieve the same
performance.

The previous per-webhook configuration key "num_conns" is no longer
supported. Instead the global configuration setting
"setNumWebHookConnsPerThread" is used.

For example:

            setNumWebHookConnsPerThread(10)

The default is 10 connections per webhook thread.

Add configuration setting "setWebHookTimeoutSecs"

The function setWebHookTimeoutSecs() is used to control the time for
webhook requests, e.g.:

setWebHookTimeoutSecs(2)

Support for querying replication status in showStringStatsDB()

The showStringStatsDB() command now shows whether each StatsDB is
configured for replication or not.

For example:

            > showStringStatsDB()
            DB Name        Repl? Win Size/No Max Size  Cur Size  Field Name       Type
            MyDB1          yes   1/15        524288    0         countLogins      int
                                                                 diffPasswords    hll
            MyDB2          no    600/6       5000      2093      diffIPs          hll

Add Sibling received success/fail stats to siblings() command

The siblings() command now shows success and failure stats about
received messages as well as sent messages.

For example:

    > siblings()
    Address                             Send Successes  Send Failures  Rcv Successes   Rcv Failures     Note
    127.0.0.1:4001                      0               0              17              0                
    127.0.0.1:4002                      0               0              0               0                Self

New Custom Stats Framework

Two new functions, "addCustomStat" and "incCustomStat" can be used
to keep track of custom statistics. A new custom counter is created
with "addCustomStat", e.g.

addCustomStat("custom_stat1")

Custom statistics are counters which track statistics over a 5 minute
period. Every 5 minutes, the current values of all the custom stats
counters are logged to the wforce log file, before the counters are reset.

Stats can be incremented with the "incCustomStat" command:

incCustomStat("custom_stat1")

New stats for all commands

Previously there were no statistics logged for all the REST API
commands; only allow and report commands. Now all REST API commands
are tracked and statistics are reported, including for custom
endpoints created from Lua.

For example:

command stats last 300 secs: addBLEntry=42 allow=393827 allow_allowed=299221 allow_blacklisted=3224 allow_denied=9884 allow_tarpitted=8373 delBLEntry=3 getBL=3949 getDBStats=3229 ping=83764 report=38473 reset=0 stats=0 syncDBs=0 syncDone=0 

custom stats last 300 secs: customFunc1=3401

GeoIP2 support (MMDB-style DBs)

Maxmind are in the process of deprecating the GeoIP Legacy DB support,
therefore this release supports GeoIP2 format databases, i.e. the MMDB
format.

This release therefore deprecates the GeoIP legacy functions, which
will be removed in a later release. The following functions are
deprecated:

• initGeoIPDB()
• initGeoIPCityDB()
• initGeoIPISPDB()
• lookupCountry()
• lookupISP()
• lookupCity()

Due to differences in the way that the GeoIP2 API works, GeoIP
Databases must be opened by specifying the filename of the database to
be used. For example:

newGeoIP2DB("CityDB", "/usr/share/GeoIP/GeoLite2-City.mmdb")

To retrieve a GeoIP DB to conduct queries against, use the following
command:

local citydb = getGeoIP2DB("CityDB")

Once a database has been assigned to a local variable, it can be
queried, for example:

    my_country = countrydb:lookupCountry(newCA("8.8.8.8"))
my_country = countrydb:lookupCountry(lt.remote)

local my_isp = ispdb:lookupISP(newCA("128.243.16.21"))

local gip_record = citydb:lookupCity(lt.remote)
    local my_city = gip_record.city
    local my_latitude = gip_record.latitude

For full details see "man wforce.conf".

New twResetField() function for statsDBs

The twReset() function can be used to reset all the fields for a given
key, but previously there was no way to reset an individual field. Now
the function "twResetField()" can be used to achieve this, e.g.:

statsdb:twResetField(lt.login, "countLogins")

Configurable accuracy for HLL and CountMin types

The HLL and CountMin types of StatsDB entries are probablistic data
structures, which trade accuracy for memory usage. Previously the
accuracy (and thus memory usage) was hardcoded, however now their
accuracy can be tuned. Increasing accuracy however means a
(potentially very large) increase in memory usage, so extreme care
must be taken before modifying these parameters.

The function setHLLBits() can be used to change the accuracy of the
HLL type. The value supplied can be between 4 and 30, with the default
value being 6.

The function setCountMinBits() can be used to set the accuracy of the
CountMin type.

See the wforce.conf manpage for full details.

DB Synchronization for newly started wforce instances

Normally when a wforce instance starts, it has a "fresh" set of Stats
DBs, and therefore can take a reasonable period of time (an hour or
more depending on the policy) before it starts giving the same answers
as other wforce servers in a cluster which have been running for some
time. This issue is now addressed with the ability to tell a wforce
server to find another server which has been running for longer than a
configurable period of time, from which it can download the entire set
of Stats DBs. While a server is in the process of downloading the
Stats DBs from another server, it is in a "warmup" state; this fact is
reflected in a new return value from the "ping" REST API endpoint.

In order to enable this feature, the "addSyncHosts()" function must be
used, once for each host that will be contacted on startup, for
example:

    -- Add 10.2.3.1:8084 as a sync host,
    -- and use the password "super"
    -- Send the DB dump to 10.2.1.1:4001
    -- and let me know on 10.2.1.1:8084 when the dump is finished
    addSyncHost("10.2.3.1:8084", "super", "10.2.1.1:4001", "10.2.1.1:8084") 

The default time that the sync hosts must have been "up" for is 3600
seconds, however that can be configured using
"setMinSyncHostUptime()", e.g.:

    setMinSyncHostUptime(1800)

The replication of data between the sync host and the wforce instance
that is starting up is always performed over TCP.

See wforce.conf for full details.

Support for replication over TCP

The addSiblings() and setSiblings() functions now take an extra
(optional) parameter that specifies whether the replication should use
UDP or TCP. The default is UDP. If the protocol is specified, the port
must also be specified.

For example:

setSiblings({"127.0.1.2", "127.0.1.3:4004", "127.0.2.23:4004:tcp"})
addSibling("192.168.1.23")
    addSibling("192.168.1.23:4001:udp")
    addSibling("192.168.1.23:4003:tcp")

Customizable log facility via a command line option

The new "-f" or "--facility" command line option can be used to set
the syslog facility used for wforce logging.

For example:

wforce -f "local0"

New trackalert daemon

A new daem...

Release v2.0.0-rc1

Release Notes for OX Abuse Shield 2.0.0

New Features/Bug Fixes

• Fix extra : at the end of custom log lines when kv table is empty
• Return additional info about blacklist in allow and getDBStats RESTAPI functions
• New Lua functions to remove blacklist entries
• Add configuration setting "setNumWebHookConnsPerThread"
• Add configuration setting "setWebHookTimeoutSecs"
• Add support for querying replication status in showStringStatsDB()
• Add sibling received success/fail stats to siblings() command
• New custom stats framework
• New stats for all commands, including custom commands
• GeoIP2 support (MMDB-style DBs)
• New twResetField() function for statsDBs
• Configurable accuracy for HLL and CountMin types
• DB Synchronization for newly started wforce instances
• Support for replication over TCP
• Customizable log facility via a command line option
• New trackalert daemon
• Logstash Configuration and Elasticsearch Templates
• Kibana Reports and Dashboards
• Report API

Fix Extra : at end of Custom Log Lines

The Lua infoLog, errorLog etc. functions would previously, when called
as 'errorLog("foo", {})' log:

foo :

Now the same call will log only:

foo

Return additional info about blacklist in allow and getDBStats REST API functions

The allow command will now return additional information when an
IP/Login is blacklisted. The 'r_attrs' object will contain four new
fields:

• expiration - A string showing the date/time when the blacklist will
  expire
• reason - A string stating why the blacklist was created
• key - What was blacllisted, i.e. either ip, login or iplogin
• blacklisted - This will be set to 1

The getDBStats command will return additional information about
blacklisted objects:

• bl_expire - A string showing the date/time when the blacklist will
  expire
• lb_reason - A string stating why the blacklist was created

New Lua functions to remove blacklist entries

The following new Lua functions are available:

• unblacklistNetmask
• unblacklistIP
• unblacklistLogin
• unblacklistIPLogin

See the wforce.conf manpage for more details.

New Configuration Setting setNumWebHookConnsPerThread

The webhook support has been completely refactored in order to achieve
much higher performance with fewer resources. Previously a very high
number of webhook threads was required to achieve good performance,
whereas now a much smaller number of threads can achieve the same
performance.

The previous per-webhook configuration key "num_conns" is no longer
supported. Instead the global configuration setting
"setNumWebHookConnsPerThread" is used.

For example:

            setNumWebHookConnsPerThread(10)

The default is 10 connections per webhook thread.

Add configuration setting "setWebHookTimeoutSecs"

The function setWebHookTimeoutSecs() is used to control the time for
webhook requests, e.g.:

setWebHookTimeoutSecs(2)

Support for querying replication status in showStringStatsDB()

The showStringStatsDB() command now shows whether each StatsDB is
configured for replication or not.

For example:

            > showStringStatsDB()
            DB Name        Repl? Win Size/No Max Size  Cur Size  Field Name       Type
            MyDB1          yes   1/15        524288    0         countLogins      int
                                                                 diffPasswords    hll
            MyDB2          no    600/6       5000      2093      diffIPs          hll

Add Sibling received success/fail stats to siblings() command

The siblings() command now shows success and failure stats about
received messages as well as sent messages.

For example:

    > siblings()
    Address                             Send Successes  Send Failures  Rcv Successes   Rcv Failures     Note
    127.0.0.1:4001                      0               0              17              0                
    127.0.0.1:4002                      0               0              0               0                Self

New Custom Stats Framework

Two new functions, "addCustomStat" and "incCustomStat" can be used
to keep track of custom statistics. A new custom counter is created
with "addCustomStat", e.g.

addCustomStat("custom_stat1")

Custom statistics are counters which track statistics over a 5 minute
period. Every 5 minutes, the current values of all the custom stats
counters are logged to the wforce log file, before the counters are reset.

Stats can be incremented with the "incCustomStat" command:

incCustomStat("custom_stat1")

New stats for all commands

Previously there were no statistics logged for all the REST API
commands; only allow and report commands. Now all REST API commands
are tracked and statistics are reported, including for custom
endpoints created from Lua.

For example:

command stats last 300 secs: addBLEntry=42 allow=393827 allow_allowed=299221 allow_blacklisted=3224 allow_denied=9884 allow_tarpitted=8373 delBLEntry=3 getBL=3949 getDBStats=3229 ping=83764 report=38473 reset=0 stats=0 syncDBs=0 syncDone=0 

custom stats last 300 secs: customFunc1=3401

GeoIP2 support (MMDB-style DBs)

Maxmind are in the process of deprecating the GeoIP Legacy DB support,
therefore this release supports GeoIP2 format databases, i.e. the MMDB
format.

This release therefore deprecates the GeoIP legacy functions, which
will be removed in a later release. The following functions are
deprecated:

• initGeoIPDB()
• initGeoIPCityDB()
• initGeoIPISPDB()
• lookupCountry()
• lookupISP()
• lookupCity()

Due to differences in the way that the GeoIP2 API works, GeoIP
Databases must be opened by specifying the filename of the database to
be used. For example:

newGeoIP2DB("CityDB", "/usr/share/GeoIP/GeoLite2-City.mmdb")

To retrieve a GeoIP DB to conduct queries against, use the following
command:

local citydb = getGeoIP2DB("CityDB")

Once a database has been assigned to a local variable, it can be
queried, for example:

    my_country = countrydb:lookupCountry(newCA("8.8.8.8"))
my_country = countrydb:lookupCountry(lt.remote)

local my_isp = ispdb:lookupISP(newCA("128.243.16.21"))

local gip_record = citydb:lookupCity(lt.remote)
    local my_city = gip_record.city
    local my_latitude = gip_record.latitude

For full details see "man wforce.conf".

New twResetField() function for statsDBs

The twReset() function can be used to reset all the fields for a given
key, but previously there was no way to reset an individual field. Now
the function "twResetField()" can be used to achieve this, e.g.:

statsdb:twResetField(lt.login, "countLogins")

Configurable accuracy for HLL and CountMin types

The HLL and CountMin types of StatsDB entries are probablistic data
structures, which trade accuracy for memory usage. Previously the
accuracy (and thus memory usage) was hardcoded, however now their
accuracy can be tuned. Increasing accuracy however means a
(potentially very large) increase in memory usage, so extreme care
must be taken before modifying these parameters.

The function setHLLBits() can be used to change the accuracy of the
HLL type. The value supplied can be between 4 and 30, with the default
value being 6.

The function setCountMinBits() can be used to set the accuracy of the
CountMin type.

See the wforce.conf manpage for full details.

DB Synchronization for newly started wforce instances

Normally when a wforce instance starts, it has a "fresh" set of Stats
DBs, and therefore can take a reasonable period of time (an hour or
more depending on the policy) before it starts giving the same answers
as other wforce servers in a cluster which have been running for some
time. This issue is now addressed with the ability to tell a wforce
server to find another server which has been running for longer than a
configurable period of time, from which it can download the entire set
of Stats DBs. While a server is in the process of downloading the
Stats DBs from another server, it is in a "warmup" state; this fact is
reflected in a new return value from the "ping" REST API endpoint.

In order to enable this feature, the "addSyncHosts()" function must be
used, once for each host that will be contacted on startup, for
example:

    -- Add 10.2.3.1:8084 as a sync host,
    -- and use the password "super"
    -- Send the DB dump to 10.2.1.1:4001
    -- and let me know on 10.2.1.1:8084 when the dump is finished
    addSyncHost("10.2.3.1:8084", "super", "10.2.1.1:4001", "10.2.1.1:8084") 

The default time that the sync hosts must have been "up" for is 3600
seconds, however that can be configured using
"setMinSyncHostUptime()", e.g.:

    setMinSyncHostUptime(1800)

The replication of data between the sync host and the wforce instance
that is starting up is always performed over TCP.

See wforce.conf for full details.

Support for replication over TCP

The addSiblings() and setSiblings() functions now take an extra
(optional) parameter that specifies whether the replication should use
UDP or TCP. The default is UDP. If the protocol is specified, the port
must also be specified.

For example:

setSiblings({"127.0.1.2", "127.0.1.3:4004", "127.0.2.23:4004:tcp"})
addSibling("192.168.1.23")
    addSibling("192.168.1.23:4001:udp")
    addSibling("192.168.1.23:4003:tcp")

Customizable log facility via a command line option

The new "-f" or "--facility" command line option can be used to set
the syslog facility used for wforce logging.

For example:

wforce -f "local0"

New trackalert daemon

A new daem...