Improve rule loading times in rule tree

[ppcad]

• Fix cnf to dnf conversion and improve its performance
• Optimize rule tree sorter performance
• Add rule tree changes to readme and changelog

[ekneg54]

as we reimplement metrics now, I suggest to wait with this to get vision about what is actual slow on loading rules and wich rules.

Am I right, that improve rule loading times is only done because restarting pipelines seems to be slow?
There are multiple things that could getting slow. Loading lists? loading rules? And so there are different ways to deal with them. But before dealing with them we should get vision.

We implement some additional metrics for restarting pipelines. Having that, you should be able to identify what is actual slow in your configuration.

A timeout feels like a workaround and does not solves the actual problem with the risk that some rules will be missing in processing.

[ppcad]

as we reimplement metrics now, I suggest to wait with this to get vision about what is actual slow on loading rules and wich rules.

Am I right, that improve rule loading times is only done because restarting pipelines seems to be slow? There are multiple things that could getting slow. Loading lists? loading rules? And so there are different ways to deal with them. But before dealing with them we should get vision.

We implement some additional metrics for restarting pipelines. Having that, you should be able to identify what is actual slow in your configuration.

A timeout feels like a workaround and does not solves the actual problem with the risk that some rules will be missing in processing.

This change does include an optimization of rule sorting and dnf conversion. The algorithms did some redundant looping.
The dnf conversion can still explode in load time, even if the implementation would be perfectly optimized.
The conversion has simply such a high complexity.
So the timeout is needed to filter out those cases.
It does furthermore fix a bug with rule matching (see tests).

[ppcad]

I removed the timeout.
Please have a look @ekneg54.

[ekneg54]

minor change in Changelog please

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e085fe1) 92.23% compared to head (3c6aa2c) 92.33%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #464      +/-   ##
==========================================
+ Coverage   92.23%   92.33%   +0.10%     
==========================================
  Files         130      130              
  Lines        9478     9486       +8     
==========================================
+ Hits         8742     8759      +17     
+ Misses        736      727       -9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.