This release includes several Improvements on the worker component:
-
Improved speed while decompressing files
-
RDFAnalyzer - Detects the fetched file serialization instead of using brute force
-
CkanFetcher - paginates fetched data in order to improve performance.
-
Included an abstract analyzer that has the TripleEncoder class. This class provides the method encodeTriple that encodes the triple characters. All the analyzers are using it.
Also, significant changes were made to the frontier component:
-
Included the UriFilterConfigurator class. This class allows the user to combine multiples filters, for focused crawling. The UriFilterconfigurator requires at least one KnownUriFilter.
-
Included the DepthFilter class. This filter allows depth crawling.
-
Recrawling of outdated URI's.