Distributed and easy to use

A tremendous work was done:

• distributed-frontera and frontera were merged together into the single project: to make it easier to use and understand,
• Backend was completely redesigned. Now it's consisting of Queue, Metadata and States objects for low-level code and higher-level Backend implementations for crawling policies,
• Added definition of run modes: single process, distributed spiders, distributed spider and backend.
• Overall distributed concept is now integrated into Frontera, making difference between usage of components in single process and distributed spiders/backend run modes clearer.
• Significantly restructured and augmented documentation, addressing user needs in a more accessible way.
• Much less configuration footprint.

Enjoy this new year release and let us know what you think!