v2.8.29

Whats Changed

Fix request interception remote connections. Intercept builder now uses spider::features::chrome_common::RequestInterceptConfiguration and adds more control.

• chrome performance improvement reducing dup events
• chore(chrome): add set extra headers
• chore(smart): add http fallback chrome smart mode request
• chore(chrome): add spoofed plugins
• chore(real-browser): add mouse movement waf
• chore(chrome): patch logs stealth mode
• chore(page): fix url join empty slash
• chore(chrome): fix return page response headers and cookies
• chore(page): add empty page validation
• chore(config): add serializable crawl configuration
• chore(retry): add check 502 notfound retry

Full Changelog: v2.7.1...v2.8.29

v2.7.1

Whats Changed

• add chrome remote connection proxy ability.
• add context handling and disposing chrome.
• chore(chrome): fix concurrent pages opening remote ws connections
• chore(chrome): add cookie setting browser
• chore(chrome): fix connecting to chrome when using a LB
• feat(website): add retry and rate limiting handling

Full Changelog: v2.6.15...v2.7.1

v2.6.15

• fix parsing links for top level redirected domains
• add website.with_preserve_host_header
• default tls reqwest_native_tls_native_roots

Full Changelog: v2.5.2...v2.6.15

HTML Transformations

Whats Changed

We Open Sourced our transformation utils for Spider cloud that provides high performance output to markdown, text, and other formats.

You can install spider_transformations on it's own or use the feature flag transformations when installing spider_utils.

use spider::tokio;
use spider::website::Website;
use spider_utils::spider_transformations::transformation::content::{
    transform_content, ReturnFormat, TransformConfig,
};
use tokio::io::AsyncWriteExt;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("https://rsseau.fr");
    let mut rx2: tokio::sync::broadcast::Receiver<spider::page::Page> =
        website.subscribe(0).unwrap();
    let mut stdout = tokio::io::stdout();

    let mut conf = TransformConfig::default();
    conf.return_format = ReturnFormat::Markdown;

    let join_handle = tokio::spawn(async move {
        while let Ok(res) = rx2.recv().await {
            let markup = transform_content(&res, &conf, &None, &None);

            let _ = stdout
                .write_all(format!("- {}\n {}\n", res.get_url(), markup).as_bytes())
                .await;
        }
        stdout
    });

    let start = std::time::Instant::now();
    website.crawl().await;
    website.unsubscribe();
    let duration = start.elapsed();
    let mut stdout = join_handle.await.unwrap();

    let _ = stdout
        .write_all(
            format!(
                "Time elapsed in website.crawl() is: {:?} for total pages: {:?}",
                duration,
                website.get_links().len()
            )
            .as_bytes(),
        )
        .await;
}

Full Changelog: v2.5.2...v2.6.2

v2.5.3

Whats Changed

1. Add visited links string interning performance increase on look up and memory reduction on store. #204

Full Changelog: v2.4.1...v2.5.3

v2.4.1

Whats Changed

The screenshot performance has drastically increased by taking advantage of chromes params to handle full_screen without re-adjusting the layout and the optimize_for_speed param. This works well the concurrent interception handling to avoid stalling on re-layout. If you use the crawler to take screenshots it is recommended to upgrade.

• perf(chrome): add major screenshot performance custom command
• chore(utils): add trie match all base path
• chore(examples): add css scraping example

Full Changelog: v2.3.5...v2.4.1

v2.3.5

Whats Changed

Major performance improvement on chrome enabling concurrent request interception for resource heavy pages.

• add response headers when chrome is used
• add hybrid cache response and headers chrome
• fix chrome sub pages setup
• perf(chrome): add concurrent request interception

Full Changelog: v2.2.18...v2.3.5

v2.2.18

Whats Changed

We can now auto detect locales without losing out on performance. We default enabled the encoding flag for this change!

• get_html now properly encodes the HTML instead of UTF8 default encoding
• bump [email protected]
• fix chrome hang on ws connections handler
• fix fetch stream infinite loop on error
• fix chrome frame setting url ( this temp prevents hybrid caching from having the req/res for the page )

let mut website: Website = Website::new("https://tenki.jp");
// all of the content output has the proper encoding automatically

Full Changelog: v2.1.9...v2.2.18

v2.1.9

Whats New

This release brings bug fixes with chrome opening pages causing hangs. The builder method website.with_return_page_links can be used to attach the links found on the web page to the page object.

• chore(chrome): fix instances being left open from ignorable handler errors
• chore(scrape): add sitemap and smart [#206]
• feat(page): add return page links configuration
• chore(config): fix budget reset on crawl end

Thanks @DimitriTimoz

Full Changelog: v2.0.6...v2.1.9

v2.0.6

What's Changed

• add http response cookies map
• fix chrome fs feature flag build
• Update README.md by @James4Ever0 in #203

New Contributors

• @James4Ever0 made their first contribution in #203

Full Changelog: v2.0.3...v2.0.6