JavaScript pages

[psa-neutronian]

I'm attempting to use Storm Crawler for JavaScript pages and am having some difficulty in creating a generalised solution. Also, it seems that Storm Crawler doesn't play nice with WebDriver for Firefox, aka Gecko Driver (or maybe non-Puppeteer WebDriver in general, I haven't gotten that far yet).

So, I have a two part question:

1. Has anyone else successfully used Storm Crawler with Gecko Driver?
2. Any suggestions on where to start for getting Storm Crawler to do a generalised search through rendered HTML for a link matching a description (e.g. "about us", or "terms and conditions") then following that link, which may be XHR, and grabbing said text? For paths to files, the text should be fairly easy (it's finding the links when they're JavaScript based that's challenging), for overlays, I'm completely stumped.

[jnioche]

I have not used Gecko Driver at all.
There is a selenium protocol implementation; I described its use in this blog

[psa-neutronian]

Thanks, that blog post was my starting point and very helpful.

I was hoping someone else had already done the work to update it from a Selenium specific implementation to a general WebDriver implementation (at least when I gave it a go, Gecko Driver didn't function with the current implementation, I wasn't able to get it to work). Selenium is dead, so I was looking at how to use Firefox and/or Chrome (Puppeteer is popular, but currently Chrome focused, thus the WebDriver preference).

Any suggestions on navigating overlays and XHR content?

[sebastian-nagel]

What about Playwright for Java?

[jnioche]

@psa-neutronian any luck with Playwright?

[rzo1]

We now have Playwright protocol ready for usage.