Produce a scrape of a smallish Wiktionary with dev-1.14 for testing

[Jaifroid]

In general, it would be a good idea to do some road-testing of Wikimedia ZIMs other than Wikipedia with the new API (assuming other types also use this), but also there are a few issues where longstanding problems may have been fixed by dev (or others potentially introduced). For example, #2073 and the very similar #1033.

A smallish one with full features might be wiktionary_es_all_max (latest version we have is https://library.kiwix.org/viewer#wiktionary_es_all_maxi_2024-06, was produced by 1.13). This is just 890MB, so seems good for testing.

[audiodude]

Do we need to do this through zimfarm, or is it okay if I just do the scrape on my local machine and send you the ZIM?

[Jaifroid]

It depends whether it needs wider testing on other clients, I suppose. I only really test thoroughly on KJS Browser Extension and on Kiwix PWA. If I find an issue, I try to corroborate that the issue is also on other clients (Kiwix Desktop and Kiwix Android), or whether it's a problem with the JS client(s) only, so I do test a bit more widely, and then report if I find something significant that needs fixing at scraper level or in other clients.

In this case I'll be particularly keen to see whether #2073 is solved or not, and will report back on that.

So, whatever is easiest for you!

[audiodude]

So I'm unable to produce a es.wiktionary.org ZIM right now because of #2003.

I'm getting:

[error] [2024-11-05T19:14:56.416Z] Cannot render [] into an article
[error] [2024-11-05T19:14:56.417Z] Error downloading article awalk
[error] [2024-11-05T19:14:56.417Z] Failed to run mwoffliner after [2611s]: {
	"name": "Error",
	"message": "Cannot render [] into an article"
}
[error] [2024-11-05T19:14:56.417Z] 

**********

Cannot render [] into an article

**********

[Jaifroid]

Thanks for trying @audiodude!

[kelson42]

Tried again with wiktionary_es on Zimfarm and same result:, see https://farm.openzim.org/pipeline/b25161de-d764-4337-844e-19ec0a12afe4/debug

[Jaifroid]

Thanks @kelson42! Hmm, weird error. Who would know what it means?! Looks like some empty array was passed somewhere.

[audiodude]

Thanks @kelson42! Hmm, weird error. Who would know what it means?! Looks like some empty array was passed somewhere.

As linked above, it's #2003. The wiki itself is returning an empty page in the API response.

[kelson42]

I have made a test run with a "small" Wikipedia: wikipedia_ca.

Unfortunately it died pretty quickly, see https://farm.openzim.org/pipeline/d75b3ef3-3fec-4914-8bed-050be72960f7/debug. I suspect here another small bug.

[kelson42]

I have made a test run with a "small" Wikipedia: wikipedia_ca.

Unfortunately it died pretty quickly, see https://farm.openzim.org/pipeline/d75b3ef3-3fec-4914-8bed-050be72960f7/debug. I suspect here another small bug.

I have open an issue for this last one #2113. I'm more concerned if Wikipedia is not scrap-able (for the moment) as Wiktionary.

[kelson42]

Should we keep this issue open? Not sure if it still helps clearly to identify new peoblems?

[Jaifroid]

Well, we haven't yet been able to produce a Wiktionary, and it seems important to be able to test readers not only on Wikipedia ZIMs produced with the new API. It could be useful to keep this open as a reminder until the request has been fulfilled, but it's not my Repo, so up to you!

[benoit74]

I agree that test coverage of 1.14 is very small (insufficient) from my PoV for now.