zimcheck performance improvement

[MiguelRocha]

zimcheck performance is quite important since for big articles can take several minutes (maybe hours).

There are a few performance improvements that can be done in zimcheck specific code:

• Decrease the number of function calls to getLinks()
• Improve getLinks() performance
• Improve isExternalUrl()

The first 2 points are responsible for 18% while the last one takes 20% speedtime.

Also, there maybe some performance gains if we find a better way to validate the URLs. Currently we are calling getArticle() from libzim and this takes around 50% of the time since it needs access the direntCache. (related: openzim/libzim#385 (comment))

[kelson42]

@MiguelRocha Good to me to implement. If the amount of changes is subtantial, please split this ticket.

[veloman-yunkan]

@MiguelRocha Do you mind if I work on this in combination with optimizations on the libzim side?

[MiguelRocha]

@veloman-yunkan please feel free to do it. And let me know if you need any help. I have a draft branch that already fixes the first point that I described and I will create a PR after finishing other issues.

[kelson42]

@MiguelRocha Would you be able to move the isExternalUrl() in an other ticket please
@veloman-yunkan Thank you for your offer, I propose @MiguelRocha make a PR with what he already has done and then let discuss this ticket again.

[kelson42]

@mgautierfr @MiguelRocha I believe this should not be closed because the following points need to be implemented:

• Improve getLinks() performance
• Improve isExternalUrl()

[mgautierfr]

Yes, it was automatically closed when I've merged #162.

[kelson42]

There is no obvious and consensual solution to optimize getLinks() and isExternalUrl(). Closing for the moment and we will open a new ticket if a clear (speed?) problem happens again.