Generate links from the page dump instead of using the links dump #21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Merged
Previously, it would leave partially-processed files (jamming up further stages), or even continue executing (for Python).
latest-md5sums.txt is not kept up to date, so every download phase would fail because the hashes don't match.
Aria2c is a commandline torrent downloader that is *much* faster than individual Wikipedia HTTP mirrors (>10MB/s for many files). It will be optionally used if present in $PATH.
rmmh
force-pushed
the
generate-from-pages
branch
from
7ae2158 to
bcaf067
Compare
This lets us ignore noisy links from footer sections (See Also, References, etc) and templates and simplifies the pipeline. Extracting links in Go takes ~160 CPU minutes and <8GB RAM. The 14GB multistream page has many independent bzip streams, allowing it to be processed in parallel in <30 real minutes on a 4C/8T workstation and potentially even less on a large VM instance.
rmmh
force-pushed
the
generate-from-pages
branch
from
bcaf067 to
3dcf67a
Compare
|
Thanks for this PR! I really appreciate your excitement and effort. Unfortunately, I've never written Go and I don't quite feel comfortable accepting so much critical Go code into my database build pipeline. I know how many edge cases I came across from simply parsing the Wikipedia SQL files, I can only imagine how many there will be when parsing the page text itself. If a bug does come up in the Go code, I'll likely be unable to solve it without a lot of wasted effort. So I think I'm going to close this and maybe revisit #19 another time. Thanks again! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This lets us ignore noisy links from footer sections (See Also,
References, etc) and templates and simplifies the pipeline.
Extracting links in Go takes ~160 CPU minutes and <8GB RAM.
The 14GB multistream page has many independent bzip streams, allowing it
to be processed in parallel in <30 real minutes on a 4C/8T workstation
and potentially even less on a large VM instance.
This is based on #20, and includes all of its commits.
Fixes #19.