Replace broken links on import + add a script to download links #64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We noticed that some links 404 but still return a 200 level status code. This is very annoying. It is also not something we can do much about, as we only control one website. The upshot is that our external link checker can't do anything with these links. If we check for that URL, it will say 200 all good, even if the user is actually seeing the Page not found message. However, we can find/replace stuff that we import, so if we know what URLs have this behaviour, we can resolve the problem when the file is imported. Right now there are 3 links we have identified: | bad link | good link | |--------|--------| | [www.grants.gov/web/grants/search-grants.html](https://www.grants.gov/web/grants/search-grants.html) | [grants.gov/search-grants](https://grants.gov/search-grants) | | [www.grants.gov/web/grants/forms/sf-424-family.html](https://www.grants.gov/web/grants/forms/sf-424-family.html) | [grants.gov/forms/forms-repository/sf-424-family](https://grants.gov/forms/forms-repository/sf-424-family) | | [www.cdc.gov/grants/dictionary/index.html](https://www.cdc.gov/grants/dictionary/index.html) | [www.cdc.gov/grants/dictionary-of-terms/](https://www.cdc.gov/grants/dictionary-of-terms/) |
pcraig3
force-pushed
the
grants-gov-get-better
branch
from
a3427d0
to
0c55039
Compare
Pretty simple: added a script that loops through all non-archived NOFOs and will pull all external links. This helps us do analysis on them if we need to for some reason. (In this case, our reason was to manually replace bad links)
We were not using this, it was a one-off. Plus, the new script handles this same functionality. The best code is no code at all.™️
pcraig3
force-pushed
the
grants-gov-get-better
branch
from
0c55039
to
cef5f3b
Compare
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR does 3 things:
1. auto-replace known broken links on import
We noticed that some links 404 but still return a 200 level status code. This is very annoying. It is also not something we can do much about, as we only control one website.
However, we can find/replace stuff that we import, so if we know what URLs have this behaviour, we can resolve the problem when the file is imported. Right now there are 3 links we have identified:
So that's super and we love it.
2. adds a script that we can use to grab all links across all NOFOs
Pretty simple: added a script that loops through all non-archived NOFOs and will pull all external links. This helps us do analysis on them if we need to for some reason. (In this case, our reason was to manually replace bad links)
3. Removes an old view that would download a list of links from an individual NOFO
We don't need it! So let's axe it.