Update redwood / Add Docker / Fix Scraper #13
Conversation
|
Deploy request for countrycovid19 pending review. Review with commit 36c24b1 |
|
Someone is attempting to deploy a commit to a Personal Account owned by @lachlanjc on Vercel. @lachlanjc first needs to authorize it. |
|
Resolves #12 |
|
This pull request is being automatically deployed with Vercel (learn more). 🔍 Inspect: https://vercel.com/lachlanjc/covid19/ASmkjC8tyoKFtPo4x7nKrNiDcuwJ |
|
Wow, thanks for bringing this back to life!! Yeah, this was one of the first releases of Redwood we ran it on. Some observations/questions:
|
|
Hmm, that's curious. I will try the site out in docker, to make sure there's a straight-forward approach to getting this set up. That should help figure out netlify (don't have much experience with it). I imagine Those new types files were added automatically (and updated frequently!). Might be worth adding them to .gitignore. New scraping just altered the selector for the script tag, and parsed the data differently; that's it. shouldn't be anything new. Maybe the script tags are in a different format now, and maybe that's why the daily stats numbers are in error. I'm not sure what it was doing before though. |
|
Interesting. Yeah, let's ignore the types. The scraper never worked consistently—that was a project we started with the hopes the site could update itself, but I ended up just manually entering the numbers into the database using SQL every day until last summer, when the visualization wasn't that useful anymore. |
| @@ -0,0 +1,38 @@ | |||
| -- CreateTable | |||
| CREATE TABLE "Day" ( | |||
There was a problem hiding this comment.
This new migration is incompatible with production. Likely needs to be reset
| const services = importAll('api', 'services') | ||
| import schemas from 'src/graphql/**/*.{js,ts}' | ||
| import { db } from 'src/lib/db' | ||
| import services from 'src/services/**/*.{js,ts}' |
There was a problem hiding this comment.
All copied from a new redwood project
| deleteCountry | ||
| } from './countries' | ||
|
|
||
| describe('countries', () => { |
There was a problem hiding this comment.
Guessing redwood created these somehow too
…less logs; proper returning statusCodes
|
Docker was a bit tough, but I got it up eventually. Added a few more countries. Changed default country to be an environment variable. Caching one of the queries to prevent re-renders. Removed redwood-hack directory and seems to work fine. Also I think I fixed the scraper. Try it out if you have docker: Scraper seems to take about 10min |
|
proof of scraper fix (I think): 🤔
|
| { | ||
| iso: 'cad', | ||
| worldometersSlug: 'canada', | ||
| name: 'Canada', |
| * | ||
| * @template T | ||
| * @param {Array<T>} array | ||
| * @param {(item: T) => void} callback |
There was a problem hiding this comment.
jsdoc adds types to javascript: https://medium.com/@antonkrinitsyn/jsdoc-generic-types-typescript-db213cf48640
| async function asyncForEach(array, callback) { | ||
| const promises = array.map((item) => callback(item)) | ||
|
|
||
| return Promise.allSettled(promises) |
There was a problem hiding this comment.
allSettled means that some or all could fail 🤷
| cors: { | ||
| origin: '*', | ||
| credentials: true, |
There was a problem hiding this comment.
cors is just to make docker work; if you want I could make this conditional
| htmlAttempt = $(`div#${chartId} + script`).html() | ||
| const hackedChartJSON = htmlAttempt.split('Highcharts.chart(')[1]; | ||
|
|
||
| const xAxisCategories = hackedChartJSON.match(/xAxis:.*?{.*?categories\:.*?(\[.*?\])/s) |
There was a problem hiding this comment.
here's where we just get xAxis categories directly
|
|
||
| const xVals = JSON.parse(xAxisCategories[1]) | ||
|
|
||
| const seriesData = hackedChartJSON.match(/series:.*?data:.*?(\[.*?\])/s) |
There was a problem hiding this comment.
and this gets series data directly
| // need to manually set year to 2020 | ||
| let day = new Date(val + ", 2020") | ||
| const day = new Date(val) |
There was a problem hiding this comment.
I assume that's not true anymore
| return { | ||
| statusCode: 200, | ||
| body: JSON.stringify('Success!') | ||
| } |
There was a problem hiding this comment.
new format for redwood function handlers
|
|
||
| services: | ||
| web: | ||
| build: . |
| web: | ||
| build: . | ||
| ports: | ||
| - 8910:8910 |
There was a problem hiding this comment.
binds host port 8910 to container port 8910
| build: . | ||
| ports: | ||
| - 8910:8910 | ||
| command: yarn rw dev web |
There was a problem hiding this comment.
main command to bring up web side
| environment: | ||
| REDWOOD_ENV_DEFAULT_COUNTRY: usa | ||
| REDWOOD_ENV_DEFAULT_COUNTRY_NAME: United States |
There was a problem hiding this comment.
dev could easily change these if they wanted
| restart: always | ||
| volumes: | ||
| # keeps database after container stops | ||
| - db:/app/api/prisma |
There was a problem hiding this comment.
this is a named volume which docker keeps internally
| @@ -1,5 +1,5 @@ | |||
| [build] | |||
| command = "cp redwood-hack/api-dbInstance.js node_modules/@redwoodjs/api/dist/dbInstance.js && yarn rw db up --no-db-client && yarn rw build" | |||
| command = "yarn rw prisma migrate dev && yarn rw build" | |||
| export const beforeQuery = () => { | ||
| return { | ||
| // never change query with new variables | ||
| variables, |
There was a problem hiding this comment.
the query (below) never uses variables, so this prevents it from fetching what it thinks is a new query
| variables, | ||
| fetchPolicy: 'cache-and-network', | ||
| // countries don't change mid-session | ||
| nextFetchPolicy: 'no-cache' |
There was a problem hiding this comment.
meant for this to be cache-only, but maybe doesn't matter
There was a problem hiding this comment.
cache-only breaks. Maybe should revert to cache-first (default)
I had issues bringing up the project and I realized everything was out-of-date. Ran
yarn rw update, and compared with a brand new project to fix up any issues.Also began fixing the scrape function. Appears to be getting the same data for each graph. Maybe a WIP: