Use PortalJS to create a Data Portal for data on Github in 5 minutes

[anuveyatsu]

Purpose and Principles

A demo and tutorial creating a portal for csv datasets on github to demonstrate what PortalJS can do, especially for simple data:

• Don't require users to create their own data (to do the tutorial) i.e. use github.com/datasets
• Don't require Frictionless?
• Could even be one file but doesn't make a lot of sense
• Focus on Catalog
• Showcase is secondary ... (or not even part of this initial tutorial - cf the fivethirtyeight catalog)
• Instructions for how to use with your own files? (is that even needed)

Future

• data elsewhere than github e.g. s3, r2 etc

Outcome visioning

Can clone a template repo, set some variables to say which datasets which you want to show, deploy to e.g. Vercel and you have a running portal ...

... and then read a tutorial with instructions on how you can then customize it and do more ...

Brainstorm and Prioritization

Questions

• Is it just CSV file or CSV plus a README per "dataset"? ✅2023-04-18 i would allow just file but support both
• Do we include the showcase? ✅2023-04-18 no, not for now
• Is site statically generated or dynamically? ✅2023-04-17 prefer static
• What is the implementation structure? ✅2023-04-18 just a single index.jsx with getStaticProps to load data files specificied in a config.json (or whatever config file we have). See below for outline
• Tutorial structure
  • Introduction / motivation:
  • The result: screenshot of the result
  • We are going to use these 5 datasets on github.com/datasets
    • LATER: you can prepare you own

Next steps

Notes

• What is a Portal? ✅2023-04-17 A site with landing page, search (catalog), dataset page + resource page (sometimes these two are consolidated).
• What are Portal's components? ✅ Search: search input, facets, single search result, list of search results, pagination. This also largely depends on what's your metadata schema, search backend (e.g., for facets and pagination) etc. For git-backed portal we may not need to have all these but simple search (?)

Spec for `config.json

{
  datasets: [
    { 
      file: // is it a path 
      readme: 
    },
   {
   },
}

[rufuspollock]

Maybe we could start a simple spike solution e.g.

• a nextjs app in examples
• simple index page - inspiration from e.g. https://electionlab.mit.edu/data or https://data.fivethirtyeight.com/
• borrow a bit from datahub-next for data loading if needed (@demenech would know the code)
• Later we could refactor components out into the common library
• And later still split this out into a repo template

[luccasmmg]

I have one question, is the data-portal going to theoretically index everything inside the user/org (ex: /datasets) or is it going to be only one repo? If it's only one repo is it going to be a frictionless one? If so, is search functionality needed if all the datasets can fit into a single datapackage.json?

[rufuspollock]

@luccasmmg

is the data-portal going to theoretically index everything inside the user/org (ex: /datasets) or is it going to be only one repo?

it will be given an explicit list of data files to list. So it won't do any searching. Aim for demo is just 3-10 datasets/data files.

If it's only one repo is it going to be a frictionless one?

It doesn't need to be frictionless.

[demenech]

I have one question, is the data-portal going to theoretically index everything inside the user/org (ex: /datasets) or is it going to be only one repo?

Also curious about this one. Is the user going to provide a list of repos e.g @datasets/finance-vix, @datasets/oil-prices? Or is it a list of data files inside a repo e.g in @datasets/oil-prices -> brent-daily.csv, brent-weekly.csv?

Edit: in other words, what is a dataset, and what is a data file?

[rufuspollock]

Actual list of files IMO - much more useful as you can then have multiple files/datasets per repo - cf https://data.fivethirtyeight.com (reproducing that would be kind of cool btw)

[luccasmmg]

borrow a bit from datahub-next for data loading if needed (@demenech would know the code)

by this i assume you are meaning the dataset page in datahub-next that fetches data from github and then creates a page, right? The code around this page

[anuveyatsu]

I believe so but @demenech can confirm here.

[demenech]

Yup. This one, https://github.com/datopian/datahub-next/blob/with-SSR/pages/%40org/%5Borg%5D/%5Bproject%5D/index.tsx and https://github.com/datopian/datahub-next/blob/with-SSR/lib/project.ts.

[rufuspollock]

Note: we should nail an acceptance or sketch before we begin. My simplest version would be to reproduce https://data.fivethirtyeight.com (maybe only first 3-5 datasets).

[luccasmmg]

@rufuspollock just so im clear, is this more or less what you wanted(ignore the design for now)? https://portaljs-jet.vercel.app/

Basically, imagining im a dude who wants to have a data portal, i would have some repos somewhere, i would basically
1 - I would edit some sort of config file, like this one
2 - On build time the portal grabs the info from all of those repos and builds two things
- An index page with a list(No search for now)
- An individual page for every dataset, like this

Please take a look into that deployment i've shared, and in the code inside of here which tbh is pretty standard stuff, just copying things from datahub-next and modifying to fit the needs

Obs: About styling, i didnt do like you said because i started working on this yesterday and i was not sure on what was wanted, so i just grabed the first tailwindui thing that i saw to make it easy, if you want something more like a list/table is the first table here ok? In any case, what sort of information do you want displayed? I just used the:

• owner
• repo
• excerpt
• a link for the github
  because thats more or less what i had, i can probably get the Last Updated by using Octokit and probably some other git informations(Number of stars, forks etc) If the repo contains a data-package i also have access to the informations there.

Please just confirm if this is more or less the direction that you want to follow and what to change

[luccasmmg]

@anuveyatsu

• I thought about just having the list of urls, but then i was thinking, we could probably add some extra configurations in the future, for example, the title of the portal and so on and so forth, but if you want its really easy to change between these two options
• Those make sense, here is an example of how it would look like as a table
  

Obs: Will work on fetching stuff on github once we have a design and Rufus gives his input

[luccasmmg]

@anuveyatsu just updated the example to be more like a table and using information straight out of the github api(last updated + repo description)

[rufuspollock]

This is great 👏👏 🎉

Next steps:

• Create issue here on portaljs from this discussion and make up the acceptance from what you are already doing 😄
• Get this in examples and have auto-deploy and a README that has screenshot, link to demo, how to use yourself (npx create-next-app path, plus some small info about how it works (e.g. how to configure what is used)
• Give demo url a nicer url perhaps e.g. ex-{exname}.portaljs.org
• 🏆 write a super simple blog post about this for portaljs blog! (KISS: motivation, screenshot, link to example, demo link etc)

Brainstorm a list of improvements / alternatives e.g. (add to issue as ideas)

• Display e.g. could layout like https://electionlab.mit.edu/data (i.e. cars vs table)
• Handling multiple data files in a repo e.g. fivethirtyeight examples
• Non frictionless data e.g. just a data file, or just a data file and readme

[luccasmmg]

Is this blog post good enough? https://hackmd.io/@6v5rryFoQBqA6oMVm1pNKw/ryIM0Bpzn im not too sure where should i put it, is it somewhere in here? should we create a blog folder?

[rufuspollock]

@luccasmmg could you mention the issue here and let's move the key tasks (you checked) into acceptance (and the brainstorm into the relevant part of the isssue)

Also move your follow up question to the issue and i can answer there. Generally i suggest that you have @demenech review your draft (even better than hackmd would be to open a PR as prefer commenting on a PR than a hackmd).

Finally, let's close this discussion and point at the issue ...

[luccasmmg]

All remaining conversation should be directed at #768