Catalog Item schema

[butchtm]

This is to document the format of the catalog item.

Fields:

• id - generated? (initial pass: use filename (without extension) on folder -- guaranteed to be unique per folder if only using .yml as extension and lower-case. )

• name (required)

• description (required)

• card type (required) - values: 'model', 'dataset'

• links (one or more links, at least 1 required)

  • each link can/must have:
    • url (required)
    • description (required)
    • type(required)
      • proposed values: 'model-weights', 'dataset-csv', 'dataset-geojson', 'documentation', 'repository', 'code-python', 'code-rlang', 'code-notebook' (note: can be expanded in the future)

• organization - (required) (same as contributor?) - contains the following:

  • name (required)
  • url (optional)
  • description (optional)

• date-added (required)

• date-modified (optional)

• tags (optional but highly recommended) - one or more tags where each tag is a word (no spaces, lower-case, no underscores, dash-allowed)

• country/region (optional) - one or more countries/region, where each country/region is validated against a list

• year/period (optional)

• detail-image-url (generated) NEW

• used-by (optional) - list of items

  • each item has:
    • name (required)
    • url (optional)

• evaluation metrics (optional) - one or more evaluation metric or a link

  • each evaluation metric can have:
    * metric type (required)
    * value (required)
  • a link:
    • description (required)
    • url (required)

• data columns (optional)

  • each column can/must have:
    • name (required)
    • type(optional) - string/number/date/etc
    • hxltag (optional)

• sample data (optional) rows:

  • each row can/must have a list of values (each list must match list of columns)

Related items:

• Catalog card page
• Figma Prototype

[butchtm]

Sample yaml file:

name: 'Poverty Mapping Model for Timor Leste'
description: 'This is a sklearn random forest regressor model  for predicting relative wealth for Timor Leste.  (add more details here later...)'
card-type: model
organization: 
  name: "Thinking Machines Data Sciences Inc"
  url: 'https://thinkingmachin.es'
date-added: '2023-03-02'
tags: 
  - poverty-mapping
  - timor-leste
country-region: timor-leste
# alternate format for multi country/region
country-region: 
  - timor-leste
  - philippines
  - malaysia
year-period: 2016
evaluation-metrics:
  - metric:
      metric-type: rsquared
      value: 0.59
 - link:
     description: 'Additional evaluation metrics discussion'
     url:  https://thinkingmachines.github.io/unicef-ai4d-poverty-mapping/notebooks/2023-01-17-initial-model-ph-mm-tl-kh/model_tl.html#evaluate-model-training-using-cross-validation
links:
  - url: https://github.com/thinkingmachines/unicef-ai4d-poverty-mapping/blob/main/notebooks/2023-02-21-single-country-rollouts/model_tl_rollout.pkl
    description: 'Timor Leste Poverty Mapping Model Weights (.pkl). See related notebooks for usage'
    type: model-weights
  - url: https://github.com/thinkingmachines/unicef-ai4d-poverty-mapping/blob/main/notebooks/2023-02-21-single-country-rollouts/rollout_output_tl.geojson
    description: 'Country wide rollout (inference) of the Poverty Mapping Model on Timor Leste using 2.4km grids (bingtile quadkey level 14)'
    type: 'dataset-geojson'
  - url: https://github.com/thinkingmachines/unicef-ai4d-poverty-mapping/blob/main/notebooks/2023-02-21-single-country-rollouts/2023-03-02-tl_final_model_rollout.ipynb
    description: 'Timor Leste Model Rollout Part 3 (Feature Engineering and Model Prediction on Rollout Grids)'
    type: 'code-notebook'
  - url: https://github.com/thinkingmachines/unicef-ai4d-poverty-mapping/blob/main/notebooks/2023-02-21-single-country-rollouts/2023_02-21-tl_final_model_training.ipynb
    description: 'Timor Leste Model Rollout Part 1 (Training Final Single-country Model)'
    type: 'code-notebook'
  - url: https://github.com/thinkingmachines/unicef-ai4d-poverty-mapping/blob/main/notebooks/2023-02-21-single-country-rollouts/2023-03-01-tl_generate_populated_grids.ipynb
    description: 'Timor Leste Model Rollout Part 2 (Generate Roll-out Grids)'
    type: 'code-notebook'
 

[ghost]

Hi Butch! these lgtm! Let's wait for Binky's feedback.

[binkymilk]

Looks good to me! Question on country-region though - are we expecting values here that are not countries?

[binkymilk]

Ohh, is it better to rename to location instead? Also, is this going to be an array of countries or just one region like SEA? Just thinking how filtering works in this case

[butchtm]

location to me refers to a place - like a point on a map. regions are more neutral in that they can cover supra-national, national and sub-national boundaries. I only added country to qualify it for the most common use cases.

And no - it won't be an array of countries, just a single item -- are we thinking of doing a drop down here? It might be possible to just extract the list of countries/regions from the catalog items themselves, so if the data provider thinks up of an arbitrary subdivision, we'll just add that to the list?

I'd like to get @AnthonyMockler 's feedback though. Its early enough in the development stage to change it without impacting any timelines.

[AnthonyMockler]

Country-region is one where we don't want responses to be too freeform - Otherwise we'll end up with a proliferation of country-region tags (e.g. we put laos as the country-region for one dataset, somebody else puts lao-pdr, now someone looking for the Laos datasets doesn't know what to filter by)
I think a list of iso3s would work for most things, but what if the dataset only covers a city / province? Maybe it still works.

[butchtm]

hi @AnthonyMockler

My suggestion is that we create a list (initially populated from iso3166) -- I've seen many github repos that contain this already so we don't even have to type it in.

We then use this list to validate the country/region names that specified in the catalog items, rejecting/flagging those catalog items which are not included in the list when they (the dataset/model providers) create a PR (which triggers a github action to validate the changes) to include their datasets/models into the catalog.

BTW, we are already thinking of including validation for other stuff as well - eg. making sure required entries are complete, the url links are valid, etc. so that even before the maintainers review the PR, some checks have already been done to ensure only those datasets/models that have met some quality criteria get reviewed before being allowed to be merged into the research bank catalog.

They (the data providers) can also make a PR to add items to the country/region list -- but in the end, the research bank maintainers (initially TM, later on EAPRO) can decide whether to approve the PR modifying the list, expanding the list when it makes sense to do so.

WDYT?

[AnthonyMockler]

@butchtm Yep sounds smart - Many of the datasets that data analysts / dataviz peeps use are already tagged with the alpha3 (e.g. PHL)

[binkymilk]

Hi @butchtm, just want to clarify something. If the data provider has indicated a training dataset, can they also include the data-columns and sample-data for it? How about if there are multiple results (if this is possible)?

Just curious because right now, there's only one data-columns and sample-data field which I assume is for the resulting dataset. Should we make these fields an array instead that specifies:

1. its id (the id of the metadata that is linked to a specific dataset, whether training or a results dataset)
2. its schema or sample data

data-columns: [
{ id: <id/name of the dataset?>, columns: [{ type: '', name: '' }]},
{ id: <id/name of the dataset?>, columns: [{ type: '', name: '' }]},
...
]

sample-data: [
{ id: <id/name of the dataset?>, sample: [x, x, x, x]},
{ id: <id/name of the dataset?>, sample: [x, x, x, x]},
]

Something like this or any schema that basically links the columns and sample data to its dataset.

[butchtm]

My initial idea was to pick one csv/geojson per catalog item.

But since there can more than 1 link per catalog item pointing to a csv/geojson then each link, yes I can grab the column headers from those as well.

One other reason for only using 1 csv/geojson link per catalog item was the possible slow down of the processing routines going through lots of links and fetching the first few rows of each.

Maybe we can ask @AnthonyMockler what he thinks about this.

[binkymilk]

Points for clarification:

1. Should we include data structure/schema and data preview for training datasets?
2. Is it possible to have 2 different datasets for a model (the contents, not the file format)?
3. If yes, do we want to show schema and preview for the other links?

Notes:

• Processing routines will be slower since it will be going through lots of links and fetching the first few rows of each.

[binkymilk]

1. Just the results
2. Possible to have different datasets - but we can just choose the first one for schema and preview

[butchtm]

Update 04/13/2023

• Added the hxltag field in the data-columns entry in order to show any hxl tags included in the dataset.

[butchtm]

Hi @binkymilk I'm adding this new field in the generated catalog.json item:

• detail-image-url (generated/optional) - use generated image from the first geojson dataset found or point to a pre-generated image in the public/assets/regions folder using the country/region field if no geojson dataset is found or if the geojson file size exceeds a size limit of 150MB. If a geojson file meets the criteria, a image is generated in the public/assets/items folder using the catalog item id as the file name (w/ a .png extension).

So this means that catalog items will have an additional optional property called detail-image-url that has a relative url ( e.g. assets/items/povmap-philippines.png or assets/regions/thailand.png - depending on whether its using an image using a geojson dataset or using the region-country image as a fallback.

[butchtm]

#135 Supporting multi-country regions.
The catalog item schema has now been loosened to allow multiple values for the country-region field (i.e. the model/dataset can cover multiple countries.
In order to support this, the following updates need to be made:

• Update the validate_catalog.py script to allow for a list of strings (can still be a string) -- its still optional. If its a list, each item in the list is still validated against the list in validation/regions.txt.
• Update the transform_catalog.py script so that it now generates a multi-country image for the combination of countries/regions covered if it doesn't exist yet, otherwise it will just use the existing multi-country image if it has already been generated. These multi-country images will be located in public/assets/multi. It will then update the detail-image-url of the catalog item to use this multi-country image url.
• Update the catalogue item schema CatalogueItem.type.ts so that country-region can hold either a string or a list of strings
• Update the components CatalogueItemCard.tsx and any other component displaying country-region to (possibly) display a list of countries instead of just 1 country -- this will probably be a comma-separated list.
• Update the search functionality so that multiple entries for country-region can be generated for each catalog item so that all countries for a multi-country dataset/model return the item when searched for each country in the list.