Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Universal dataset catalog #1642

Merged
merged 8 commits into from
Dec 10, 2024
Merged

Universal dataset catalog #1642

merged 8 commits into from
Dec 10, 2024

Conversation

bhousel
Copy link
Contributor

@bhousel bhousel commented Dec 9, 2024

For a while, we've had the catalog of Esri datasets, but the Microsoft and Meta datasets were separate from this.

As we start offering more sources of data (Microsoft, Meta, Overture, and more?) it would be great to get all the datasets into a single catalog that our users can search through. That's what this PR does!

  • Introduce RapidDataset class to contain all the common props that a dataset needs
  • Show RapidDatasets in the menus and catalog
  • Add a lot of methods for working with datasets into RapidSystem
  • RapidSystem now prepares the catalog at startup - it will getAvailableDatasets from the 3 data-providing services (mapwithai,esri,overture)
  • This moves a bunch of responsibility around - some things that were in the UI code are now in RapidSystem - some stuff that was duplicated isn't anymore.
Screenshot 2024-12-09 at 11 58 16 AM

Still todo:

  • Make sure all the dataset have snippet, itemUrl, thumbnail (only the Esri ones still do)
  • Make sure as datasets get added, they default to a new color (currently they're all magenta)
  • Adjust the text (it's still written for only Esri datasets)

We've had the catalog of Esri datasets, but the Microsoft and Meta datasets
were separate from this.

As we start offering more sources of data (Microsoft, Meta, Overture, and more?)
it would be great to get all the datasets into a single catalog that our users
can search through.  That's what this commit does!

- Introduce `RapidDataset` class to contain all the common props that a dataset needs
- Show RapidDatasets in the menus and catalog
- Add a lot of methods for working with datasets into RapidSystem
- RapidSystem now prepares the catalog at startup - it will getAvailableDatasets
  from the 3 data-providing services (`mapwithai`,`esri`,`overture`)
- This moves a bunch of responsibility around - some things that were in the
  UI code are now in RapidSystem - some stuff that was duplicated isn't anymore.

Still todo:
- make sure all the dataset have snippet, itemUrl, thumbnail (only the Esri ones still do)
- make sure as datasets get added, they default to a new color (currently they're all magenta)
I'm trying to match the legacy color-choosing behavior from before Rapid#1642 (which changed a bunch of things)
This code is a bit weird - I don't like it and we should change it...
- If adding fbRoads/msBuildings, choose "Rapid magenta".
- If adding an Overture dataset, choose "Overture cyan".
- If adding an Esri dataset, choose a color based on how many datasets were added already.
This gives the added datasets some background grey color and
a message like "dataset added", next to the "remove" button
Before they would stay in their spot.
Now, adding moves them to the top, removing puts them back in their place.
Most of the datasets are still from ArcGIS, but some will be from Meta,
Microsoft, Overture, etc too.
@bhousel
Copy link
Contributor Author

bhousel commented Dec 10, 2024

This is done! People who have used the dataset picker will still find it familiar, but these tweaks will help us add more datasets to Rapid from providers like Meta and Overture without hardcoding them onto the Rapid menu like we did before.

Screenshot 2024-12-10 at 11 34 39 AM Screenshot 2024-12-10 at 11 34 52 AM

supports: #1005 #1207 #1469 #1501 #1595 #1608

@bhousel bhousel merged commit 2344772 into main Dec 10, 2024
5 checks passed
@bhousel bhousel deleted the dataset_catalog branch December 10, 2024 16:40
bhousel added a commit that referenced this pull request Dec 23, 2024
…sync

(closes #1653)

This regressed in #1642
There is an extra step that we need to run for the Esri datasets where we fetch
the dataset's schema from a different url. This previously happened in the UI code,
but now the responsibility for doing that step needs to move to the EsriService code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants