Mods for new Maps #340

tim-moody · 2020-09-16T17:32:06Z

Backend

Add backend for iiab-install-map-region
Handle Base mbtiles for maps and satellite
Add backend for iiab-extend-sat.py
Add backend for Get OSM WiP for estimated space calcs
Add backend for wikidata overlays
Add backend for osm version
Need to run /usr/bin/iiab-maps-finish.py after downloads
Add backend to install base world map and base satellite mbtiles

Space Estimate

? do space calc when adding job so can just read it in add_wip and remove_wip

Front End Phase I

No changes to UI or space calcs, just use new catalog and new back end
Read cmdsrv config to get map version

Front End Phase II

Read cmdsrv config to get map version
Fork Admin Console page from maps if not already done
Enforce single selection for mbtile sets (why not allow multiple selections?) - NO
Disable selection for already installed
Add buttons where cli syntax is
Add ajax calls to backend for each function
Get WiP space stats
Add action for refresh mbtile sets catalog - NO This is deferred while catalog static
Add action to get base map and satellite mbtiles
Add new top level menu for maps - NO Keeping in current as only 2 menu items.
Add help sections

Source for backend:

/usr/bin/iiab-install-map-region does download of tiles and also satellite
/usr/bin/iiab-maps-finish.py calcs various data structures (initial coordinates and updated vector_map_idx
/usr/bin/iiab-extend-sat.py

Where are the constants for:

source_url=https://archive.org/download
SATELLITE=satellite_z0-z9_v3.mbtiles
working_dir=/library/working/maps

#1. https://github.com/iiab/maps/blob/master/osm-source/pages/viewer/scripts/iiab-install-map-region L4-10 and
#2,3. https://github.com/iiab/maps/blob/master/osm-source/pages/viewer/scripts/iiab-extend-sat.py L44-46

source_url=https://archive.org/download
SATELLITE=satellite_z0-z9_v3.mbtiles
working_dir=/library/working/maps
PLANET=osm-planet_z0-z10_2019.mbtiles
SAN_JOSE=osm_san_jose_z11-z14_2019.mbtiles
PREVIEW_SAT=satellite_z0-z6_v3.mbtiles
PREVIEW_OSM=planet_z0-z6_2019.mbtiles
work_dir = '/library/working/maps'
osm_dir = '/library/www/osm-vector-maps/maplist/assets'
sat_dir = '/library/www/osm-vector-maps/viewer/tiles'

Logic:

remove preview files in osm-vect_v2_install_step3.sh if exist
inst-map-tiles, allow override of download url to handle satellite, which is not in catalog

Code in #351 and #352

tim-moody · 2020-09-17T13:45:42Z

Source for backend:

/usr/bin/iiab-install-map-region does download of tiles and also satellite
/usr/bin/iiab-maps-finish.py calcs various data structures (initial coordinates and updated vector_map_idx
/usr/bin/iiab-extend-sat.py

Where are the constants for:

source_url=https://archive.org/download
SATELLITE=satellite_z0-z9_v3.mbtiles
working_dir=/library/working/maps

georgejhunt · 2020-09-17T15:39:39Z

#1. https://github.com/iiab/maps/blob/master/osm-source/pages/viewer/scripts/iiab-install-map-region L4-10 and
#2,3. https://github.com/iiab/maps/blob/master/osm-source/pages/viewer/scripts/iiab-extend-sat.py L44-46

tim-moody · 2020-09-17T16:54:03Z

@georgejhunt I notice that iiab-install-map-region computes the source url rather than reading from catalog. Looks like I should get the url from detail_url

What will happen to the catalog if there is an osm_central_america_z11-z14_202X.mbtiles? will osm_central_america_z11-z14_2019.mbtiles remain or be replaced?

tim-moody · 2020-09-17T18:34:04Z

@georgejhunt can I take the existence of /library/www/osm-vector-maps/installer/ as an indication that we have the new version of maps. I want to leave backwards compatibility, at least for now.

georgejhunt · 2020-09-17T18:36:59Z

My thinking about the catalog is somewhat unformed. I was thinking that the model of the kiwix catalog was what I should shoot for. In that case, I think the 2019 map would fall off the catalog, and be replaced with the 2020 version (with the same perma-ref). The color would indicate "installed" with the hover indicating the new map available. Installing the 2020 version would cause the deletion of the 2019 version in /osm-vector-map/tile/. That is not implemented in my code. I'm sure you have thought about this whole issue much more than I. Quote: can I take the existence of /library/www/osm-vector-maps/installer/ as an indication that we have the new version of maps. I want to leave backwards compatibility, at least for now. yes

…

On Thu, Sep 17, 2020 at 9:54 AM Tim Moody ***@***.***> wrote: @georgejhunt <https://github.com/georgejhunt> I notice that iiab-install-map-region computes the source url rather than reading from catalog. Looks like I should get the url from detail_url What will happen to the catalog if there is an osm_central_america_z11-z14_202X.mbtiles? will osm_central_america_z11-z14_2019.mbtiles remain or be replaced? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#340 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOTQHESVQDL735MQCFD5M3SGI5LXANCNFSM4RPFLQBA> .

tim-moody · 2020-09-17T18:41:06Z

The current code uses url not detailed_url. Do they differ to provide backwards compatibility?

georgejhunt · 2020-09-17T18:46:40Z

That was my intention. Url and dedailed_urf have such different meanings, I thought it would be confusing to overload "url" with zip package AND region_mbtile, as if they referred to the same thing.

…

On Thu, Sep 17, 2020 at 11:41 AM Tim Moody ***@***.***> wrote: The current code uses url not detailed_url. Do they differ to provide backwards compatibility? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

tim-moody · 2020-09-25T17:33:29Z

@georgejhunt I'm getting confused again. What is the relationship between map-catalog.json, regions.json, and vector-map-idx.json? Is only map-catalog.json used in the current code?

/etc/iiab/map-catalog.json:

{
  "maps": {
    "osm_africa_z11-z14_2019.mbtiles": {
      "bittorrent_url": "https://archive.org/download/osm_africa_z11-z14_2019.mbtiles/osm_africa_z11-z14_2019.mbtiles_archive.torrent",
      "center_lat": "2.0",
      "center_lon": "20.0",
      "date": "2019-10-08",
      "detail_url": "https://archive.org/download/osm_africa_z11-z14_2019.mbtiles/osm_africa_z11-z14_2019.mbtiles",

/etc/iiab/regions.json:

{
  "regions": {
    "africa": {
      "center_lat": "2.0",
      "center_lon": "20.0",
      "date": "2017-07-03",
      "east": "63.56",
      "north": "46.00",
      "osm_size": "10099986432",
      "perma_ref": "en-osm-omt_africa",
      "sat_is_regional": "False",
      "sat_size": "976416768",
      "sat_url": "https://archive.org/download/satellite_z0-z9_v3.mbtiles/satellite_z0-z9_v3.mbtiles",
      "seq": 50,
      "size": "11076403200",
      "south": "-84.0",
      "title": "Africa",
      "url": "https://archive.org/download/en-osm-omt_africa_2017-07-03_v0.23/en-osm-omt_africa_2017-07-03_v0.23.zip",

/library/www/html/common/assets/vector-map-idx.json:

{
  "en-osm-omt_south_asia": {
    "file_name": "osm_south_asia_z11-z14_2019.mbtiles",
    "menu_item": "en-osm-omt_south_asia",
    "size": 14300418048,
    "date": "2019-10-08",
    "region": "south_asia",
    "language": "en"
  },
  "en-osm-omt_san_jose": {
    "file_name": "osm_san_jose_z11-z14_2019.mbtiles",
    "menu_item": "en-osm-omt_san_jose",
    "size": 2867826688,
    "date": "2019-10-08",
    "region": "san_jose",
    "language": "en"
  }
}

georgejhunt · 2020-09-25T18:39:31Z

Only map_catalog.json is used currently. Regions.json (in my catalog generation) is the source of map_catalog (I was trying to make transition seamless). vector-map-idx.json is the copy of the catalog which is written during the install process that should capture the info that might go away when the catalog is changed.

…

On Fri, Sep 25, 2020 at 10:33 AM Tim Moody ***@***.***> wrote: @georgejhunt <https://github.com/georgejhunt> I'm getting confused again. What is the relationship between map-catalog.json, regions.json, and vector-map-idx.json? Is only map-catalog.json used in the current code? /etc/iiab/map-catalog.json: { "maps": { "osm_africa_z11-z14_2019.mbtiles": { "bittorrent_url": "https://archive.org/download/osm_africa_z11-z14_2019.mbtiles/osm_africa_z11-z14_2019.mbtiles_archive.torrent", "center_lat": "2.0", "center_lon": "20.0", "date": "2019-10-08", "detail_url": "https://archive.org/download/osm_africa_z11-z14_2019.mbtiles/osm_africa_z11-z14_2019.mbtiles", /etc/iiab/regions.json: { "regions": { "africa": { "center_lat": "2.0", "center_lon": "20.0", "date": "2017-07-03", "east": "63.56", "north": "46.00", "osm_size": "10099986432", "perma_ref": "en-osm-omt_africa", "sat_is_regional": "False", "sat_size": "976416768", "sat_url": "https://archive.org/download/satellite_z0-z9_v3.mbtiles/satellite_z0-z9_v3.mbtiles", "seq": 50, "size": "11076403200", "south": "-84.0", "title": "Africa", "url": "https://archive.org/download/en-osm-omt_africa_2017-07-03_v0.23/en-osm-omt_africa_2017-07-03_v0.23.zip", /library/www/html/common/assets/vector-map-idx.json: { "en-osm-omt_south_asia": { "file_name": "osm_south_asia_z11-z14_2019.mbtiles", "menu_item": "en-osm-omt_south_asia", "size": 14300418048, "date": "2019-10-08", "region": "south_asia", "language": "en" }, "en-osm-omt_san_jose": { "file_name": "osm_san_jose_z11-z14_2019.mbtiles", "menu_item": "en-osm-omt_san_jose", "size": 2867826688, "date": "2019-10-08", "region": "san_jose", "language": "en" } } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#340 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOTQHH4PSXBVWXRWVOXX7LSHTH7PANCNFSM4RPFLQBA> .

tim-moody · 2020-09-25T21:00:24Z

@georgejhunt I got Failed to open /library/www/osm-vector-maps/viewer/tiles/satellite_z0-z9_v3.mbtiles -- Quitting

I have in /library/www/osm-vector-maps/viewer/tiles/
-rw-r--r-- 1 root root 35438592 Aug 22 18:42 planet_z0-z6_2019.mbtiles
-rw-r--r-- 1 root root 1810432 Aug 22 18:39 satellite_z0-z4_v3.mbtiles

I assume this is because the install was short circuited to save time.
What is the operation to get a full set of world and satellite tiles?

iiab-install-map-region with no arguments?

georgejhunt · 2020-09-25T23:14:44Z

I think I'd try "./runroles osm-vector-maps"

…

On Fri, Sep 25, 2020 at 2:00 PM Tim Moody ***@***.***> wrote: @georgejhunt I got Failed to open /library/www/osm-vector-maps/viewer/tiles/satellite_z0-z9_v3.mbtiles -- Quitting I have in /library/www/osm-vector-maps/viewer/tiles/ -rw-r--r-- 1 root root 35438592 Aug 22 18:42 planet_z0-z6_2019.mbtiles -rw-r--r-- 1 root root 1810432 Aug 22 18:39 satellite_z0-z4_v3.mbtiles I assume this is because the install was short circuited to save time. What is the operation to get a full set of world and satellite tiles? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

tim-moody · 2020-09-26T13:22:27Z

I thinks this was changed in iiab/iiab@809de60#diff-a3c03d404704b5bd901f6e04df80ef64

Now I think we need a button or something to do the initial download.

tim-moody · 2020-09-26T14:58:24Z

@georgejhunt can you help me get the output of iiab-extend-sat.py to stdout so it gets into the job status?

tim-moody · 2020-09-26T15:06:23Z

could just be buffering as the output appeared when it finished.

"job_output": "tileX:6303 tileY:3699 zoom:13 alredy downloaded: 0 added:690\ntileX:6303 tileY:3709 zoom:13 alredy downloaded: 0 added:700\ntileX:6304 tileY:3696 zoom:13 alredy downloaded: 0 added:710\ntileX:6304 tileY:3706 zoom:13 alredy downloaded: 0 added:720\ntileX:6305 tileY:3693 zoom:13 alredy downloaded: 0 added:730\ntileX:6305 tileY:3703 zoom:13 alredy downloaded: 0 added:740\ntileX:6306 tileY:3690 zoom:13 alredy downloaded: 0 added:750\ntileX:6306 tileY:3700 zoom:13 alredy downloaded: 0 added:760\ntileX:6306 tileY:3710 zoom:13 alredy downloaded: 0 added:770\nTotal time: 0 hrs: 3 min: 0 sec Duplicates:0 Total_tiles Added:773\n"

tim-moody assigned georgejhunt Sep 16, 2020

holta mentioned this issue Sep 16, 2020

New OSM Vector Maps need clean UX + clean instructions ("pick a continent or region") iiab/iiab#1605

Open

13 tasks

holta mentioned this issue Sep 17, 2020

Structural merges at the end of a release cycle are Not Ok iiab/iiab#2531

Merged

holta mentioned this issue Sep 26, 2020

IIAB Maps installer page explanations overhauled iiab/maps#29

Merged

This comment has been minimized.

Sign in to view

holta mentioned this issue Sep 28, 2020

WIP: What's Changed in Maps for the IIAB 7.2 Release? [OSM] iiab/iiab#2551

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mods for new Maps #340

Mods for new Maps #340

tim-moody commented Sep 16, 2020 •

edited

Loading

tim-moody commented Sep 17, 2020 •

edited

Loading

georgejhunt commented Sep 17, 2020

tim-moody commented Sep 17, 2020

tim-moody commented Sep 17, 2020

georgejhunt commented Sep 17, 2020 via email

tim-moody commented Sep 17, 2020 •

edited

Loading

georgejhunt commented Sep 17, 2020 via email

tim-moody commented Sep 25, 2020

georgejhunt commented Sep 25, 2020 via email

tim-moody commented Sep 25, 2020 •

edited

Loading

georgejhunt commented Sep 25, 2020 via email

tim-moody commented Sep 26, 2020

tim-moody commented Sep 26, 2020

tim-moody commented Sep 26, 2020

This comment has been minimized.

Mods for new Maps #340

Mods for new Maps #340

Comments

tim-moody commented Sep 16, 2020 • edited Loading

Backend

Space Estimate

Front End Phase I

Front End Phase II

tim-moody commented Sep 17, 2020 • edited Loading

georgejhunt commented Sep 17, 2020

tim-moody commented Sep 17, 2020

tim-moody commented Sep 17, 2020

georgejhunt commented Sep 17, 2020 via email

tim-moody commented Sep 17, 2020 • edited Loading

georgejhunt commented Sep 17, 2020 via email

tim-moody commented Sep 25, 2020

georgejhunt commented Sep 25, 2020 via email

tim-moody commented Sep 25, 2020 • edited Loading

georgejhunt commented Sep 25, 2020 via email

tim-moody commented Sep 26, 2020

tim-moody commented Sep 26, 2020

tim-moody commented Sep 26, 2020

This comment has been minimized.

tim-moody commented Sep 16, 2020 •

edited

Loading

tim-moody commented Sep 17, 2020 •

edited

Loading

tim-moody commented Sep 17, 2020 •

edited

Loading

tim-moody commented Sep 25, 2020 •

edited

Loading