Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter name tags #1115

Merged
merged 1 commit into from
Nov 26, 2024
Merged

Filter name tags #1115

merged 1 commit into from
Nov 26, 2024

Conversation

msbarry
Copy link
Contributor

@msbarry msbarry commented Nov 26, 2024

Fix #1114 by filtering name tags that don't correspond to different languages.

Copy link

sonarcloud bot commented Nov 26, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
D Maintainability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Copy link

This Branch 2a83c15 Base cbeba1b
0:01:10 DEB [archive] - Tile stats:
0:01:10 DEB [archive] - Biggest tiles (gzipped)
1. 14/4942/6092 (157k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:85k)
2. 9/154/190 (144k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k)
3. 10/308/380 (136k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k)
4. 10/308/381 (135k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k)
5. 14/4941/6092 (112k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:64k)
6. 14/4941/6093 (111k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k)
7. 14/4940/6092 (100k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k)
8. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k)
9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k)
10. 11/616/761 (95k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k)
0:01:10 DEB [archive] - Max tile sizes
                      z0    z1    z2    z3    z4    z5    z6    z7    z8    z9   z10   z11   z12   z13   z14   all
           boundary  151   336   409   544   872   332   437   552   802  1.6k    2k  6.9k  6.2k  5.6k  4.5k  6.9k
              water 7.7k  3.7k  8.6k  5.5k  2.6k  5.1k   15k   18k   16k   26k   15k   13k   17k   15k   12k   26k
              place    0     0   441   441   441   640   714    1k  1.6k  3.1k  5.7k  3.3k  1.7k   803   948  5.7k
            landuse    0     0     0     0   549   695  1.6k  6.7k   17k   44k   59k   50k   38k   19k   12k   59k
     transportation    0     0     0     0   311   774  1.2k    4k  5.6k   17k   13k   17k   62k   47k   33k   62k
           waterway    0     0     0     0   112   119     0     0     0    3k  2.3k    2k  2.1k  4.9k  2.4k  4.9k
               park    0     0     0     0     0     0  1.3k  4.3k  9.7k   18k   13k  8.2k  3.7k  3.4k  4.4k   18k
transportation_name    0     0     0     0     0     0   287   364  1.1k  1.9k  5.5k  4.7k  3.9k  3.4k   18k   18k
          landcover    0     0     0     0     0     0     0  9.6k   29k   85k   72k   81k   53k   30k   25k   85k
      mountain_peak    0     0     0     0     0     0     0  1.1k  1.8k  3.4k  4.3k  2.8k  1.4k  1.4k   869  4.3k
         water_name    0     0     0     0     0     0     0     0     0   486   461   433   452  1.2k  1.5k  1.5k
    aerodrome_label    0     0     0     0     0     0     0     0     0     0   666   328   273   221   221   666
            aeroway    0     0     0     0     0     0     0     0     0     0  1.6k  2.1k    3k  3.4k  2.8k  3.4k
                poi    0     0     0     0     0     0     0     0     0     0     0     0   568   565   85k   85k
           building    0     0     0     0     0     0     0     0     0     0     0     0     0   59k   92k   92k
        housenumber    0     0     0     0     0     0     0     0     0     0     0     0     0     0   35k   35k
          full tile 7.9k    4k  9.5k  6.4k  3.7k    6k   20k   40k   82k  195k  182k  135k  113k  128k  247k  247k
            gzipped 6.2k  3.5k  7.1k  5.2k  3.1k  4.8k   14k   28k   59k  144k  136k   99k   83k   92k  157k  157k
0:01:10 DEB [archive] -    Max tile: 247k (gzipped: 157k)
0:01:10 DEB [archive] -    Avg tile: 5.4k (gzipped: 4k) using weighted average based on OSM traffic
0:01:10 DEB [archive] -     # tiles: 4,115,031
0:01:10 DEB [archive] -  # features: 5,518,011
0:01:10 INF [archive] - Finished in 19s cpu:1m11s avg:3.7
0:01:10 INF [archive] -   read    1x(3% 0.6s wait:18s done:1s)
0:01:10 INF [archive] -   encode  4x(57% 11s wait:2s)
0:01:10 INF [archive] -   write   1x(22% 4s wait:13s)
0:01:10 INF [archive] - Finished in 1m10s cpu:3m37s gc:1s avg:3.1
0:01:10 INF [archive] - FINISHED!
0:01:10 INF [archive] - 
0:01:10 INF [archive] - ----------------------------------------
0:01:10 INF [archive] - data errors:
0:01:10 INF [archive] - 	render_snap_fix_input	16,675
0:01:10 INF [archive] - 	osm_multipolygon_missing_way	360
0:01:10 INF [archive] - 	osm_boundary_missing_way	66
0:01:10 INF [archive] - 	merge_snap_fix_input	12
0:01:10 INF [archive] - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:01:10 INF [archive] - 	render_snap_fix_input2	1
0:01:10 INF [archive] - 	omt_fix_water_before_ne_intersect	1
0:01:10 INF [archive] - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	1
0:01:10 INF [archive] - 	feature_point_on_surface_osm_invalid_multipolygon_empty_after_fix	1
0:01:10 INF [archive] - ----------------------------------------
0:01:10 INF [archive] - 	overall          1m10s cpu:3m37s gc:1s avg:3.1
0:01:10 INF [archive] - 	lake_centerlines 3s cpu:6s avg:2
0:01:10 INF [archive] - 	  read     1x(16% 0.5s done:2s)
0:01:10 INF [archive] - 	  process  4x(0% 0s done:2s)
0:01:10 INF [archive] - 	  write    1x(0% 0s done:2s)
0:01:10 INF [archive] - 	water_polygons   15s cpu:41s avg:2.8
0:01:10 INF [archive] - 	  read     1x(41% 6s done:7s)
0:01:10 INF [archive] - 	  process  4x(27% 4s wait:4s done:5s)
0:01:10 INF [archive] - 	  write    1x(4% 0.6s wait:9s done:5s)
0:01:10 INF [archive] - 	natural_earth    12s cpu:18s avg:1.6
0:01:10 INF [archive] - 	  read     1x(52% 6s done:5s)
0:01:10 INF [archive] - 	  process  4x(7% 0.8s wait:6s done:5s)
0:01:10 INF [archive] - 	  write    1x(0% 0s wait:6s done:5s)
0:01:10 INF [archive] - 	osm_pass1        2s cpu:6s avg:3.3
0:01:10 INF [archive] - 	  read     1x(2% 0s wait:2s)
0:01:10 INF [archive] - 	  parse    4x(33% 0.6s)
0:01:10 INF [archive] - 	  process  1x(71% 1s)
0:01:10 INF [archive] - 	osm_pass2        18s cpu:1m10s avg:3.9
0:01:10 INF [archive] - 	  read     1x(0% 0s wait:10s done:7s)
0:01:10 INF [archive] - 	  process  4x(78% 14s)
0:01:10 INF [archive] - 	  write    1x(2% 0.4s wait:17s)
0:01:10 INF [archive] - 	ne_lakes         0s cpu:0s avg:0
0:01:10 INF [archive] - 	boundaries       0s cpu:0s avg:1.4
0:01:10 INF [archive] - 	agg_stop         0s cpu:0s avg:0
0:01:10 INF [archive] - 	sort             1s cpu:3s avg:2.5
0:01:10 INF [archive] - 	  worker  1x(52% 0.7s)
0:01:10 INF [archive] - 	archive          19s cpu:1m11s avg:3.7
0:01:10 INF [archive] - 	  read    1x(3% 0.6s wait:18s done:1s)
0:01:10 INF [archive] - 	  encode  4x(57% 11s wait:2s)
0:01:10 INF [archive] - 	  write   1x(22% 4s wait:13s)
0:01:10 INF [archive] - ----------------------------------------
0:01:10 INF [archive] - 	archive	108MB
0:01:10 INF [archive] - 	features	284MB
-rw-r--r-- 1 runner docker 87M Nov 26 01:21 run.jar
0:01:04 DEB [archive] - Tile stats:
0:01:04 DEB [archive] - Biggest tiles (gzipped)
1. 14/4942/6092 (157k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:85k)
2. 9/154/190 (144k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k)
3. 10/308/380 (136k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k)
4. 10/308/381 (135k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k)
5. 14/4941/6092 (112k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:64k)
6. 14/4941/6093 (111k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k)
7. 14/4940/6092 (100k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k)
8. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k)
9. 14/4942/6091 (96k) https://onthegomap.github.io/planetiler-demo/#14.5/41.84501/-71.40015 (building:79k)
10. 11/616/761 (95k) https://onthegomap.github.io/planetiler-demo/#11.5/41.83679/-71.63086 (landcover:72k)
0:01:04 DEB [archive] - Max tile sizes
                      z0    z1    z2    z3    z4    z5    z6    z7    z8    z9   z10   z11   z12   z13   z14   all
           boundary  151   336   409   544   872   332   437   552   802  1.6k    2k  6.9k  6.2k  5.6k  4.5k  6.9k
              water 7.7k  3.7k  8.6k  5.5k  2.6k  5.1k   15k   18k   16k   26k   15k   13k   17k   15k   12k   26k
              place    0     0   441   441   441   640   714    1k  1.6k  3.1k  5.7k  3.3k  1.7k   803   948  5.7k
            landuse    0     0     0     0   549   695  1.6k  6.7k   17k   44k   59k   50k   38k   19k   12k   59k
     transportation    0     0     0     0   311   774  1.2k    4k  5.6k   17k   13k   17k   62k   47k   33k   62k
           waterway    0     0     0     0   112   119     0     0     0    3k  2.3k    2k  2.1k  4.9k  2.4k  4.9k
               park    0     0     0     0     0     0  1.3k  4.3k  9.7k   18k   13k  8.2k  3.7k  3.4k  4.4k   18k
transportation_name    0     0     0     0     0     0   287   364  1.1k  1.9k  5.5k  4.7k  3.9k  3.4k   18k   18k
          landcover    0     0     0     0     0     0     0  9.6k   29k   85k   72k   81k   53k   30k   25k   85k
      mountain_peak    0     0     0     0     0     0     0  1.1k  1.8k  3.4k  4.3k  2.8k  1.4k  1.4k   869  4.3k
         water_name    0     0     0     0     0     0     0     0     0   486   461   433   452  1.2k  1.5k  1.5k
    aerodrome_label    0     0     0     0     0     0     0     0     0     0   666   328   273   221   221   666
            aeroway    0     0     0     0     0     0     0     0     0     0  1.6k  2.1k    3k  3.4k  2.8k  3.4k
                poi    0     0     0     0     0     0     0     0     0     0     0     0   568   565   85k   85k
           building    0     0     0     0     0     0     0     0     0     0     0     0     0   59k   92k   92k
        housenumber    0     0     0     0     0     0     0     0     0     0     0     0     0     0   35k   35k
          full tile 7.9k    4k  9.5k  6.4k  3.7k    6k   20k   40k   82k  195k  182k  135k  113k  128k  247k  247k
            gzipped 6.2k  3.5k  7.1k  5.2k  3.1k  4.8k   14k   28k   59k  144k  136k   99k   83k   92k  157k  157k
0:01:04 DEB [archive] -    Max tile: 247k (gzipped: 157k)
0:01:04 DEB [archive] -    Avg tile: 5.4k (gzipped: 4k) using weighted average based on OSM traffic
0:01:04 DEB [archive] -     # tiles: 4,115,031
0:01:04 DEB [archive] -  # features: 5,518,011
0:01:04 INF [archive] - Finished in 20s cpu:1m12s avg:3.7
0:01:04 INF [archive] -   read    1x(3% 0.5s wait:18s done:1s)
0:01:04 INF [archive] -   encode  4x(56% 11s wait:2s done:1s)
0:01:04 INF [archive] -   write   1x(21% 4s wait:13s done:1s)
0:01:04 INF [archive] - Finished in 1m5s cpu:3m33s gc:1s avg:3.3
0:01:04 INF [archive] - FINISHED!
0:01:04 INF [archive] - 
0:01:04 INF [archive] - ----------------------------------------
0:01:04 INF [archive] - data errors:
0:01:04 INF [archive] - 	render_snap_fix_input	16,675
0:01:04 INF [archive] - 	osm_multipolygon_missing_way	360
0:01:04 INF [archive] - 	osm_boundary_missing_way	66
0:01:04 INF [archive] - 	merge_snap_fix_input	12
0:01:04 INF [archive] - 	feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix	2
0:01:04 INF [archive] - 	render_snap_fix_input2	1
0:01:04 INF [archive] - 	omt_fix_water_before_ne_intersect	1
0:01:04 INF [archive] - 	feature_polygon_osm_invalid_multipolygon_empty_after_fix	1
0:01:04 INF [archive] - 	feature_point_on_surface_osm_invalid_multipolygon_empty_after_fix	1
0:01:04 INF [archive] - ----------------------------------------
0:01:04 INF [archive] - 	overall          1m5s cpu:3m33s gc:1s avg:3.3
0:01:04 INF [archive] - 	lake_centerlines 2s cpu:5s avg:2.5
0:01:04 INF [archive] - 	  read     1x(22% 0.5s done:2s)
0:01:04 INF [archive] - 	  process  4x(0% 0s done:2s)
0:01:04 INF [archive] - 	  write    1x(0% 0s done:2s)
0:01:04 INF [archive] - 	water_polygons   15s cpu:40s avg:2.7
0:01:04 INF [archive] - 	  read     1x(40% 6s done:7s)
0:01:04 INF [archive] - 	  process  4x(27% 4s wait:4s done:5s)
0:01:04 INF [archive] - 	  write    1x(4% 0.5s wait:9s done:5s)
0:01:04 INF [archive] - 	natural_earth    6s cpu:12s avg:2
0:01:04 INF [archive] - 	  read     1x(95% 6s)
0:01:04 INF [archive] - 	  process  4x(13% 0.8s wait:6s)
0:01:04 INF [archive] - 	  write    1x(0% 0s wait:6s)
0:01:04 INF [archive] - 	osm_pass1        2s cpu:6s avg:3.1
0:01:04 INF [archive] - 	  read     1x(2% 0s wait:2s)
0:01:04 INF [archive] - 	  parse    4x(30% 0.6s)
0:01:04 INF [archive] - 	  process  1x(73% 1s)
0:01:04 INF [archive] - 	osm_pass2        18s cpu:1m13s avg:4
0:01:04 INF [archive] - 	  read     1x(0% 0s wait:11s done:7s)
0:01:04 INF [archive] - 	  process  4x(76% 14s)
0:01:04 INF [archive] - 	  write    1x(2% 0.4s wait:18s)
0:01:04 INF [archive] - 	ne_lakes         0s cpu:0s avg:0
0:01:04 INF [archive] - 	boundaries       0s cpu:0s avg:1.2
0:01:04 INF [archive] - 	agg_stop         0s cpu:0s avg:0
0:01:04 INF [archive] - 	sort             1s cpu:3s avg:2.6
0:01:04 INF [archive] - 	  worker  1x(51% 0.7s)
0:01:04 INF [archive] - 	archive          20s cpu:1m12s avg:3.7
0:01:04 INF [archive] - 	  read    1x(3% 0.5s wait:18s done:1s)
0:01:04 INF [archive] - 	  encode  4x(56% 11s wait:2s done:1s)
0:01:04 INF [archive] - 	  write   1x(21% 4s wait:13s done:1s)
0:01:04 INF [archive] - ----------------------------------------
0:01:04 INF [archive] - 	archive	108MB
0:01:04 INF [archive] - 	features	284MB
-rw-r--r-- 1 runner docker 87M Nov 26 01:22 run.jar

Full logs: https://github.com/onthegomap/planetiler/actions/runs/12021905765

@msbarry msbarry merged commit e049876 into main Nov 26, 2024
11 of 12 checks passed
@@ -10,7 +10,7 @@ public class LanguageUtils {
// See https://wiki.openstreetmap.org/wiki/Multilingual_names
public static final Predicate<String> VALID_NAME_TAGS =
Pattern
.compile("^name:[a-z]{2,3}(-[a-z]{4})?([-_](x-)?[a-z]{2,})?(-([a-z]{2}|[0-9]{3}))?$", Pattern.CASE_INSENSITIVE)
.compile("^name:[a-z]{2,3}(-[a-z]{4})?([-_](x-)?[a-z]{2,})?(-([a-z]{2}|\\d{3}))?$", Pattern.CASE_INSENSITIVE)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized this pattern is case-insensitive. This means it’ll pull in both name:tec=* and name:TEC=* – the latter is not a localized name. The usual case conventions for IETF language tags apply (language codes as xyz, script codes as Wxyz, country codes as XY). Some miscased keys have been common in the past, but as far as I know, they’re now deprecated (see also osmus/tileservice#21).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@1ec5 how does this change and test cases look? #1117

@@ -10,7 +10,7 @@ public class LanguageUtils {
// See https://wiki.openstreetmap.org/wiki/Multilingual_names
public static final Predicate<String> VALID_NAME_TAGS =
Pattern
.compile("^name:[a-z]{2,3}(-[a-z]{4})?([-_](x-)?[a-z]{2,})?(-([a-z]{2}|[0-9]{3}))?$", Pattern.CASE_INSENSITIVE)
.compile("^name:[a-z]{2,3}(-[a-z]{4})?([-_](x-)?[a-z]{2,})?(-([a-z]{2}|\\d{3}))?$", Pattern.CASE_INSENSITIVE)

This comment was marked as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OsmTranslationProvider should validate name subkeys
2 participants