Add index for all wof languages (when differents to default language)

[Joxit]

What is this for ?

This importer did not index preferred names of all languages.
We need this for the multi-lang search in autocomplete.

I remove all elements when they are present in the default index. That means, if we say New York in the default language and New York in FRA, I will drop the FRA entry. This should prevent a massive increase of ES indexes.

For now, only name:{lang}_x_preferred is used, I need to add the support to label:{lang}_x_preferred
Should I Add variant too ?

related: pelias/api#1296

[orangejulius]

Oooh, looking forward to this. Keep in mind I just made a PR (#444) that drastically changes the format of some tests. You'll probably have to rebase :( On the plus side there are 1000 fewer lines in that test file :)

[Joxit]

ha ha thx I saw that, I force push :p

[Joxit]

Updated, I added the support for label:* and variants 😄 ready for review !

[Joxit]

Added commit for tests and iso639. WOF is using iso639 in their names. That caused some fail with
spanish/spain (spa vs esp).

[orangejulius]

Cool. this looks good to me. I ran a quick import and the size of WOF data increased from 1.8GB to 2.8GB, which is well worth it!

[NickStallman]

Hey @Joxit I just tried doing a planet import and it looks like there is a clash between WOF and this patch.

2019-06-23T22:33:33.340Z - error: [whosonfirst] {
  "id": 1310593459,
  "name": "Bahrdorf",
  "name_aliases": [],
  "name_langs": {
    "de": [
      [
        "Bahrdorf (Samtgemeinde Velpke)"
      ]
    ]
  },```

It looks like the nested array needs to be compacted down to a single array.

[Joxit]

Hi @NickStallman,
Are you using bundle import or SQLite ?
It seems that 1310593459 is now deprecated in the repository whosonfirst-data-admin-de

I tried a Germany import only (which contains Bahrdorf) and I didn't find any error (with the docker pelias/whosonfirst:master) 🤔

[bboure]

@Joxit @orangejulius I think this change might have introduced a small bug.
When a language has a preferred name equal to the default, but also has a variant, then the variant is returned by the API.

Example: Madrid
In English, preferred is "Madrid" (same as default), which is skipped.
There is also a variant: "MAD".
This results in ES like this:

"name": {
    "default": [
        "Madrid",
        "MAD"
    ],
	"en": "MAD",
}

And the API returns "MAD".

"properties": {
        "id": "101748283",
        "gid": "whosonfirst:locality:101748283",
        "layer": "locality",
        "source": "whosonfirst",
        "source_id": "101748283",
        "name": "MAD",
        "accuracy": "centroid",
        "country": "Spain",
        "country_gid": "whosonfirst:country:85633129",
        "country_a": "ESP",
        "macroregion": "Community of Madrid",
        "macroregion_gid": "whosonfirst:macroregion:404227387",
        "region": "Provincia de Madrid",
        "region_gid": "whosonfirst:region:85682783",
        "region_a": "MD",
        "locality": "Madrid",
        "locality_gid": "whosonfirst:locality:101748283",
        "continent": "Europe",
        "continent_gid": "whosonfirst:continent:102191581",
        "label": "MAD, Madrid, Spain"
      },

[Joxit]

Hi @bboure, is your placeholder up and contains Madrid ?
When you use search, it uses placeholder to get the correct name in the correct language.
When placeholder is not present, it fallbacks on ES name.

[bboure]

@Joxit I was indeed missing data in placeholder. It now works as expected.
Thank you