Allowed characters in keys

[tordanik]

The study's section 4.3 on "Limiting Length and Allowed Characters in Strings" talks about string length, but is mostly silent about allowed characters.

Personally, I feel that the consensus against certain characters in keys is quite strong and would justify enforcing such restrictions through the API. As evidence, I would like to point to:

• existing validator rules
• wiki documentation about syntactic conventions
• decisions in evolving tagging standards. (For example, I myself once wrote a proposal which faced opposition because it would have necessitated characters such as '=' or '<' in keys. The community favored the conditional restrictions syntax which moves these characters into the value.)

Technologically, this would be a comparatively straightforward change with inexpensive server-side validation. Seeing how the API already rejects certain characters (for compatibility with XML), this change also would not impose any fundamentally new challenge for users of the API.

The benefits are proportional to the relatively low cost: small but tangible. In one example, not permitting the equals character to be used in keys would make the very common =-separated text representation of OSM tags unambiguous. This is beneficial for query languages, URLs, and similar human-readable interfaces to OSM data.

[mmd-osm]

That's one of the topics where I'd strongly recommend to start addressing issue right at the source (= in the editing applications), before adding such checks on API level only.

Checks on API level implies that you have to include a list of non-permitted characters in /api/capabilities, so that editing apps can take those rules into account even before trying an upload.

Simply returning HTTP 400 Bad request for uploads with non-permitted characters, without telling anyone what today's forbidden characters are, won't work. It's usually game over for your changeset, and you will have lots of unhappy users.

I think this topic doesn't strictly require a version bump to API 0.7. Maybe talk to @simonpoole, he has relevant details to share on the topic.

[501Ghost]

These problematic characters are already monitored on Taginfo: https://taginfo.openstreetmap.org/reports/characters_in_keys#problem
Most of it is very hard to decrypt except for the handful of people who know all the tagging practices and have the time to even look at these things. In my opinion the most common ones on this list could be retagged to something less offensive (like communication:4G+ to communication:4G_plus) and the rest could be deleted with minimal loss of data. An alternative would be to keep these tags in a stagnant copy of the OSM DB with the current data model for the sake of QA, which would be an obscure niche even for those who are familiar with QA.

[mvl22]

I feel that the consensus against certain characters in keys is quite strong and would justify enforcing such restrictions through the API.

I agree. This would be sensible to implement at the API side and is probably low risk.

A version bump would be best, but if not one could envisage a 'strict' mode, similar to a 'use strict' kind of model, where the API expects things like this to cause a hard error. Strict mode could be off by default, but switched on by a certain date.

[simonpoole]

I find it rather difficult to justify doing anything at all on this topic, the number of keys in this category is minuscule, even minuscule compared to other problematic keys. Not to mention that the data consuming applications should be at least robust enough to ignore any input they can't deal with.

That said, as @mmd-osm pointed out, simply enforcing this in the API is extremely user unfriendly, we should at least provide editing apps with a fighting chance of warning their users about "illegal" characters on entry, that is when errors can still be easily fixed.

I suppose I could make a case for reserving certain characters for "future use" given that there is undoutably going to be even more pressure to structure and move values (in particular names, which can't really be restricted without some kind of agreement on normalisation) in to keys going forward.

[501Ghost]

If you really care about user experience, you won't even let users upload data that the API would reject in any case. That's the purpose of including such information in /api/capabilities and have the editing apps enforce it.

I agree.

We already have that in place for aerial imagery, where users can't draw buildings from Google imagery as an example. Please familiarize yourself with this concept, as it's essential to understand how the API communicates any restrictions back to editing apps.

That's interesting. I never tried to use Google for OSM (for obvious reasons), so I am indeed unfamiliar with this concept.

Conflict-like error handling is really the worst possible user experience I can think of.

I merely made the suggestion. How would you communicate to mappers that they can't upload changesets that contain disallowed characters in keys? Maybe it's something that can also be applied to editing conflicts if you think the user experience of that can be improved.

I'm not sure I'm getting your point here. Can you elaborate a bit how and where exactly this introduces complexity? Please be as specific as possible and name concrete examples.

https://wiki.openstreetmap.org/wiki/Proposed_features/Defaults has introduced some awkward tags to OSM such as def:highway=footway;access:bicycle=yes. You need to meditate on OSM tagging practices like a monk before you can understand this kind of advanced tagging within a single key, which would be acceptable for some avid mappers but not for most users of the data. Seeing how this idea went nowhere I'd say the complexity of it made it impractical.

[simonpoole]

I probably would have used the limits on the number of way nodes and relation members as restrictions that are conveyed to editing apps and can be enforced/handled while editing is in progress.

Conflict handling is a particularly bad example, because there is no reliable way to detect conflicts till the time the API backend tries to merge the uploaded changes and we have no choice than to handle such issues very late. Nearly everybody considers this problematic, but as said there is no way around it.

[mmd-osm]

So let's take a look at https://api.openstreetmap.org/api/0.6/capabilities:

The google imagery I was talking about is in the policy section. Way nodes and relations member limits can be found in the API section. You can find other limits in there, as such the maximum number of elements per changeset, which is currently 10'000.

Editing apps are supposed to fetch this info during startup, and then enforce them wherever applicable. Of course, the API will also make sure that api limits are not exceeded.

Adding some non-allowed characters would have to go in this file as well. For a halfway sane process, you would have to announce those changes well ahead of time, and give editing app developers plently of time to implement changes on their end. The same applies to the API itself. Once that is done, you can flip the switch by changing capabilities file contents on the osm.org server. These changes become effective immediately on a global scale.

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="OpenStreetMap server" copyright="OpenStreetMap and contributors" attribution="http://www.openstreetmap.org/copyright" license="http://opendatacommons.org/licenses/odbl/1-0/">
  <api>
    <version minimum="0.6" maximum="0.6"/>
    <area maximum="0.25"/>
    <note_area maximum="25"/>
    <tracepoints per_page="5000"/>
    <waynodes maximum="2000"/>
    <relationmembers maximum="32000"/>
    <changesets maximum_elements="10000"/>
    <timeout seconds="300"/>
    <status database="online" api="online" gpx="online"/>
  </api>
  <policy>
    <imagery>
      <blacklist regex=".*\.google(apis)?\..*/.*"/>
      <blacklist regex="http://xdworld\.vworld\.kr:8080/.*"/>
      <blacklist regex=".*\.here\.com[/:].*"/>
    </imagery>
  </policy>
</osm>

[mvl22]

Given that 'the number of keys in this category is minuscule' and the Taginfo report shows hardly any instances, I see little reason why not to put in place a tightened API response, since it seems hardly anyone would be affected.

Even if the user experience for an unhandled API response error would be a bit poor, current practice shows that it would hardly ever happen anyway. I'd argue it's better for some tiny miniscule number of people to get an unhandled response and then do a search to try to discover what they did wrong, than to have invalid data entering the database.

Is there a formal definition of the allow characters somewhere? If not the first step is surely to formalise the de-facto situation by consensus.

[mmd-osm]

I'd argue it's better for some tiny miniscule number of people to get an unhandled response and then do a search to try to discover what they did wrong

I can assure you that this won't happen. We will not be implementing such checks on API side without having corresponding checks in the editing apps in place.

[501Ghost]

I mentioned this discussion on https://wiki.openstreetmap.org/wiki/Talk:Proposed_features/Defaults#Problematic_characters_in_keys