Unicode handling meta-issue [JIRA: CLIENTS-791]

[seancribbs]

We need to review and have a policy for all handling of Unicode character data. This issue consolidates and supercedes:

• Setting key to something unicode will make it fail #287
• MapReduce doesn't accept non-unicode inputs #156
• Client incorrectly rejects keys, buckets that are of the unicode datatype #32

Key issues:

• Uniform handling of object identifiers (bucket-type, bucket, key)
• Uniform handling of values encoded to/parsed from JSON

[reiddraper]

I'm happy to help here if I can. Do we have some uniform policy we use in the other clients? Say Java?

[seancribbs]

The plan I'm thinking of is this:

1. At the RiakClient object level, allow the user to set the desired charset. It should probably default to utf-8. If the user ever passes a str instead of a unicode type, we should probably assume it's already encoded (but I'm less firm on this point, it needs more investigation).
2. In all places where identifiers come back from Riak (put w/o key, key lists, bucket lists, index queries), decode them into the configured charset.

[reiddraper]

I suppose the issue with 2. is that if there is already binary data written, we won't be able to decode it as UTF-8 when we read. Would we want/need a 'bypass' decoder that returns str instead of unicode objects? I believe the Java client has had to deal with this as well, but maybe they just only work with Byte Arrays. cc @broach

[seancribbs]

I agree with your reservations about 2. Without roundtrip charset information however, we can't be sure of anything. As of now, we just puke back at the user if they give us a unicode.

[reiddraper]

As of now, we just puke back at the user if they give us a unicode.

Not sure I completely followed this.

[seancribbs]

As of now, we just puke back at the user if they give us a unicode.

Not sure I completely followed this.

https://github.com/basho/riak-python-client/blob/master/riak/riak_object.py#L113-L117

[reiddraper]

Oh yes, right. Get it now. We're certainly in a place where incremental improvement 'is a thing'.

[macintux]

Worth mentioning: we've explicitly disallowed the creation of non-ascii YZ indexes until we sort out filesystem compatibility and Riak's internal handling. I imagine the clients will just hand back the (not terribly informative) error message, but fyi.