Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQL JSON field type with string should decode in python3 #273

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

howmp
Copy link

@howmp howmp commented Jul 10, 2018

Fix #272

Table:

CREATE TABLE test (id int, value json);

With data:

id value
1 {"string1":"string2"}

But event's values in python3 is :

{"id":1,"value":{b"string1":b"string2"}

And when use json.dumps will raise TypeError: keys must be a string

According to rfc7159 : https://tools.ietf.org/html/rfc7159#section-8.1

JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32. The default
encoding is UTF-8

So should decode as utf-8

@howmp
Copy link
Author

howmp commented Jul 10, 2018

Please restart build, seems network error.

https://travis-ci.org/noplay/python-mysql-replication/jobs/402200041

@baloo

@baloo baloo closed this Jul 11, 2018
@baloo baloo reopened this Jul 11, 2018
@baloo
Copy link
Collaborator

baloo commented Jul 11, 2018

I'm not the owner of the project so can't restart directly. have to open/close :/ sorry

@howmp
Copy link
Author

howmp commented Jul 12, 2018

Failed when install cryptography in python 3.3,But I didn't change CI

@baloo
Copy link
Collaborator

baloo commented Sep 5, 2018

We've been converting charset all over the place. I dislike that a lot (it's not our job as a lib). Forcing encoding to utf-8 would be a nightmare to maintain on the long run. I'm not sure we want that.

We should find a better way for user to provide a encoding (even if we default to utf-8). Consider a column with two json fields, one utf8, one utf16.

@howmp
Copy link
Author

howmp commented Sep 6, 2018

In the documentation: https://dev.mysql.com/doc/refman/5.7/en/json.html

MySQL handles strings used in JSON context using the utf8mb4 character set and utf8mb4_bin collation. Strings in other character sets are converted to utf8mb4 as necessary. (For strings in the ascii or utf8 character sets, no conversion is needed because ascii and utf8 are subsets of utf8mb4.)

Json field can't be utf16

@rudaporto
Copy link

It would be good to remove support for Python 3.3 since nobody should be using it anymore (I guess).

Makes sense?

@howmp
Copy link
Author

howmp commented Oct 11, 2018

It would be good to remove support for Python 3.3 since nobody should be using it anymore (I guess).

Makes sense?

Yeah. I agree

Copy link

@telemmaite telemmaite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not move forward with this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants