Workaroud for UnicodeDecodeError

[matthieuxyz]

Why:

Fix a random UnicodeDecodeError, when using AdbCommands.Logcat.

Steps to reproduce

• Add multi-bytes characters to log file
• Try to use AdbCommands.Logcat

Expected result

Output result as unicode string (or str in python3)

Actual result

UnicodeDecodeError is sometime raised with the following message:

ERROR: 'utf-8' codec can't decode byte 0xe3 in position 4095: unexpected end of data

The exception is raised from the method AdbMessage.StreamingCommand() from the module adb_protocol.py:

    @classmethod
    def StreamingCommand(cls, usb, service, command='', timeout_ms=None):
        """
        ...
        """
        if not isinstance(command, bytes):
            command = command.encode('utf8')
        connection = cls.Open(
            usb, destination=b'%s:%s' % (service, command),
            timeout_ms=timeout_ms)
        for data in connection.ReadUntilClose():
            yield data.decode('utf-8')  # <---- HERE

Investigation

When streaming output of command, a max length is provided to UsbHandle.BulkRead. This length is (if I understood correctly) given in bytes.

So the end of a bulk of bytes data can append at any given bytes.

But in utf-8, characters may be encoded on multiples bytes (ex: あ (U+3042) is encoded as 0xE3 0x81 0x82 in utf-8).

Since, the bulk may end at any given bytes, it is possible for a multi-byte sequence to be split between two bulks (ex: [b'...\xe3', b'\x81\x82...'] ).

But, decoding will fail if a multi-byte sequence is incomplete (try to run b'AAA\xe3'.decode('utf-8') in a python console).

What:

This pull request contains...

• A test case with a @skip decorator to illustrate the problem.

• New "Bytes" methods that implements a "workaround" (ie. using bytes and decode only when stream is complete)

• New tests to cover the new "Bytes" methods

There is probably a better solution, but this is what worked for me. And I hope that it will help to illustrate the problem with StreamingCommand.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[coveralls]

Coverage increased (+1.2%) to 44.888% when pulling 5f45f7c on matthieuxyz:master into f4e597f on google:master.

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[JeffLIrion]

@matthieuxyz thanks for the detailed explanation! I implemented this fix for the adb-shell package, although I didn't add a specific test for this scenario. Could you please take a look and let me know if the change looks correct?

JeffLIrion/adb_shell#34

[matthieuxyz]

Seems good to me. I made a pull request with additional tests to make sure:

JeffLIrion/adb_shell#35