Add Python Pickle format #150

moreati · 2019-05-19T22:54:46Z

Remaining tasks

Add protocol 4 types, opcodes etc

serialization/pickle.ksy

KOLANICH · 2019-05-20T06:48:21Z

serialization/pickle.ksy

+  - id: ops
+    type: op
+    repeat: eos
+  # TODO is there a way to declare a trailing STOP is required?


This doesn't work in combination with repeat: eos though, right? (The array will always consume all of the data, so the contents field will never have any data left and will always fail.)

If STOP can only appear at the end of the data, you could use repeat: until to terminate the array after the first STOP.

_io magic is to the rescue. One can determine the stream size and create a substream containing all but the last bytes. So the last bytes are to be used for contents and the rest is occupied by a substream containing repeat: eos

KOLANICH · 2019-05-20T06:50:14Z

serialization/pickle.ksy

+  They are deserialised by importing that constructor, and calling it.
+doc-ref:  https://github.com/python/cpython/blob/3.3/Lib/pickletools.py
+seq:
+  # TODO is there a way to declare PROTO is optional, but only valid at position 0?


There is, but it would require some _io hacks.

dgelessus · 2019-05-20T08:52:50Z

serialization/pickle.ksy

+        encoding: ASCII
+        terminator: 0x0a # "\n"
+    doc: Integer, encoded with the ASCII chracters [0-9-], followed by 'L'.
+  # TODO Can kaitai express constraint that these are quoted?


Would be difficult without something like kaitai-io/kaitai_struct#81.

serialization/pickle.ksy

These are not generally used, they were not intented to be in the initial commit.

This means efery parsed op will have the same attributes, and that every known opcode is explicitly tested for. So unknown opcodes are unambiguous. Explicit is better than implicit.

Also noticed that a positive exponent would be encoded as 1e+100, not 1e100.

serialization/python_pickle.ksy

dgelessus · 2019-05-23T23:26:32Z

By the way, PEP 574, which proposes version 5 of the Pickle protocol, has been accepted for Python 3.8. This may be interesting in the future, but right now it's way too early to support version 5 - it hasn't even been implemented in CPython yet, and the PEP is missing some important information, like concrete byte values for the new opcodes.

moreati · 2019-05-24T05:47:51Z

PEP 574, which proposes version 5 of the Pickle protocol, has been accepted for Python 3.8

Huh, thank you. Thought I'd seen that a a while back, and it had been rejected. Must have confused it with another proposal.

GreyCat

Looks good to me, thanks! In my perspective, it's ready to merge. @KOLANICH, @dgelessus — do you see any blockers, or I can go ahead?

dgelessus

Looks good to me. 👍

There are still a couple of TODO comments in the spec, but I think we can leave them in as a reminder, in case Kaitai Struct supports the necessary features in the future.

serialization/python_pickle.ksy

moreati

Sorry, didn't mean to start a review, and I (as the submitter) can't close it

serialization/python_pickle.ksy

KOLANICH · 2019-05-25T20:35:06Z

serialization/python_pickle.ksy

+      -orig-id: PROTO
+      doc: identify pickle protocol
+    0x81:
+      id: "newobj"


Since the original names are preserved in -orig-id how about changing the ids? For example adding words delimiters?

Not sure I understand, do you mean something like id: new_object?

for this id, exactly this.

Should I treat the type identifiers similarly? e.g. unicodestring8 -> unicode_string8

ETA For the current type identifiers I used the ArgumentDescriptor.name attributes in pickletools.py, e.g. https://github.com/python/cpython/blob/v3.7.3/Lib/pickletools.py#L702

Of course you should!

For the current type identifiers I used the ArgumentDescriptor.name attributes in pickletools.py

We have -orig-id for that, right?

We have -orig-id for that, right?

That covers the opcode names in the enum. I was asking about the names of the argument types, further up. Those are of the form

unicodestring8: seq: id: len ...

from my reading of the KSY spec I could add an -orig-id to len, but not to unicodestring8 as a whole. Did I miss something?

Any attr beginning from a dash can be added anywhere - they are ignored by KSC and they are for third-party tools.

serialization/python_pickle.ksy

GreyCat · 2019-05-28T15:38:00Z

Looks like vast majority of pressing issues was resolved, so I'll go ahead and merge this in. Thanks for this tremendous effort, everyone, especially @moreati!

Add Python Pickle format

b2312a3

KOLANICH reviewed May 20, 2019

View reviewed changes

serialization/pickle.ksy Outdated Show resolved Hide resolved

KOLANICH reviewed May 20, 2019

View reviewed changes

dgelessus reviewed May 20, 2019

View reviewed changes

serialization/pickle.ksy Outdated Show resolved Hide resolved

serialization/pickle.ksy Outdated Show resolved Hide resolved

moreati added 7 commits May 20, 2019 21:26

Remove personal file extensions

31febf9

These are not generally used, they were not intented to be in the initial commit.

Add forgotten long4 and float cases

25ccdf6

Enable cases for opcodes that take no argument

281cdce

This means efery parsed op will have the same attributes, and that every known opcode is explicitly tested for. So unknown opcodes are unambiguous. Explicit is better than implicit.

Note that floatnl type can encode infinties

5f04792

Also noticed that a positive exponent would be encoded as 1e+100, not 1e100.

Declare stringnl and stringnl_noescape encodings

5f473d6

Rename pickle format to python_pickle

ffa2a9a

Spelling correction in title

7fe651c

dgelessus reviewed May 20, 2019

View reviewed changes

serialization/python_pickle.ksy Outdated Show resolved Hide resolved

serialization/python_pickle.ksy Show resolved Hide resolved

serialization/python_pickle.ksy Outdated Show resolved Hide resolved

moreati added 9 commits May 21, 2019 20:58

Add cross references

0d0c03b

Spelling correction

33bc259

Remove encoding from string1 & string4, add explanation

31ea16e

Tweak formatting, separate types with whitespace

583c4e6

Remove protocol from title, add brief explanations of protocols

936ffd1

Mention security risk of unpickling abitrary pickles

67190fd

Add link to Pickle protocol 2 Python Enhancement Proposal (PEP)

65e4636

Correct maximum length in doc of bytes4 type

fb72641

Add pickle protocol 4 opcodes and types

eb2eda2

GreyCat reviewed May 23, 2019

View reviewed changes

serialization/python_pickle.ksy Outdated Show resolved Hide resolved

Align indentation of enums

ce1aab3

GreyCat self-requested a review May 24, 2019 14:11

GreyCat approved these changes May 24, 2019

View reviewed changes

dgelessus approved these changes May 24, 2019

View reviewed changes

KOLANICH reviewed May 24, 2019

View reviewed changes

serialization/python_pickle.ksy Show resolved Hide resolved

KOLANICH reviewed May 24, 2019

View reviewed changes

serialization/python_pickle.ksy Show resolved Hide resolved

KOLANICH reviewed May 24, 2019

View reviewed changes

serialization/python_pickle.ksy Outdated Show resolved Hide resolved

KOLANICH reviewed May 24, 2019

View reviewed changes

serialization/python_pickle.ksy Outdated Show resolved Hide resolved

moreati commented May 25, 2019

View reviewed changes

serialization/python_pickle.ksy Outdated Show resolved Hide resolved

moreati added 2 commits May 25, 2019 21:18

Use lowercase for all encoding: declerations

50c81b9

Document protocol 0 False/True special case

b10f693

KOLANICH reviewed May 25, 2019

View reviewed changes

serialization/python_pickle.ksy Show resolved Hide resolved

KOLANICH reviewed May 25, 2019

View reviewed changes

serialization/python_pickle.ksy Show resolved Hide resolved

GreyCat merged commit e28b3f1 into kaitai-io:master May 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Python Pickle format #150

Add Python Pickle format #150

moreati commented May 19, 2019 •

edited

Loading

KOLANICH May 20, 2019

dgelessus May 20, 2019

KOLANICH May 20, 2019

KOLANICH May 20, 2019

dgelessus May 20, 2019

dgelessus commented May 23, 2019

moreati commented May 24, 2019

GreyCat left a comment

dgelessus left a comment

moreati left a comment

KOLANICH May 25, 2019

moreati May 25, 2019

KOLANICH May 25, 2019 •

edited

Loading

moreati May 25, 2019 •

edited

Loading

KOLANICH May 25, 2019

moreati May 27, 2019

KOLANICH May 27, 2019 •

edited

Loading

GreyCat commented May 28, 2019

Add Python Pickle format #150

Add Python Pickle format #150

Conversation

moreati commented May 19, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgelessus commented May 23, 2019

moreati commented May 24, 2019

GreyCat left a comment

Choose a reason for hiding this comment

dgelessus left a comment

Choose a reason for hiding this comment

moreati left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KOLANICH May 25, 2019 • edited Loading

Choose a reason for hiding this comment

moreati May 25, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KOLANICH May 27, 2019 • edited Loading

Choose a reason for hiding this comment

GreyCat commented May 28, 2019

moreati commented May 19, 2019 •

edited

Loading

KOLANICH May 25, 2019 •

edited

Loading

moreati May 25, 2019 •

edited

Loading

KOLANICH May 27, 2019 •

edited

Loading