Go language support

[5k3105]

I believe the appropriate lower level libraries to use will be:
https://golang.org/pkg/bytes/
https://golang.org/pkg/os/

general outline of a go program. Taken from a sprite editor I was making for it's 'low levelness'.
github.com/virtao/GoTypeBytes has a lot of functions on bytes that seem applicable for this library.

package main // name of ksc here I think

import (
        "github.com/virtao/GoTypeBytes"
	"golang.org/pkg/bytes"
	"golang.org/pkg/os"
)

type pallete struct {
	r, g, b []int
}

func newpallete() {

	file, err := os.Open(filename) // For read access.
	if err != nil {
		e = "1"
	}

	fi, err := os.Stat(filename)
	if err != nil {
		e = "2"
	}

	data := make([]byte, fi.Size())
	fmt.Println("fi.Size:", fi.Size())

	count, err := file.Read(data)
	file.Close()
	if err != nil {
		e = "3"
	}

	fmt.Printf("read %d bytes: %q\n", count, data[:count])

	var bt bytes.Buffer

	bt.Write(data[:count])

	var r, g, b []int

	for ; count > 0; count-- {

		r = append(r, typeBytes.BytesToInt(bt.Next(1)))

		g = append(g, typeBytes.BytesToInt(bt.Next(1)))

		b = append(b, typeBytes.BytesToInt(bt.Next(1)))

		bt.Next(3)

	}

	p := &pallete{
		r: r,
		g: g,
		b: b,
	}

	fmt.Println(e)

	return p
}

[GreyCat]

Thanks for the suggestion! Let's go with our checklist. I presume you're acquainted with KS, so going to section 2. What would be Go equivalents of the following:

• KS "type"
• KS "stream"
• KS primitive types (i.e. what maps to what — integers, floats, strings, byte arrays, generic arrays) — please also elaborate on extra stuff mentioned in that section like encodings and null support

What are the basic Go coding standards / universally accepted practices that we'll follow (also as per checklist, section 2.4)?

[5k3105]

just taking a stab at this --

KS type:
https://golang.org/ref/spec#Numeric_types

uint8       the set of all unsigned  8-bit integers (0 to 255)
uint16      the set of all unsigned 16-bit integers (0 to 65535)
uint32      the set of all unsigned 32-bit integers (0 to 4294967295)
uint64      the set of all unsigned 64-bit integers (0 to 18446744073709551615)

int8        the set of all signed  8-bit integers (-128 to 127)
int16       the set of all signed 16-bit integers (-32768 to 32767)
int32       the set of all signed 32-bit integers (-2147483648 to 2147483647)
int64       the set of all signed 64-bit integers (-9223372036854775808 to 9223372036854775807)

float32     the set of all IEEE-754 32-bit floating-point numbers
float64     the set of all IEEE-754 64-bit floating-point numbers

complex64   the set of all complex numbers with float32 real and imaginary parts
complex128  the set of all complex numbers with float64 real and imaginary parts

byte        alias for uint8
rune        alias for int32

also take a look at https://github.com/virtao/GoTypeBytes/blob/master/typeBytes.go for conversion from and to these types directly from bytes.

KS 'stream':
https://golang.org/pkg/bytes/
https://golang.org/pkg/io/
https://golang.org/pkg/os/

I believe these are the packages required but I have not experimented with this yet :)

a collection of fields parsed sequentially (seq)

A slice is a dynamically sized array of type. https://golang.org/ref/spec#Slice_types

fields := []int{1,2,3}

a number of fields parsed randomly or calculated (instances)

Not sure what you want here. There is a key value hash map in Go. So if key is 'field' then:

fields := make(map[string]interface{}) 

this would create a kv map with a string as the key and interface ('any' type that can be cast to correct type later). https://golang.org/ref/spec#Interface_types

illustration of slice and map: https://play.golang.org/p/K1gaQ1zhHg

definition of nested types (types)

is this structs? https://golang.org/ref/spec#Struct_types

lists of integer constants (enums)
http://stackoverflow.com/questions/14426366/what-is-an-idiomatic-way-of-representing-enums-in-go#14426447

https://github.com/virtao/GoTypeBytes/blob/master/typeBytes.go
That link also has these enums defined:

const (
	INT_SIZE       int = int(unsafe.Sizeof(0))
	UINT_SIZE      int = int(unsafe.Sizeof(uint(0)))
	FLOAT32_SIZE   int = 4
	FLOAT64_SIZE   int = 8
	DEFAULT_ENDIAN int = 0
	BIG_ENDIAN     int = 1
	LITTLE_ENDIAN  int = 2
)

[GreyCat]

KS type:

https://golang.org/ref/spec#Numeric_types

This list is also useful, but the primary question was how to present KS concept of "type". In the majority of target languages, it corresponds to a class. seq attributes usually correspond to private class members + getter / setter methods or properties. instances are more or less the same, but they are lazy (and thus have some sort of "not-yet-parsed" state by default) and they usually require more than one line of code in a getter, given that they usually have some sort of position, sizes, etc.

Reading a seq is usually implemented in a _read method, writing of seq is implemented in _write. In normal, read-only parser mode, class constructor gets KaitaiStream and automatically calls _read to do sequence parsing from that given stream.

How all that maps to Go concepts and practices?

a number of fields parsed randomly or calculated (instances)

Not sure what you want here.

Um, please don't take offense, but may be you'd want to get slightly more familiar with KS concepts, such as seq, instances, enums, nested types, etc?

[5k3105]

KS type seems to follow: https://golang.org/ref/spec#Types

There are usually no getter setter methods. They are treated as a variable but can also be given methods:

https://gobyexample.com/methods

gives an example of this but with a more complex type, a struct. Yet it is still a 'type' you can see in it's declaration.

Give me a bit longer, I am new to KS and I need to read more. Thanks.

[GreyCat]

They are treated as a variable but can also be given methods:

Yeah, that should be useful as well. We could use that for "parse instances" (which define some out-of-sequence data located at some arbitrary offset in the stream) and "value instances" (which are basically values, calculated using other available attributes).

Give me a bit longer, I am new to KS and I need to read more. Thanks.

No rush, take your time :) Also, you might be interested in announcing this work on support of Go in KS in some Go communities. I've heard that they are very friendly and open to new things — so, chances are, we'll get more people interested in this support contributing ideas. N heads is better than one, anyway :)

[5k3105]

I'll post something up on the reddit :) Thanks!

[5k3105]

https://www.reddit.com/r/golang/comments/63oa75/kaitai_struct_a_new_way_to_develop_parsers_for/

[tarm]

I saw this on Reddit and am suddenly interested in Kaitai.

I am familiar with Go, but not with Kaitai or scala. I have read through the documentation about adding support for a new language and believe it should be pretty straightforward to get started with a prototype for Go.

Here are my thoughts on how the representation should be in Go (also a basic introduction to some Go concepts):

• Each top level .ksy should generate a "Go package" by default (later an option could be added for customizing the package name etc), and all the generated source code should go into a single .go file.
• names should be CamelCase
• seq should match to a "Go slice" of a particular type. Go syntax for a slice of uint32's is []uint32. Slices can be extended to at runtime with the append() builtin. For example: arr = append(arr, 5)
• Kaitai my_name type should be represented like this

// MyType doc string
type MyName struct {
    Field0  uint32   // Field Doc string
    ...
}

• Nested types are not directly supported in Go. The protobuf compiler handles this by prefixing the nested type with the parent type name and I think that's what the Kaitai compiler should do as well. type ParentType_NestedType struct {...}
• The typical way to handle binary data for a particular type is to add methods to that type and they should be called Marshal() and Unmarshal(). Those methods should return the Go error type and operate on byte slices: []byte.
• The Kaitai stream could be implemented in a straightforward way as an extension of the bytes.Reader type. I believe the primary thing that would need to be added is reading of bits within a byte, but that should be straightforward.
• Instances should be fields within a top level type (struct)
• Go has pointers. Optional fields should be represented as pointers and the user can be responsible for checking whether the pointer is nil
• Go does not have "tagged unions" like some languages. A union can be represented as an interface{}, but it still might be preferred to have a series of pointers (depends on the use case).
• Go typically does not use Getters and Setters as much as some languages, but I think a Getter method could be an appropriate way to represent a Kaitai value type
• Kaitai enums should be represented as as series of typed constants in Go. That would look like this:

type IpProtocol uint8

const (
    Icmp IpProtocol = 1
    Tcp  IpProtocol = 6
    Udp  IpProtocol = 17
)

• Go is namespaced by package but otherwise the names for Kaitai types, enumerations, and enumeration values could all collide. If that is allowed in Kaitai (likely), and collisions are a concern then the generated names might have to be Prefixed_ or something. Protobuf handles that by prefixing enum values with the enum type name: IpProtocol_Tcp IpProtocol = 6.

I tried prototyping the interface with an Unmarshal() method, but I think that should return the number of bytes consumed from the stream. Here is how the standard library does in with the asn1 encoding but that is also a bit awkward: https://golang.org/pkg/encoding/asn1/#Unmarshal

The API for the protobuf encoding and decoding might be interesting (it's also a little bit confusing when you first look at it because of the way the decoding references autogenerated metadata fields): https://godoc.org/github.com/golang/protobuf/proto

Here is an example of how I might generate code:
https://play.golang.org/p/fNbIPlOcpQ
The top level struct (ExampleFile) is still a little bit awkward and could use some more thinking.

What are the next steps? Explicitly following the "new language" document? @GreyCat Are you available to write the scala code generation if a reasonable template can be provided? I can work more on the Streaming API.

[tarm]

Here is what the streaming API might look like in Go: https://play.golang.org/p/O1vKDZroZ1

The io.ReadSeeker could be either a an os.File or a bytes.Reader

[GreyCat]

@tarm Thanks for your interest!

names should be CamelCase

Judging by the examples, probably you mean that type names, sequence attribute names and enums-as-consts should be UpperCamelCase. What about instances?

The protobuf compiler handles this by prefixing the nested type with the parent type name and I think that's what the Kaitai compiler should do as well.

Ok, not a problem.

The typical way to handle binary data for a particular type is to add methods to that type and they should be called Marshal() and Unmarshal(). Those methods should return the Go error type and operate on byte slices: []byte.

I believe that goes well with seq, but what about instances? Is there any way to provide lazy parsing support of arbitrary data in current and/or other streams?

Here is an example of how I might generate code: https://play.golang.org/p/fNbIPlOcpQ

Thanks, it makes much more sense now! I believe that stuff like that:

	nn, err := e.TrackTitle.Unmarshal(b[n:])
	n += nn
	if err != nil {
		return n, err
	}

can be incapsulated into some sort of KaitaiStream-like library?

What are the next steps? Explicitly following the "new language" document?

We need:

1. A minimal working example of what exactly we should generate with KS — and that generally means minimal working implementation of KaitaiStream / KaitaiStruct (if it would be of any use).
2. Some testing code for that, ideally, one that we can incorporate into our tests repo.

Then I add new target -t go to the compiler, and we try to get it to generate exactly the HelloWorld.go code that you've provided based on hello_world.ksy. As soon as we'll succeed there, we'll start CI testing. Then just keep on porting more test specs to Go and fixing the compiler gradually more and more to make it work.

@GreyCat Are you available to write the scala code generation if a reasonable template can be provided? I can work more on the Streaming API.

Sure, I'm totally up to it ;)

[GreyCat]

Here is what the streaming API might look like in Go: https://play.golang.org/p/O1vKDZroZ1

Cool! I'd like to start a Go runtime repository, but we need to choose name that Go will be referred to everywhere in KS. I guess there are two alternatives - shall we use go or golang - which one is better?

The io.ReadSeeker could be either a an os.File or a bytes.Reader

By the way, what about reading performance when using os.File - would it buffer read calls, so you don't end up with 1000s of one-byte read syscalls?

[tarm]

Instances could also be represented as methods on the top level type. (I'm not familiar with how common they are and what their uses are).

For naming, I think go is preferred.

Is the generated code typically standalone (like the Unmarshal example I gave) or does it typically use metadata/reflection and runtime assistance from the Stream object? Obviously either is possible, but one is probably easier from the existing code generation structure. If reflection is typical, then it might be possible to make something pretty clean looking by putting all the metadata into field attributes that Go calls struct tags. That looks like this https://play.golang.org/p/mtGr8J_szY

os.File is not buffered. It is normally to wrap the io.File in a bufio.Reader though if you want to buffer. Unfortunately bufio.Reader does not have a Seek() method, so there may be a little bit more fiddling to do.

OK let me work more on hello world.

[tarm]

BTW, could you point me to what example generated code looks like? Hello world python for example. I see the unit test in the spec directory, but haven't seen generated code.

[GreyCat]

I've created https://github.com/kaitai-io/kaitai_struct_go_runtime — feel free to start runtime project there. Would you be ok with MIT license, as most our other runtimes use?

Is the generated code typically standalone (like the Unmarshal example I gave) or does it typically use metadata/reflection and runtime assistance from the Stream object?

If I understand properly, generally, there is no metadata/reflection usage inside generated code to maximize performance / efficiency. There is no "parsing state" (except for the state that Stream object provides), we try to minimize various conditions / checks.

If reflection is typical, then it might be possible to make something pretty clean looking by putting all the metadata into field attributes that Go calls struct tags.

That's an interesting feature, but that looks more like "an interpreter of provided metadata, encoded in tags", not "compiler". Generally, "compiler" approach is more efficient.

BTW, could you point me to what example generated code looks like? Hello world python for example. I see the unit test in the spec directory, but haven't seen generated code.

You can use ksc to compile it yourself, or even can use http://kaitai.io/repl/ to check out generation results. For Python, for example, it generates:

# This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild

import array
import struct
import zlib
from enum import Enum
from pkg_resources import parse_version

from kaitaistruct import __version__ as ks_version, KaitaiStruct, KaitaiStream, BytesIO


if parse_version(ks_version) < parse_version('0.7'):
    raise Exception("Incompatible Kaitai Struct Python API: 0.7 or later is required, but you have %s" % (ks_version))

class HelloWorld(KaitaiStruct):
    def __init__(self, _io, _parent=None, _root=None):
        self._io = _io
        self._parent = _parent
        self._root = _root if _root else self
        self.one = self._io.read_u1()

[ghost]

@tarm The following issues contain answers on some questions:

• PHP 7 support
• Perl support

[tarm]

Thanks for the pointers! I've made a little progress and would like to share and get feedback:

Runtime stream library: https://github.com/tarm/kaitai_struct_go_runtime/blob/master/kaitai/stream.go

• The bit and byte reading and conversions should be complete.
• Some of the APIs are not clear to me and it would be helpful if the porting guide had a little more description of the purpose (I have FIXMEs in there right now)
  -- strz what is the intention of all the arguments?
  -- do the processing functions operate on the buffers in-place? I assumed so.
  -- what is the read_bits_array api?
  -- process_rotate_left what the api? Is it really needed? It looks like several runtimes have incomplete implementations.
• In the abstract, I don't fully understand a seekable stream API. If it's seekable, then you must be able to rewind which means it's not a stream unless you've buffered it. If you're going to have to buffer it, then you might as well buffer the whole stream before processing. Go has good interfaces for streaming and good interfaces for seeking on buffered data, but not both at the same time. Is Seek() only needed for instances with positions? Can you point to a .ksy that needs a Seek() method? I'm tempted to defer the Seek() implementation for a little while until I can explore how the generated code for those should look. Maybe there could be 2 APIs, streaming and seeking, and only .ksy's the need seeking would consume the seeking interface?

Hello world manual compile and a unit test for it:
https://github.com/tarm/kaitai_struct_tests/tree/master/spec/go
It looks like the other unit tests are derived from the ruby tests? I am planning to continue with a couple more manual conversions and unit tests, but let me know if those should be autogenerated.

I do not know what the "Kaitai Struct" base type/interface would need to be and would like to explore some examples to get a feeling.

I think next steps for me would be to manually convert a few more .ksy files that have more features. Any suggested progression from Hello World? Perhaps something with a parent reference and something with instances?

Are there plans to have an interface to write files from structures? Naively it seems like all the operations could just be done in reverse.

[GreyCat]

Ok, first I'm gonna answer questions on runtime API:

func (k *Stream) EOF() bool {
// Not sure about this one. In Go, an io.EOF is returned as
// an error from a Read() when the EOF is reached. EOF
// handling can then be done like this:
...

Generally, there are two approaches to EOF handling in this world:

• EOF returns true when we've reached position when no further reading is possible
• EOF returns true after we've reached end, and made a read request that failed (thus stream is marked as "EOFed")

Given that we inevitably deal with languages that have both schemes, we had to choose a single common pattern here, and we've chosen the first one. Languages like C++ and Python, that rely on the second pattern, will thus have slightly more complex implementations - like this one in C++ or this one in Python. Probably we'll have to implement this with Go as well.

// Go's string type can contain any bytes. The Go range operator
// assumes that the encoding is UTF-8 and some standard Go libraries
// also would like UTF-8. For now we'll leave any advanced
// conversions up to the user.

What's the difference between a string a []byte then? If there's some standard to use UTF-8, probably we'll just need to convert everything to UTF-8. It's not like we have many things that depend on exact string behavior right now, but generally it's expected that length method called on a string will return a number of characters, not bytes, reverse will reverse characters correctly, etc.

// FIXME what does this method do?
func (k *Stream) ReadBitsArray(n uint) error {

Generally, it is supposed to read a bitset / bit array data type with given number of bits. See #112 for more details, it's right now in planning stage. Probably we'll use BitSet for Java, BitArray for .NET, std::bitset for C++, etc.

func (k *Stream) ReadStrEOS(encoding string) (string, error) {
func (k *Stream) ReadStrByteLimit(limit int, encoding string) (string, error) {

I see that you've implemented older v0.6 API here — probably because I've referred you to some outdated documentation, and I'm truly sorry for that. Modern 0.7 API does not have distinct string reading methods, instead it uses byte array reading methods (read_bytes, read_bytes_full, read_bytes_term) + convert byte array to string method (bytes_to_str). Please take a look at any existing runtimes to see what's going on with byte arrays and strings there in meantime, and I'll try to bring all API specs up to date soon.

// FIXME what is group_size ?
func ProcessRotateLeft(data []byte, amount int, group_size int) {

Generally, it is supposed to split given byte array into groups of group_size bytes, and rotate each of these groups left by a given amount of bits, wrapping on an edge of the group. Thus, for example, rotating 12 34 ab cd with amount of 4 and group_size of 2, should bring us 23 41 bc da:

12       34      |ab       cd
00010010 00110100|10101011 11001101
   /         /
  /         /
 /         /
00100011 01000001|10111100 11011010
23       41      |bc       da

Using default group_size of 1 will rotate everything within a single byte, thus yielding:

12       34      |ab       cd
00010010 00110100|10101011 11001101
   /         /
  /         /
 /         /
00100001 01000011|10111010 11011100
21       43      |ba       dc

[GreyCat]

-- strz what is the intention of all the arguments?

Generally, strz is deprecated in favor of read_bytes_term. Here's more or less typical implementation for languages that lack native "read until terminator" functionality. Arguments match .ksy keys closely, that is:

• "term" is a terminating byte
• "include_term" is a boolean; if it's true, then we need to include terminator byte into a resulting byte array
• "consume_term" is a boolean; if it's true, we're expected to leave the stream pointer after reading the terminator byte, so further reading will proceed after the terminator; if it's false, we're expected to leave pointer pointing exactly at the terminator byte, so further reading will read that terminator byte again.
• "eos_error" is a boolean; if it's true, then reading up to the end of stream without finding the terminator is considered an error (so typically an exception is raised → I guess it should mean returning an error for Go), it it's false, then we just treat reaching the end of stream as success condition and return whatever we've got without triggering any errors.

-- do the processing functions operate on the buffers in-place? I assumed so.

Normally, they operate at copies and return them, but it might be customized for particular target language.

In the abstract, I don't fully understand a seekable stream API. If it's seekable, then you must be able to rewind which means it's not a stream unless you've buffered it. If you're going to have to buffer it, then you might as well buffer the whole stream before processing. Go has good interfaces for streaming and good interfaces for seeking on buffered data, but not both at the same time.

It's clearly a terminology question ;) I see that Go's understand of "stream" term implies that it's something that is not seekable and you may be able only to rewind it with some extra pain (doing that buffering). In other languages (and more or less in KS), "stream" is used as a term that just describes an abstraction over generic reading / writing API that deals with files and in-memory byte arrays. Thus it's perfectly fine to have a "seekable stream": for example, Ruby's IO or Python's io work in that paradigm.

Is Seek() only needed for instances with positions?

Yes, it's more or less so.

Can you point to a .ksy that needs a Seek() method?

Sure, here's probably the minimal one: https://github.com/kaitai-io/kaitai_struct_tests/blob/master/formats/instance_std.ksy

For example, in Python it generates:

    @property
    def header(self):
        if hasattr(self, '_m_header'):
            return self._m_header if hasattr(self, '_m_header') else None

        _pos = self._io.pos()
        self._io.seek(2)
        self._m_header = (self._io.read_bytes(5)).decode(u"ASCII")
        self._io.seek(_pos)
        return self._m_header if hasattr(self, '_m_header') else None

I'm tempted to defer the Seek() implementation for a little while until I can explore how the generated code for those should look. Maybe there could be 2 APIs, streaming and seeking, and only .ksy's the need seeking would consume the seeking interface?

That's up to you to decide as Go expert ;) The general requirement is that parsing many real-life file formats (and even some network packets, like DNS) requires seeking. If you feel like providing several implementations (one that to be used when seeking is required and one that can be used for simpler types with no seeking) — sure, why not :)

The only language that might have benefited from such approach so far was probably Java (it has unseekable stuff like FileInputStream). I've discussed that with like a dozen of Java experts, and we've more or less came to a conclusion that it's generally unpractical to support it.

[5k3105]

Happy to read along :) 👍

[GreyCat]

@tarm I've started GoCompiler, and went over your rendition of hello_world.go.

I have a few questions:

1. As far as I can see, Unmarshall needs mandatory stream argument s *kaitai.Stream. Normally, default stream is expected to be stored in a class, so it can be reused by instances. Instance attributes are normally supposed to be called exactly the same as sequence attributes, i.e. whether you have:

seq:
  - id: foo1
    type: u1
instances:
  foo2:
    pos: 1234
    type: u1

You can access both foo1 and foo2 the same way. In Java, for example, it's r.foo1() and r.foo2(), in Python, it's r.foo1 and r.foo2, etc.

So, shall we add some field (traditionally we call it _io, but Go would probably have slightly different idea on identifier naming) into a type, and store s *kaitai.Stream argument into that field in Unmarshall?

2. This HelloWorld type completely ignores references to root and parent objects. Of course, they are not really needed for HelloWorld, but probably we don't want to drop that support anyway — who knows, maybe someone would want to include HelloWorld type in some other type?

Normally, we allow to supply root and parent arguments in a class constructor. Given that Unmarshall is effectively a constructor, may be we should add root and parent arguments there as well, and store them in a type?

[matttproud]

Is this anything you would want some help with? I stumbled across this project on a whim after having implemented decoders for a bunch of legacy game file formats in Go. I would be very interested in avoiding future tedium by defining future data spec in Kaitai, and letting a code generated codex do the rest of my work.

Is it possible to author a per-language emitter as a plugin from IDL in a native language à la Protocol Buffers?

[GreyCat]

@matttproud Please follow the discussion in this issue. I've asked some questions and I'm still waiting for Go expert opinions on them. It's not like I can move this whole thing alone. Unfortunately, it looks like that we've lost original proposers and experts @5k3105 and @tarm :(

Is it possible to author a per-language emitter as a plugin from IDL in a native language à la Protocol Buffers?

If I understood you correctly, this is already like that. All we need to write to support a certain target language is ThatLanguageNameCompiler class (which is mostly a template that gets filled with values to generate KS output) and ThatLanguageNameTranslator class (which is also more or less the template, but used for expression language).

[cugu]

Usually Unmarshal should be something like:

func Unmarshal(data []byte, ks *KaitaiStruct) error {
	…
}

See protobuf, json or asn

The streaming api could be something like:

byteStream := []byte{0xf8, 0xf9, 0xfa, 0xfb, 0xfc, 0xfd, 0xfe, 0xff}
reader := kaitai.NewReader(byteStream)
dec := kaitai.NewDecoder(reader)

kaitaiRootStruct := FooBar{}

err := dec.Decode(&kaitaiRootStruct)
if err != nil {
	log.Fatal(err)
}

Similar in json

That way the decoder handles the stream and the relations (parent, root) etc. This Go way pretty much contradicts the Kaitai way of storing the stream and relations in every object.

[cugu]

To just add relations (IO, parent, root) to @tarm 's hello world see https://github.com/cugu/kaitai_struct_go/blob/master/hello_world.go and https://github.com/cugu/kaitai_struct_go/blob/master/struct.go

[GreyCat]

@cugu My main question is whether Unmarshall is strictly required, or it is just a convention, and how heavily it is enforced / frowned upon if we'll do it slightly differently? As far as I understand, it's really just a static method that fills in a pre-allocated structure.

The problem is that KS generally is not that simple, and you don't have a luxury of parsing anything in one function and just going in with approach "I'll call this one can and after that I'm guaranteed to have all the fields of structure X filles in".

There is lazy parsing, and it is especially important for "parse instances": it is used to parse multi-terabyte file systems without loading everything into memory at once. Normally, you just generate something like that in languages like Java:

    public IndexEntry index() {
        if (this.index != null)
            return this.index;
        long _pos = this._io.pos();
        this._io.seek(indexOffset());
        this.index = new IndexEntry(this._io, this, _root);
        this._io.seek(_pos);
        return this.index;
    }

Note that it makes heavy use of this._io, i.e. object remembers a reference to a stream from which data should be acquired.

I see that you've proposed to add Metadata member and type to store that information. It will probably be ok with IO (although probably the correct naming is _Metadata in this case, so it won't clash with any existing member named Metadata). How would it handle parent relationships, as usually parent data type is not just "Struct", but inferred in ksy compile time to be something more specific, i.e., for example, how would you compile nav_parent.ksy.

[tarm]

@GreyCat Thanks for all the background and suggestions. It's very helpful. Sorry for being MIA, $DAYJOB has kept me busy. I have been thinking about it though!

Sorry, I have not yet updated the stream api with your comments, but the comments and changes to the stream API seem straightforward.

Go does not have python's __attr__ so the way to make seq consistent with instance is to either have getter accessor methods everywhere or have all fields be flat.

As @cugu mentioned, there is definitely a preference in Go to have data types be flat and add functionality using methods. That is different than the standard KS way. Obviously that does not work for terabyte filesystem parsing.

However, if you have to parse a terabyte filesystem, you could still use the auto generated code but only call the "inner" unmarshal functions and manually write code to jump or iterate through as needed for your business logic.

As a Go user, my preference would be to have the Unmarshal() method and not embed the KS struct into each object.

I've been thinking more about the parent and root references as well. (This suggestions presumes unmarshal). You could pass in the needed leaf fields into Unmarshal() as extra arguments. That means more bookkeeping for the code generator because all the intermediate Unmarshal functions would also need those arguments, but would be easy to understand and satisfy the type checker. I just pushed an example for nav_parent:
https://github.com/tarm/kaitai_struct_tests/blob/master/spec/go/nav_parent.go

Obviously that is also different from the standard KS way as well. Go is statically typed enough to make the parent references a little painful otherwise. Doing it the more typically KS way would be easy with root, but trickier with parent references if a type could have parents of different types. How is that handled in the generated code for other statically typed languages? Is the .ksy format statically typed enough for that to not be a problem?

Finally, in nav_parent, I had to add an int() conversion to go from a uint32 field to a length in ReadBytes(). Should ReadBytes take uint64 as an argument? Does the KS compiler add the appropriate conversions for references like that? Does the KS compiler add overflow checking before conversions?

[GreyCat]

Go does not have python's __attr__ so the way to make seq consistent with instance is to either have getter accessor methods everywhere or have all fields be flat.

Ok, I start to understand that Go is a pretty low-level language, as I go ;) It's not terribly hard to do these distinctions automatically for generated code (i.e. foo.bar.baz being translated to Foo.Bar.Baz or Foo.Bar().Baz or Foo().Bar.Baz() or whatever's needed), but then again, it's not super user-friendly, I guess. Given that getter methods add no overhead, I guess we could probably generate them just in case (or at least start with that approach and tune it afterwards with CLI switches).

As a Go user, my preference would be to have the Unmarshal() method and not embed the KS struct into each object.

It is possible, but it would basically require end-user to keep references to root, parent and io somewhere, and pass it along to every instance attribute call, i.e. Foo.Bar(io, parent, root).Baz(io, parent, root), etc. Also, technically it's possible to implement both styles of code generation and use some CLI switch to choose between them. If are not 100% opposed to it, I'd start with more "traditional" object-oriented approach, i.e. storing these references in generated structs.

How is that handled in the generated code for other statically typed languages? Is the .ksy format statically typed enough for that to not be a problem?

Parent type is inferred in pre-compile-time in ksc, and is known since then. For example, in Java, nav_parent.ksy is compiled to:

public class NavParent extends KaitaiStruct {
    public NavParent(KaitaiStream _io, KaitaiStruct _parent, NavParent _root) { /* ... */  }
    public static class HeaderObj extends KaitaiStruct {
        public HeaderObj(KaitaiStream _io, NavParent _parent, NavParent _root) { /* ... */ }
    }
    public static class IndexObj extends KaitaiStruct {
        public IndexObj(KaitaiStream _io, NavParent _parent, NavParent _root) { /* ... */ }
    }
    public static class Entry extends KaitaiStruct {
        public Entry(KaitaiStream _io, IndexObj _parent, NavParent _root) { /* ... */ }
    }
}

Note that both _parent and _root types vary in all these classes, and in most cases it is not KaitaiStruct, but some more specific class that inherits from KaitaiStruct.

Finally, in nav_parent, I had to add an int() conversion to go from a uint32 field to a length in ReadBytes(). Should ReadBytes take uint64 as an argument? Does the KS compiler add the appropriate conversions for references like that? Does the KS compiler add overflow checking before conversions?

By design decision, KS expression language assumes that there's only a single one-size-fits-all integer type in a language (which is not that far from being true in many cases) and doesn't track complex integer type implicit casts, which are very different in various languages.

Implementation of stuff like ReadBytes() thus can be one of:

• automated implicit conversions (if they're supported in a language)
• doing several implementations (like readBytes(int n) + readBytes(long n), etc)
• doing conversion on call (basically, as you did with int())

There's no overflow checking everywhere, and this is almost mostly by design. Current approachin vast majority of existing situations and we don't really aim to implement 100% compliant exact emulator of some particular language in all other languages. We just say that it's "undefined behavior" (which is pretty much the standard excuse in C/C++ standard, so we're not alone here either).

[cugu]

I fixed the functions in kaitai-io/kaitai_struct_go_runtime#5

I would prefer to stay with byte. It is used in standard libraries as well. See https://golang.org/pkg/io/ and https://golang.org/pkg/bytes/

[GreyCat]

Would you want to use byte all around the generated code instead of uint8 then as well?

[cugu]

I guess we can leave the remaining ones, as they are used as numbers, not as data bytes. If this makes sense to you.

[GreyCat]

Currently, GoCompiler generates []uint8 type for byte arrays. I don't really like the idea of that distinction like "used as numbers" vs "used as data bytes": ultimately, it's all numbers, reinterpreted as needed in various contexts. It would be very hard to decide which of these 8-bit integers is a "true byte" vs "true uint8", not to mention that no other language does that distinction.

But, generally, I don't have a strong opinion on this one, this is obviously more of a taste thing.

[cugu]

I would personally keep it as it is. Another reference: https://golang.org/pkg/builtin/#byte

A suggestions for switches: https://gist.github.com/cugu/8a724fbd91f4a23d6189be7c75c9d4a3

[dgryski]

byte is definitely more common than uint8 in most Go code.

[GreyCat]

Basical enum support is in, CI completion rating is up to 32%.

Note that I had to add namespacing prefixes to both "enum" type names and to individual constants, resulting in something like this enum_0.go.

[GreyCat]

Applied some more assorted fixes, CI rating is up to 39%.

@cugu I think that bit-level reading generation is now correct and ready to be tested, but it turned out that runtime implementation is very strange and it just does an infinite loop. Could you take a look?

[cugu]

@GreyCat Nice! And here is a patch for the bit read: kaitai-io/kaitai_struct_go_runtime#8

[GreyCat]

@cugu JFYI, latest changes in runtime broke it. Namely, here pos is unused and thus causes compile-time error:

 func (k *Stream) ReadBytesTerm(term byte, includeTerm, consumeTerm, eosError bool) ([]byte, error) {
        r := bufio.NewReader(k)
-       pos, err := k.Pos()
-       if err != nil {
-               return nil, err
-       }
        slice, err := r.ReadBytes(term)

However, if I'll just delete it, it still causes panic during the tests:

github.com/kaitai-io/kaitai_struct_go_runtime/kaitai.(*Stream).ReadBytes(0xc420112a40, 0xfffffffffffffffe, 0x0, 0x0, 0xc420090cd0, 0x9, 0x0)
        kaitai_struct/tests/compiled/go/src/github.com/kaitai-io/kaitai_struct_go_runtime/kaitai/stream.go:205 +0x40
test_formats.(*ExprIoPos_AllPlusNumber).Read(0xc420094aa0, 0xc420112a40, 0xc42007a300, 0xc42007a300, 0x10, 0x0)
        kaitai_struct/tests/compiled/go/src/test_formats/expr_io_pos.go:78 +0x14b
test_formats.(*ExprIoPos).Read(0xc42007a300, 0xc420112a20, 0x6c5240, 0xc42007a300, 0xc42007a300, 0x2d2, 0x4cc980)
        kaitai_struct/tests/compiled/go/src/test_formats/expr_io_pos.go:32 +0x1e6
_kaitai_struct/tests/spec/go.TestExprIoPos(0xc4201370e0)

[cugu]

That line is gone already: https://github.com/kaitai-io/kaitai_struct_go_runtime/blob/master/kaitai/stream.go#L242

[GreyCat]

Ok, but it still results in panic, and thus produces no test report.

[GreyCat]

Done a few more passes, Go rating is up to 41%. Major things to do/fix left:

• _index
• Parameteric types
• Processing
• Proper invocation & implementation of terminated/padded byte arrays & strings
• Type switching
• Casts

Anyone wants to take on any of this list?

[13rac1]

Ok, but it still results in panic, and thus produces no test report.

kaitai-io/kaitai_struct_tests#47 This allows panicking tests.

[GreyCat]

Just a little update: thanks to @jchv contributions, Go support rating is 41% => 53%.

[tomberek]

Can Go be added to https://kaitai.io/repl/index.html ?
It would make it pretty easy to test with simple file formats and track progress.

(perhaps with a warning that it is not 100% yet)

[generalmimon]

@tomberek Please take a look https://ide.kaitai.io/devel/. It's the development version of the awesome Kaitai Web IDE. which is currently IMHO the best way to write and debug the KSY files. It has a number of useful features which really simplify the development. And it is the development version, which means that it gets updated every time someone pushes a commit to the https://github.com/kaitai-io/kaitai_struct_compiler repo, so you can be sure that you're always using the latest version without making any effort.

The Web IDE supports generating the parser to an arbitrary target language, but I see there isn't Go option as well. But I can add it if you want, it should be relatively easy.

The REPL tool is no longer actively supported at the moment. It has compiler of version 0.7, but the latest is 0.9 (it isn't yet released, but hopefully will be soon), so it's not supporting a number of useful features introduced since then (and I think most formats from the format gallery use some of them, so you can't compile it here). And the plain textarea is inconvenient for editing as well. So I strongly suggest using the mentioned Web IDE, which is much more usable.

The best would be to delete the REPL completely, because it's just confusing the new users, when they get there via the call-to-action link Try it from the homepage. I think it would make more sense to link to the Web IDE instead. What do you think, @GreyCat?

[generalmimon]

@tomberek:

It would make it pretty easy to test with simple file formats and track progress.

And if you want to track the development progress in Go, you can watch our CI (Continuous Integration) dashboard. There are the results of all tests which were created over the time. Each test consists of a test KSY format (in this folder), which uses a special feature or tests a known bug. All formats are compiled to all target languages using the latest compiler. The generated parsers are used on some prepared binary file and the parsed data structure is checked with unit test assertions (in each language separately).

In case you are interested, here is the detailed description of the CI infrastructure.

[mewmew]

@GreyCat really wonderful work on starting to implement Go support in Kaitai Struct!

How far along is it currently?

I had a look at the CI dashboard that @generalmimon posted (thanks @generalmimon!) and it seems Go it not yet listed here.

Cheers,
Robin

[generalmimon]

@mewmew:

I had a look at the CI dashboard that @generalmimon posted (thanks @generalmimon!) and it seems Go it not yet listed here.

Sorry for the confusion. That's my fault, yesterday I ported test ExprCalcArrayOps to Go and apparently the generated parser (see https://github.com/kaitai-io/ci_targets/blob/master/compiled/go/src/test_formats/expr_calc_array_ops.go) fails to build with the Go compiler. And it seems that this failure broke the whole build, so no results were collected.

For testing we have partial builders in some languages, which are able to recover even if one test is invalid. Some efforts to restore the build in case of failure can be seen also in the builder for Go, but it didn't work in this case for some reason. I'll revert the change so you can look at the results, until this issue is resolved.

[tomberek]

I can say that the Go generator works well enough for simple cases. My use case is a binary format of simple graph structure. Thanks for the great work!

[mewmew]

@generalmimon, thanks for the explanation and on working on the builders! Now Go seems to be up and running again, with a rating of 52.1%

Great job!

[mewmew]

Imports

Does the Go version of Kaitai support imports yet? It seem partially supported. However, it is possible to run into issue with an incorrect root type, which causes build failures. Details provided below.

I found an issue when using imports with the generated Go parser when using imports in *.ksy files.

Example taken from the Kaitai Struct User Guide (https://doc.kaitai.io/user_guide.html#meta-imports):

Contents of date.ksy:

meta:
  id: date
seq:
  - id: year
    type: u2le
  - id: month
    type: u2le
  - id: day
    type: u2le

Contents of filelist.ksy:

meta:
  id: filelist
  # this will import "date.ksy"
  imports:
    - date
seq:
  - id: entries
    type: entry
    repeat: eos
types:
  entry:
    seq:
      - id: filename
        type: strz
        encoding: ASCII
      # just use "date" type from date.ksy as if it was declared in
      # current file
      - id: timestamp
        type: date
      # you can access its members too!
      - id: historical_data
        size: 160
        if: timestamp.year < 1970

Commands used to compile:

$ kaitai-struct-compiler -t go --go-package foo date.ksy
$ kaitai-struct-compiler -t go --go-package foo filelist.ksy

If we now try to compile the generated Go package, we get the following error:

$ go install ./foo
# foo
foo/filelist.go:56:38: cannot use this._root (type *Filelist) as type *Date in argument to tmp4.Read

The contents of the generated Go source files are as follows.

Contents of date.go:

// This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild

package foo

import "github.com/kaitai-io/kaitai_struct_go_runtime/kaitai"

type Date struct {
	Year uint16
	Month uint16
	Day uint16
	_io *kaitai.Stream
	_root *Date
	_parent interface{}
}

func (this *Date) Read(io *kaitai.Stream, parent interface{}, root *Date) (err error) {
	this._io = io
	this._parent = parent
	this._root = root

	tmp1, err := this._io.ReadU2le()
	if err != nil {
		return err
	}
	this.Year = uint16(tmp1)
	tmp2, err := this._io.ReadU2le()
	if err != nil {
		return err
	}
	this.Month = uint16(tmp2)
	tmp3, err := this._io.ReadU2le()
	if err != nil {
		return err
	}
	this.Day = uint16(tmp3)
	return err
}

Contents of filelist.go:

// This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild

package foo

import "github.com/kaitai-io/kaitai_struct_go_runtime/kaitai"

type Filelist struct {
	Entries []*Filelist_Entry
	_io *kaitai.Stream
	_root *Filelist
	_parent interface{}
}

func (this *Filelist) Read(io *kaitai.Stream, parent interface{}, root *Filelist) (err error) {
	this._io = io
	this._parent = parent
	this._root = root

	for {
		tmp1, err := this._io.EOF()
		if err != nil {
			return err
		}
		if tmp1 {
			break
		}
		tmp2 := new(Filelist_Entry)
		err = tmp2.Read(this._io, this, this._root)
		if err != nil {
			return err
		}
		this.Entries = append(this.Entries, tmp2)
	}
	return err
}
type Filelist_Entry struct {
	Filename string
	Timestamp *Date
	HistoricalData []byte
	_io *kaitai.Stream
	_root *Filelist
	_parent *Filelist
}

func (this *Filelist_Entry) Read(io *kaitai.Stream, parent *Filelist, root *Filelist) (err error) {
	this._io = io
	this._parent = parent
	this._root = root

	tmp3, err := this._io.ReadBytesTerm(0, false, true, true)
	if err != nil {
		return err
	}
	this.Filename = string(tmp3)
	tmp4 := new(Date)
	err = tmp4.Read(this._io, this, this._root)
	if err != nil {
		return err
	}
	this.Timestamp = tmp4
	if (this.Timestamp.Year < 1970) {
		tmp5, err := this._io.ReadBytes(int(160))
		if err != nil {
			return err
		}
		tmp5 = tmp5
		this.HistoricalData = tmp5
	}
	return err
}

Change to fix build

u@x220 ~/D/g/s/g/s/d/f/foo> git diff
diff --git a/foo/date.go b/foo/date.go
index 8ab7edc..66c0b28 100644
--- a/foo/date.go
+++ b/foo/date.go
@@ -9,11 +9,11 @@ type Date struct {
        Month   uint16
        Day     uint16
        _io     *kaitai.Stream
-       _root   *Date
+       _root   interface{}
        _parent interface{}
 }
 
-func (this *Date) Read(io *kaitai.Stream, parent interface{}, root *Date) (err error) {
+func (this *Date) Read(io *kaitai.Stream, parent interface{}, root interface{}) (err error) {
        this._io = io
        this._parent = parent
        this._root = root
diff --git a/foo/filelist.go b/foo/filelist.go
index d510127..453d44f 100644
--- a/foo/filelist.go
+++ b/foo/filelist.go
@@ -7,11 +7,11 @@ import "github.com/kaitai-io/kaitai_struct_go_runtime/kaitai"
 type Filelist struct {
        Entries []*Filelist_Entry
        _io     *kaitai.Stream
-       _root   *Filelist
+       _root   interface{}
        _parent interface{}
 }
 
-func (this *Filelist) Read(io *kaitai.Stream, parent interface{}, root *Filelist) (err error) {
+func (this *Filelist) Read(io *kaitai.Stream, parent interface{}, root interface{}) (err error) {
        this._io = io
        this._parent = parent
        this._root = root
@@ -39,11 +39,11 @@ type Filelist_Entry struct {
        Timestamp      *Date
        HistoricalData []byte
        _io            *kaitai.Stream
-       _root          *Filelist
+       _root          interface{}
        _parent        *Filelist
 }
 
-func (this *Filelist_Entry) Read(io *kaitai.Stream, parent *Filelist, root *Filelist) (err error) {
+func (this *Filelist_Entry) Read(io *kaitai.Stream, parent *Filelist, root interface{}) (err error) {
        this._io = io
        this._parent = parent
        this._root = root

Perhaps there is a better solution than using interface{}? At least the above change fixes the build.

[mewmew]

Parametric types

I tried to use parametric types today, but it seems that at least part of the Go implementation is not yet complete. The parameter uses are generated properly, but the instantiation and definition of the type parameters are now.

An example taken from the Kaitai Struct User Guide follows below (https://doc.kaitai.io/user_guide.html#param-types):

Contents of params.ksy:

meta:
  id: params
  endian: le

types:
 kv_pair:
   params:
     - id: len_key
       type: u2
   seq:
     - id: key
       size: len_key
       type: str
       encoding: ASCII
     - id: value
       type: strz
       encoding: ASCII

seq:
 - id: short_pairs
   type: kv_pair(3)
   repeat: expr
   repeat-expr: 0x100
 - id: long_pairs
   type: kv_pair(8)
   repeat: expr
   repeat-expr: 0x100

Commands used to compile:

$ kaitai-struct-compiler -t go --go-package p params.ksy

If we now try to compile the generated Go package, we get the following error:

$ go install ./p
# p
./params.go:53:42: this.LenKey undefined (type *Params_KvPair has no field or method LenKey)

The contents of the generated Go source files are as follows.

Contents of params.go:

// This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild

package p

import "github.com/kaitai-io/kaitai_struct_go_runtime/kaitai"

type Params struct {
    ShortPairs []*Params_KvPair
    LongPairs []*Params_KvPair
    _io *kaitai.Stream
    _root *Params
    _parent interface{}
}

func (this *Params) Read(io *kaitai.Stream, parent interface{}, root *Params) (err error) {
    this._io = io
    this._parent = parent
    this._root = root

    this.ShortPairs = make([]*Params_KvPair, 256)
    for i := range this.ShortPairs {
        tmp1 := new(Params_KvPair)
        err = tmp1.Read(this._io, this, this._root)
        if err != nil {
            return err
        }
        this.ShortPairs[i] = tmp1
    }
    this.LongPairs = make([]*Params_KvPair, 256)
    for i := range this.LongPairs {
        tmp2 := new(Params_KvPair)
        err = tmp2.Read(this._io, this, this._root)
        if err != nil {
            return err
        }
        this.LongPairs[i] = tmp2
    }
    return err
}
type Params_KvPair struct {
    Key string
    Value string
    _io *kaitai.Stream
    _root *Params
    _parent *Params
}

func (this *Params_KvPair) Read(io *kaitai.Stream, parent *Params, root *Params) (err error) {
    this._io = io
    this._parent = parent
    this._root = root

    tmp3, err := this._io.ReadBytes(int(this.LenKey))
    if err != nil {
        return err
    }
    tmp3 = tmp3
    this.Key = string(tmp3)
    tmp4, err := this._io.ReadBytesTerm(0, false, true, true)
    if err != nil {
        return err
    }
    this.Value = string(tmp4)
    return err
}

Suggested solution

When generating parametric types, also generate a constructor function taking said parameters. E.g. for the example above.

u@x220 ~/D/g/s/g/s/d/f/p> diff -u params.go.orig params.go
--- params.go.orig  2020-02-21 15:23:40.420027468 +0100
+++ params.go   2020-02-21 15:25:44.633361203 +0100
@@ -38,6 +38,7 @@
    return err
 }
 type Params_KvPair struct {
+   LenKey int // added parameter type.
    Key string
    Value string
    _io *kaitai.Stream
@@ -45,6 +46,12 @@
    _parent *Params
 }
 
+func NewParams_KVPair(lenKey int) *Params_KvPair {
+   return &Params_KvPair{
+       LenKey: lenKey, // parameter type set when invoking constructor function.
+   }
+}
+
 func (this *Params_KvPair) Read(io *kaitai.Stream, parent *Params, root *Params) (err error) {
    this._io = io
    this._parent = parent