Goavro is a library written in Go that supports translating binary and textual Avro data to Go native data types, and conversely translating Go native data types to binary or textual Avro data. It encodes by appending to an existing or empty Go byte slice, and decodes by consuming bytes from an existing Go byte slice.
A goavro Codec is created as a stateless structure that can be
safely used in multiple go routines simultaneously.
With the exeption of features not yet supported, goavro attempts to be fully compliant with the most recent version of the Avro specification.
Documentation is available via
Also please see the example programs in the examples directory for
reference.
package main
import (
"fmt"
"github.com/karrick/goavro"
)
func main() {
codec, err := goavro.NewCodec(`
{
"type": "record",
"name": "LongList",
"fields" : [
{"name": "next", "type": ["null", "LongList"], "default": null}
]
}
`)
if err != nil {
fmt.Println(err)
}
// NOTE: May omit fields when using default value
textual := []byte(`{"next":{"LongList":{}}}`)
// Convert textual Avro data (in Avro JSON format) to native Go form
native, _, err := codec.NativeFromText(textual)
if err != nil {
fmt.Println(err)
}
// Convert native Go form to binary Avro data
binary, err := codec.BinaryFromNative(nil, native)
if err != nil {
fmt.Println(err)
}
// Convert binary Avro data back to native Go form
native, _, err = codec.NativeFromBinary(binary)
if err != nil {
fmt.Println(err)
}
// Convert native Go form to textual Avro data
textual, err = codec.TextFromNative(nil, native)
if err != nil {
fmt.Println(err)
}
// NOTE: Textual encoding will show all fields, even those with values that
// match their default values
fmt.Println(string(textual))
// Output: {"next":{"LongList":{"next":null}}}
}A Codec provides four methods for translating between a byte slice
of either binary or textual Avro data and native Go data.
The following methods convert data between native Go data and byte slices of the binary Avro representation:
BinaryFromNative
NativeFromBinary
The following methods convert data between native Go data and byte slices of the textual Avro representation:
NativeFromText
TextFromNative
Each Codec also exposes the Schema method to return a simplified
version of the JSON schema string used to create the Codec.
Goavro does not use Go's structure tags to translate data between native Go types and Avro encoded data.
When translating from either binary or textual Avro to native Go data,
goavro returns primitive Go data values for corresponding Avro data
values. That is, a Go nil is returned for an Avro null; a Go
bool for an Avro boolean; a Go []byte for an Avro bytes; a Go
float32 for an Avro float, a Go float64 for an Avro double; a
Go int64 for an Avro long; a Go int32 for an Avro int; and a
Go string for an Avro string.
For complex Avro data types, a Go []interface{} is returned for an
Avro array; a Go string for an Avro enum; a Go []byte for an
Avro fixed; a Go map[string]interface{} for an Avro map and
record.
Because of encoding rules for Avro unions, when an union's value is
null, a simple Go nil is returned. However when an union's value
is non-nil, a Go map[string]interface{} with a single key is
returned for the union. The map's single key is the Avro type name and
its value is the datum's value.
Goavro does not use Go's structure tags to translate data between native Go types and Avro encoded data.
When translating from native Go to either binary or textual Avro data,
goavro generally requires the same native Go data types as the decoder
would provide, with some exceptions for programmer convenience. Goavro
will accept any numerical data type provided there is no precision
lost when encoding the value. For instance, providing float64(3.0)
to an encoder expecting an Avro int would succeed, while sending
float64(3.5) to the same encoder would return an error.
When providing a slice of items for an encoder, the encoder will
accept either []interface{}, or any slice of the required type. For
instance, when the Avro schema specifies:
{"type":"array","items":"string"}, the encoder will accept either
[]interface{}, or []string. If given []int, the encoder will
return an error when it attempts to encode the first non-string array
value using the string encoder.
When providing a value for an Avro union, the encoder will accept
nil for a null value. If the value is non-nil, it must be a
map[string]interface{} with a single key-value pair, where the key
is the Avro type name and the value is the datum's value. As a
convenience, the Union function wraps any datum value in a map as
specified above.
func ExampleUnion() {
codec, err := goavro.NewCodec(`["null","string","int"]`)
if err != nil {
fmt.Println(err)
}
buf, err := codec.TextFromNative(nil, goavro.Union("string", "some string"))
if err != nil {
fmt.Println(err)
}
fmt.Println(string(buf))
// Output: {"string":"some string"}
}In general it is poor form to define a library API which shares the same function or method names but provides a different method signature to an accepted standard. Go has particular strong emphasis on what a Reader and Writer are, and they conflict with what the Avro specification describes as a reader and a writer.
In Go, an io.Reader reads data from the stream specified at object
instantiation time into a preallocated slice of bytes and returns both
the number of bytes read along with an error. In the Avro
specification, a reader is a function that decodes Avro data and
returns data in native form.
A Go io.Writer writes bytes from a slice of bytes to a stream
specified at its instantiation time and returns the number of bytes
written along with an error. In the Avro specification, a writer is a
function that encodes data from native form to either binary or text
Avro bytes.
The Avro specification allows for providing default values for each Avro Record field. The default value is to be used when reading instances that lack the respective field.
When reading binary Avro data, a Record is decoded by reading bytes for the first Record field, immediately followed by the second Record field, and so on. No fields may be skipped in a Record's binary encoding, so a default value is deemed unusable. If this assessment is wrong, please open a Bug, and provide one or more suitable examples, and the developers will be happy to revisit the issue.
When decoding from textual Avro data that is missing a particular record field name, if the record field has a default value, it will be used in place of the missing value.
When encoding from native Go data that is missing a particular record field name, if the record field has a default value, it will be used in place of the missing value.
With the exeption of features not yet supported, goavro attempts to be fully compliant with the most recent version of the Avro specification.
To prevent over allocation of memory when decoding OCF data, goavro
returns an error whenever an OCF block count exceeds MaxBlockCount, or
a block size exceeds MaxBlockSize. Both of these tokens are set to
math.MaxInt32, or ~2.2 GiB, but are declared as variables so a user
can change the limit if deemed necessary.
The Avro specification allows an implementation to optionally map a writer's schema to a reader's schema using aliases. Although goavro can compile schemas with aliases, it does not implement this feature.
The Avro specification describes the process by which schemas are
canonlicalized. Goavro does not canonicalize schema strings when
creating a Codec, although it does eliminate extra whitespace.
Goavro does not implement Logical Types as required by the Avro specification.
Kafka is the reason goavro was written. Similar to Avro Object Container Files being a layer of abstraction above Avro Data Serialization format, Kafka's use of Avro is a layer of abstraction that also sits above Avro Data Serialization format, but has its own schema.
Goavro does not implement any high level RPC mechanics required by the Avro specification. Avro protocol declarations, messages, message transports, message framing, handshakes, and call format are all unsupported by this library.
The Avro specification allows for providing a sory order string,
either ascending, descending, or ignore, for use when sorting
records. While goavro can create Codec instances that specify
order, those values are not used.
The Avro specification allows for providing a JSON array of strings as
alternate names for a Record field. While goavro can create Codec
instances that specify aliases, that list is ignored.