diff --git a/README.md b/README.md index b02c973..03b2b28 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,14 @@ # KARMEM -Karmem is a fast binary serialization format. It was designed to be easy to use and as fast as possible, optimized to take Golang and TinyGo's maximum performance while reaming easy to use and efficient for repeatable reads. Under tests, Karmem is ten times faster than Google Flatbuffers, with bounds-checks included. +Karmem is a fast binary serialization format. The priority of Karmem is to be +easy to use while been fast as possible. It's optimized to take Golang and +TinyGo's maximum performance and is efficient for repeatable reads, reading +different content of the same type. Karmem has proven to be ten times faster +than Google Flatbuffers, with the additional overhead of bounds-checking +included. -> ⚠️ Karmem still under development, the API is not stable. However, serialization-format itself is unlike to change and should remain backward compatible with older versions. +> ⚠️ Karmem still under development, the API is not stable. However, serialization-format itself is unlike to change and +> should remain backward compatible with older versions. # Contents @@ -20,9 +26,19 @@ Karmem is a fast binary serialization format. It was designed to be easy to use # Motivation -Karmem was create to solve one single issue: make easy to transfer data between WebAssembly host and guest. While still portable for non-WebAssembly languages. We are experimenting with an "event-command pattern" between wasm-host and wasm-guest in one project, but sharing data is very expensive, and FFI calls are not cheap either. Karmem encodes once and shares the same content with multiple guests, regardless of the language, making it very efficient. Also, even using Object-API to decode, it's fast enough, and Karmem was designed to take advantage of that pattern, avoid allocations, and re-use the same struct for multiple data. +Karmem was create to solve one single issue: make easy to transfer data between WebAssembly host and guest. While still +portable for non-WebAssembly languages. We are experimenting with an "event-command pattern" between wasm-host and +wasm-guest in one project, but sharing data is very expensive, and FFI calls are not cheap either. Karmem encodes once +and shares the same content with multiple guests, regardless of the language, making it very efficient. Also, even using +Object-API to decode, it's fast enough, and Karmem was designed to take advantage of that pattern, avoid allocations, +and re-use the same struct for multiple data. -Why not use [Witx](https://github.com/jedisct1/witx-codegen)? It is good project and aimed to WASM, however it seems more complex and defines not just data-structure, but functions, which I'm trying to avoid. Also, it is not intended to be portable to non-wasm. Why not use [Flatbuffers](https://google.github.io/flatbuffers/)? We tried, but it's not fast enough and also causes panics due to the lack of bound-checking. Why not use [Cap'n'Proto](https://capnproto.org/)? It's a good alternative but lacks implementation for Zig and AssemblyScript, which is top-priority, it also have more allocations and the generated API is harder to use, compared than Karmem. +Why not use [Witx](https://github.com/jedisct1/witx-codegen)? It is good project and aimed to WASM, however it seems +more complex and defines not just data-structure, but functions, which I'm trying to avoid. Also, it is not intended to +be portable to non-wasm. Why not use [Flatbuffers](https://google.github.io/flatbuffers/)? We tried, but it's not fast +enough and also causes panics due to the lack of bound-checking. Why not use [Cap'n'Proto](https://capnproto.org/)? It's +a good alternative but lacks implementation for Zig and AssemblyScript, which is top-priority, it also have more +allocations and the generated API is harder to use, compared than Karmem. # Usage @@ -55,7 +71,9 @@ struct AccountData table { Generate the code using `go run karmem.org/cmd/karmem build --golang -o "km" app.km`. ### Encoding + In order to encode, use should create an native struct and then encode it. + ``` var writerPool = sync.Pool{New: func() any { return karmem.NewWriter(1024) }} @@ -97,7 +115,9 @@ func main() { ``` ### Reading -Instead of decoding it to another struct, you can read some fields directly, without any additional decoding. In this example, we only need the username of each profile. + +Instead of decoding it to another struct, you can read some fields directly, without any additional decoding. In this +example, we only need the username of each profile. ``` func decodes(encoded []byte) { @@ -115,7 +135,8 @@ Notice, we use `NewAccountDataViewer`, any `Viewer` is just a Viewer, and doesn' ### Decoding -You can also decode it to an existent struct. In some cases, it's better if you re-use the same struct for multiples reads. +You can also decode it to an existent struct. In some cases, it's better if you re-use the same struct for multiples +reads. ``` var accountPool = sync.Pool{New: func() any { return new(app.AccountData) }} @@ -133,7 +154,6 @@ func decodes(encoded []byte) { } ``` - # Benchmark ### Flatbuffers vs Karmem @@ -141,6 +161,7 @@ func decodes(encoded []byte) { Using similar schema with Flatbuffers and Karmem. Karmem is almost 10 times faster than Google Flatbuffers. **Native (MacOS/ARM64 - M1):** + ``` name flatbuffers/op karmem/op delta EncodeObjectAPI-8 1.46ms ± 0% 0.32ms ± 0% -78.22% (p=0.008 n=5+5) @@ -157,7 +178,9 @@ EncodeObjectAPI-8 1.00k ± 0% 0.00k -100.00% (p=0.008 n=5+5) DecodeObjectAPI-8 108k ± 0% 1k ± 0% -99.07% (p=0.008 n=5+5) DecodeSumVec3-8 0.00 0.00 ~ (all equal) ``` + **WebAssembly on Wazero (MacOS/ARM64 - M1):** + ``` name flatbuffers/op karmem/op delta EncodeObjectAPI-8 10.1ms ± 0% 2.5ms ± 0% -75.27% (p=0.016 n=4+5) @@ -176,9 +199,12 @@ DecodeSumVec3-8 5.00 ± 0% 5.00 ± 0% ~ (all equal) ``` ### Raw-Struct vs Karmem -The performance is nearly the same when comparing reading non-serialized data from a native struct and reading it from a karmem-serialized data. + +The performance is nearly the same when comparing reading non-serialized data from a native struct and reading it from a +karmem-serialized data. **Native (MacOS/ARM64 - M1):** + ``` name old time/op new time/op delta DecodeSumVec3-8 93.7µs ± 0% 98.8µs ± 1% +5.38% (p=0.008 n=5+5) @@ -189,26 +215,28 @@ DecodeSumVec3-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta DecodeSumVec3-8 0.00 0.00 ~ (all equal) ``` + ### Karmem vs Karmem That is an comparison with all supported languages. **WebAssembly on Wazero (MacOS/ARM64 - M1):** + ``` -name \ time/op result/wasi-go-km.out result/wasi-as-km.out result/wasi-zig-km.out result/wasi-swift.out -DecodeSumVec3-8 470µs ± 0% 932µs ± 0% 231µs ± 0% 99575µs ± 5% -DecodeObjectAPI-8 1.19ms ± 0% 3.70ms ± 0% 0.61ms ± 0% 77.94ms ±17% -EncodeObjectAPI-8 2.50ms ± 0% 2.95ms ± 0% 0.71ms ± 0% 39.26ms ±14% - -name \ alloc/op result/wasi-go-km.out result/wasi-as-km.out result/wasi-zig-km.out result/wasi-swift.out -DecodeSumVec3-8 1.25kB ± 0% 12.73kB ± 0% 1.25kB ± 0% 2.99kB ± 0% -DecodeObjectAPI-8 11.8kB ± 0% 74.2kB ± 0% 161.5kB ± 0% 288.7kB ± 3% -EncodeObjectAPI-8 3.02kB ± 0% 38.37kB ± 1% 1.23kB ± 0% 2.98kB ± 0% - -name \ allocs/op result/wasi-go-km.out result/wasi-as-km.out result/wasi-zig-km.out result/wasi-swift.out -DecodeSumVec3-8 5.00 ± 0% 5.00 ± 0% 5.00 ± 0% 35.00 ± 0% -DecodeObjectAPI-8 5.00 ± 0% 4.00 ± 0% 4.00 ± 0% 35.00 ± 0% -EncodeObjectAPI-8 4.00 ± 0% 3.00 ± 0% 3.00 ± 0% 33.00 ± 0% +name \ time/op result/wasi-go-km.out result/wasi-as-km.out result/wasi-zig-km.out result/wasi-c-km.out result/wasi-swift-km.out +DecodeSumVec3-8 470µs ± 0% 932µs ± 0% 231µs ± 0% 230µs ± 0% 97822µs ± 5% +DecodeObjectAPI-8 1.19ms ± 0% 3.70ms ± 0% 0.62ms ± 0% 0.56ms ± 0% 74.72ms ± 4% +EncodeObjectAPI-8 2.52ms ± 0% 2.98ms ± 2% 0.71ms ± 0% 0.67ms ± 0% 42.45ms ± 7% + +name \ alloc/op result/wasi-go-km.out result/wasi-as-km.out result/wasi-zig-km.out result/wasi-c-km.out result/wasi-swift-km.out +DecodeSumVec3-8 1.25kB ± 0% 12.72kB ± 0% 1.25kB ± 0% 1.25kB ± 0% 2.99kB ± 0% +DecodeObjectAPI-8 11.9kB ± 1% 74.2kB ± 0% 164.3kB ± 0% 1.2kB ± 0% 291.7kB ± 3% +EncodeObjectAPI-8 3.02kB ± 0% 38.38kB ± 0% 1.23kB ± 0% 1.23kB ± 0% 2.98kB ± 0% + +name \ allocs/op result/wasi-go-km.out result/wasi-as-km.out result/wasi-zig-km.out result/wasi-c-km.out result/wasi-swift-km.out +DecodeSumVec3-8 5.00 ± 0% 5.00 ± 0% 5.00 ± 0% 5.00 ± 0% 35.00 ± 0% +DecodeObjectAPI-8 5.00 ± 0% 4.00 ± 0% 4.00 ± 0% 4.00 ± 0% 35.00 ± 0% +EncodeObjectAPI-8 4.00 ± 0% 3.00 ± 0% 3.00 ± 0% 3.00 ± 0% 33.00 ± 0% ``` # Languages @@ -219,23 +247,24 @@ Currently, we have focus on WebAssembly, and because of that those are the langu - Golang/TinyGo - ~~Swift/SwiftWasm~~ - Zig +- C ### Features -| Features | Golang | Zig | AssemblyScript | Swift | -|--|-- | -- | --| -- | -| Performance | Good | Excellent | Good | Horrible | -| Priority | High | High | High | Low | -| **Encoding** | | | | | -| Object Encoding | ✔️ |✔️ |✔️ | ✔️ | -| Raw Encoding | ❌ |❌ | ❌| ❌ | -| Zero-Copy |❌ | ❌ |❌ | ❌ | -| **Decoding** | | | | | -| Object Decoding |✔️ |✔️ |✔️ | ✔️ | -| Object Re-Use |✔️ |✔️ |✔️ | ❌ | -| Random-Access |✔️ |✔️ |✔️ | ✔️ | -| Zero-Copy |✔️ | ✔️ |✔️ | ❌ | -| Native Array | ✔️ |✔️ |❌ | ❌ | +| Features | Golang | Zig | AssemblyScript | Swift | C | +|--|-- | -- | --| -- | -- | +| Performance | Good | Excellent | Good | Horrible | Excellent | +| Priority | High | High | High | Low | High | +| **Encoding** | | | | | | +| Object Encoding | ✔️ |✔️ |✔️ | ✔️ | ✔️| +| Raw Encoding | ❌ |❌ | ❌| ❌ | ❌ | +| Zero-Copy |❌ | ❌ |❌ | ❌ | ❌ | +| **Decoding** | | | | | | +| Object Decoding |✔️ |✔️ |✔️ | ✔️ |✔️ | +| Object Re-Use |✔️ |✔️ |✔️ | ❌ |❌ | +| Random-Access |✔️ |✔️ |✔️ | ✔️ |✔️ | +| Zero-Copy |✔️ | ✔️ |✔️ | ❌ |❌ | +| Native Array | ✔️ |✔️ |❌ | ❌ |❌ | # Schema @@ -276,6 +305,7 @@ struct State table { Monsters [<2000]Monster; } ``` + ### Header: Every file must begin with: `karmem {name};`, other optional options can be defined, as shown above. @@ -283,6 +313,7 @@ Every file must begin with: `karmem {name};`, other optional options can be defi ### Types: **Primitives**: + - Unsigned Integers: `uint8`, `uint16`, `uint32`, `uint64` - Signed Integers: @@ -297,6 +328,7 @@ Every file must begin with: `karmem {name};`, other optional options can be defi It's not possible to defined optional or nullable types. **Arrays**: + - Fixed: `[{Length}]{Type}` (example: `[123]uint16`, `[3]float32`) - Dynamic: @@ -304,14 +336,18 @@ It's not possible to defined optional or nullable types. - Limited: `[<{Length}]{Type}` (example: `[<512]float64`, `[<42]byte`) -It's not possible to have slice of tables or slices of enums or slice of slices. However, it's possible to wrap those types inside one inline-struct. +It's not possible to have slice of tables or slices of enums or slice of slices. However, it's possible to wrap those +types inside one inline-struct. ### Struct: Currently, Karmem has two structs types: inline and table. **Inline:** -Inline structs, as the name suggests, are inlined when used. That reduces the size and may improve the performance. However, it can't have their definition changed. In order words: you can't edit the description of one inline struct without breaking compatibility. +Inline structs, as the name suggests, are inlined when used. That reduces the size and may improve the performance. +However, it can't have their definition changed. In order words: you can't edit the description of one inline struct +without breaking compatibility. + ``` struct Vec3 inline { X float32; @@ -319,10 +355,14 @@ Y float32; Z float32; } ``` -*That struct is exactly the same of `[3]float32` and will have the same serialization result. Because of that, any change of this struct (for instance, change it to `float64` or adding new fields) will break the compatibility.* + +*That struct is exactly the same of `[3]float32` and will have the same serialization result. Because of that, any +change of this struct (for instance, change it to `float64` or adding new fields) will break the compatibility.* **Tables:** -Tables can be used when backward compatibility matters. For example, tables can have new fields append at the bottom without breaking compatibility. +Tables can be used when backward compatibility matters. For example, tables can have new fields append at the bottom +without breaking compatibility. + ``` struct User table { Name []char; @@ -330,6 +370,7 @@ Email []char; Password []char; } ``` + Let's consider that you need another field... For tables, it's not an issue: ``` @@ -346,6 +387,7 @@ Since it's a table, you can add new fields at the bottom of the struct, and both ### Enums: Enums can be used as an alias to Integers type, such as `uint8`. + ``` enum Team uint8 { Unknown; @@ -355,21 +397,25 @@ Zombies = 255; } ``` -Enums must start with a zero value, the default value in all cases. If the value of any enum is omitted, it will use the order of enum as value. +Enums must start with a zero value, the default value in all cases. If the value of any enum is omitted, it will use the +order of enum as value. # Generator -Once you have a schema defined, you can generate the code. First, you need to `karmem` installed, get it from the releases page or run it with go. +Once you have a schema defined, you can generate the code. First, you need to `karmem` installed, get it from the +releases page or run it with go. ``` karmem build --assemblyscript -o "output-folder" your-schema.km ``` -*If you already have Golang installed, you can use `go karmem.org/cmd build --zig -o "output-folder" your-schema.km` instead.* +*If you already have Golang installed, you can use `go karmem.org/cmd build --zig -o "output-folder" your-schema.km` +instead.* **Commands:** **`build`** + - `--zig`: Enable generation for Zig - `--golang`: Enable generation for Golang - `--assemblyscript`: Enable generation for AssemblyScript @@ -383,11 +429,15 @@ Karmem is fast and is also aimed to be secure and stable for general usage. **Out Of Bounds** -Karmem includes bounds-checking to prevent out-of-bounds reading and avoid crashes and panics. That is something that Google Protobuf doesn't have, and malformed content will cause panic. However, it doesn't fix all possible vulnerabilities. +Karmem includes bounds-checking to prevent out-of-bounds reading and avoid crashes and panics. That is something that +Google Protobuf doesn't have, and malformed content will cause panic. However, it doesn't fix all possible +vulnerabilities. **Resource Exhaustion** -Karmem allows one pointer/offset can be re-used multiple times in the same message. Unfortunately, that behaviour makes it possible for a short message to generate more extensive arrays than the message size. Currently, the only mitigation for that issue is using Limited-Arrays instead of Arrays and avoiding Object-API decode. +Karmem allows one pointer/offset can be re-used multiple times in the same message. Unfortunately, that behaviour makes +it possible for a short message to generate more extensive arrays than the message size. Currently, the only mitigation +for that issue is using Limited-Arrays instead of Arrays and avoiding Object-API decode.