Skip to content

Commit

Permalink
rework Peppi format
Browse files Browse the repository at this point in the history
  • Loading branch information
hohav committed Aug 22, 2022
1 parent 773e90b commit caa0b5d
Show file tree
Hide file tree
Showing 3 changed files with 511 additions and 170 deletions.
17 changes: 10 additions & 7 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,19 +1,22 @@
[package]
name = "peppi-slp"
version = "0.2.1"
version = "0.3.0"
authors = ["melkor <[email protected]>"]
edition = "2021"

[dependencies]
arrow = "6.3"
arrow2 = { git = "https://github.com/jorgecarleitao/arrow2.git", rev = "f75ce4266e04c4084fa5666c6c5ff5efd6b5036f", features = ["io_ipc", "io_ipc_write_async", "io_ipc_compression", "io_print"] }
atty = "0.2"
clap = "2.33"
clap = { version = "3.2", features = ["cargo"] }
env_logger = "0.9"
log = "0.4"
parquet = { version = "6.3", default-features = false, features = ["arrow", "base64"] }
parquet-format = "4.0"
peppi = "1.0.0-alpha.6"
serde_json = "1.0"
peekread = "0.1"
peppi = { git = "https://github.com/hohav/peppi.git", rev = "961d95beea86b7d2800ee33f1e0d5f60f186454e" }
serde = { version = "1.0", features = ["derive"] }
serde_json = { version = "1.0", features = ["preserve_order"] }
tar = "0.4"
tempfile = "3.3.0"
xxhash-rust = { version = "0.8.5", features = ["xxh3"] }

[profile.release]
opt-level = 3
Expand Down
25 changes: 14 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,19 +66,22 @@ $ slp -s game.slp | jq
}
```

Convert a replay to Peppi format (**⚠️ experimental!**):
Convert a replay to Peppi format:

```bash
$ slp -f peppi -o foo game.slp
$ slp -f peppi -o game.slpp game.slp
```

This creates a directory `foo` with the following files:
```
start.json
end.json
metadata.json
frames.parquet
items.parquet
```
## Peppi format

The Peppi format (.slpp) is a GNU tar archive containing the following files:

- peppi.json
- metadata.json
- start.json
- start.raw
- end.json
- end.raw
- frames.arrow

Frame and item data are stored in columnar [Parquet](https://parquet.apache.org/) format. It will be larger than the original .slp file uncompressed, but smaller compressed.
The last of these, `frames.arrow`, is an [Arrow IPC](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format) file containing all of the game's frame data. This is a columnar format, which makes .slpp files about twice as compressible as .slp files.
Loading

0 comments on commit caa0b5d

Please sign in to comment.