Binary FKTables

[scarrazza]

Issue by Zaharid
Friday Jun 23, 2017 at 16:20 GMT
Originally opened as https://github.com/NNPDF/libnnpdf/issues/11

---

The alphas fits are becoming unbearable due to the slowness of the fktable processing. As a first step can we simply compress each fktable individually? That's already quite a speed up, independent of everything else (and should be working now, right?).

For the binary, I think looking at something like avro is better than inventing the format ourselves:

https://avro.apache.org/docs/current/spec.html

in my opinion it has the things we need (and not many we don't).

[felixhekhorn]

I guess PineAPPL grids are binary, so we can close - do you agree? @cschwan @scarrazza @Zaharid

[cschwan]

Yes, they are binary; more concretely the PineAPPL files are

• the Grid data structure from the PineAPPL crate,
• serialized
• with bincode and
• prefixed with a small header¹,
• transparently² compressed using LZ4, one of the fastest decompression algorithms.

As for the size, have a at look NNPDF/pineappl#45 (comment) in which I discuss the conversion of the complete https://github.com/NNPDF/applgrids repository. For (PineAPPL-)FK tables I would expect a similar picture.

Footnotes

1. the file header allows us to completely change all data structures and serialization formats in newer version if we decide we need that ↩

2. transparently means that you can run lz4 -d grid.pineappl.lz4 to decompress the file, which will get you the header + serde/bincode output. The decompressed output can also be read by every API just as well as the compressed file ↩

[RosalynLP]

:)