Compression and encryption

[cocowalla]

FASTER looks really interesting! I don't see any mentions of compression, and I was wondering if that, and encryption, would be possible within the design (maintaining the same persistence guarantees etc)?

Is there an extensibility point that would permit this with existing Devices, or if not, would it be possible to create something like a CompressedLocalStorageDevice?

My use case is storing hundreds of GBs of append-only data, with multiple readers doing stuff with data as it's committed. GZip Compression would be a real boon for this, and might even make things faster because of reduced storage IO.

[badrishc]

Yes, it's definitely doable using our generic IDevice abstraction. The API is fairly cleanly defined and we have users do things such as adding checksums, transparently to FASTER core.

You would write compressed blocks to disk and need some lookup structure to find them, since pages would be of variable size. Random reads might need to read the entire page, decompress and return the record.

To get started, see our in built implementations such as ManagedLocalStorageDevice.

[cocowalla]

Nice, are you able to point to repos where this kind of thing is being done?

[badrishc]

Not aware of public ones. However, you can also take a look at several of our meta-devices as examples. These are not actually devices, but orchestrate access to sets of other devices. It shows how you can use the device abstraction in different ways:

• TieredStorageDevice
• ShardedStorageDevice