Using Java Serializer + LZ4

[ivolucas]

Hi,
I have made some investigation on using custom implementations of the JavaSerializationTranscoder using compression .
I have created to one transcoder using GZIP (JavaGzipTranscoder) and another transcoder using LZ4 (JavaLZ4Transcoder).

The JavaGzipTranscoder has bad results but JavaLZ4Transcoder has very similar and sometimes as best times that the JavaSerializationTranscoder using your Benchmark, and the size of the resulting byte array is very small compared with JavaSerializationTranscoder and even compared with KryoTranscoder.

Given the difference of sizes I have made a new Benchmark that takes in account the read/write of the session to a memcache server.

(time is in nanos in this graph)
This gives an edge to JavaLZ4Transcoder over JavaSerializationTranscoder to session bigger that 2kB, and giving a good cut to the session size at the same time. Has JavaLZ4Transcoder can be a direct replacement to JavaSerializationTranscoder this can be a nice upgrade for a lot of people.

Results of the Benchmarks
using-lz4-Benchmark.xlsx
using-lz4-BenchmarkWithMemcache.xlsx

This investigation let me thinking that compression code be a more transversal thing. We can have classes that handles the OutputStream >> byte[] >> InputStream work providing that services to the Transcoders. Removing that code from the Transcoders, and giving the opportunity to have implementation that do compression.
What do you think about that?

[exabrial]

wow +1 Way to do your research!

[magro]

First: sorry for the long delay!
Second: really great analysis!

Removing compression code from Transcoder sounds good!
Maybe we can provide a configurable "compression", that activates/injects the related compressor?
The lz4 code might live in the core module (related classes would only be loaded if "compression=lz4" ), because the user anyways has to place the jar in the lib dir.

WDYT?