Compression ?

[davinerd]

I was thinking that maybe we need to add some sort of compression.

it's useful when you need to exfiltrate big files, but I'm wondering how we can integrate that.

My first thought was to make it mandatory, i.e. hardcode it and use it on every channel, but then I realized that it may not be good for a general point of view. Also, I would like to let the user choose which compression scheme to use.

The best way to implement it would be to add a third "chain": Encoder -> Channel -> Compression.
It shouldn't be hard to implement but it can be tricky to integrate without making sneaky-creeper messy.

The easier way would be to include compression into the encoding modules. In this way, you just need to create an encoder ad use is, maybe chaining with other encoders (like AES + zip, or RSA + rar, etc etc).

What do you think?

[DakotaNelson]

I agree that compression is a good idea.

This was actually one of the original intents for the encoders, as you mention. That way you could do something like (data) -> AES -> zip -> channel of choice.

As you noticed, though, things get a little weird when the compression is to files - for example, a zip file might not transfer over Twitter very cleanly. I see three ways of dealing with this:

• Just base64 encode everything (to guarantee that it's text) and then treat it as text - this is the current approach. On the upside, we're guaranteed that everything will work. On the downside, this makes compression less useful - base64 grows the data from its binary form (which is unavoidable).
• Don't implement encoders that don't output text - so we'll have to use encryption that outputs strictly ascii-printable characters. This SO question has some insight as to why that might not be great. (and also has an answer that suggests that we do above - compress, then base64 encode)
• Have two classes of channels - ones for text, and ones for files/binary blobs. Then, have encoders say what they output and enforce compatibility at runtime. This has the advantage of maximum efficiency (For instance, you could compress, steganographically encode into an image and then upload that image to Twitter for really high bandwidth. Another example is the SoundCloud channel right now - it can handle binary data, and we're wasting some of its capacity by only inputting text.) but at the cost of a lot of complexity. I think this would be awesome, but I'm not sure it's worth how complicated it would be. (at least... yet 😄)

Python has some built-in compression, so a quick prototype of a compression encoder should be easy to set up.

Would love to hear your thoughts on this - I like the current "base64 everything" approach because it's easy, but I'm still not sure what the right approach is.

[davinerd]

So I'd say we can keep it simple by doing:
(data) -> [encoders] -> filetype -> channel

Where [encoders] includes encoding (b64), encryption, and compression (we can leverage the 'chaining' features of encoders). Then you can specify the FileType (as written in #69 ) and then send it through the desired channel.

In this case you don't need to change our channels or encoders, but just add a new block (the effort shouldn't be too much).

What do you think?