Support for Non-Long Keys

[kmader]

A proposed addition to have non-Long keys using the IndexableRDD which is a thin class on top of IndexedRDD and has a conversion between a key of any type and a Long using the provided IndexableKey class. Only some of the functions have currently been implemented, but it is fairly straightforward to implement the rest as needed.

[concretevitamin]

How general is this interface? Let's say the integers in my case class are pretty large, then (assuming toId is still some kind of encoding or hashing) fromId may fail?

[kmader]

Those are just the example ones, clearly in this case IndexableKey stops working for Point3D as soon as y>=10 or x>=100. fromId and toId need to be unique and reversible for this to be a really practical approach.
I think however there are many scenarios where this makes sense and it makes processing much easier to keep the key as the type it is and having the IndexableRDD perform the translation automatically, rather than converting it to long and keeping track of its real value somewhere else.

[concretevitamin]

Speaking for certain NLP applications, not being able to have a bijection can be a blocker. It'd be good to have real-world examples of applications the purposed interface makes available to build.

[kmader]

Well spatial applications / imaging is a perfect application since for many applications the domain of the Point3D is known (dimensions 1000 x 1000 x 1000) which means the points can be bijectively mapped to longs using id=(1000*z+y)*1000+x. This is also generally applicable to matrices and arrays (particularly sparse ones) since they typically have well defined sizes and could also have such a mapping performed easily and rarely do they contain a number of points which cannot be expressed as a long.