Rewrite the SVD

[Ch4s3]

As discussed in #27 we need to rewrite the SVD method here. This could also be used as an opportunity to remove the monkey patch on matrix, and provide a method like svd(matrix). This will clear up #5 as well.

Here's a resource I found on SVD. I'll definitely need help on this one.

[Ch4s3]

@parkr The more I read about this, the more I think a pure Ruby implementation is just going to always give us poor performance. It might make sense to bundle a C solution into the gem since there a lot of great existing solutions. Then just check to make sure it builds correctly on various platforms.

I'm not married to this approach, but it could be the right way forward. Thoughts?

[parkr]

That seems like a good plan.

[Ch4s3]

Ok. I'll devise a plan tonight and scout out implementations. It'll be good to wrap this one up.

[parkr]

Sounds terrific. Thanks, Chase!

[Ch4s3]

Not a problem. I want to wrap up a few things in a row and get a release out in the next week(ish).

[Ch4s3]

I figured out how to build and compile C-extensions, and just need to settle on an implementation that has a good license. @parkr do you have a strong OSS license preference?

*edit this implementation looks nice, and I've seen the author's name on some papers. The license is GNU.

[parkr]

As long as the license is compatible and GitHub can use it for commercial purposes, it's fine. I usually lean for the MIT. Can MIT projects contain GNU code? I'm not sure.

[Ch4s3]

I'll read up

[Ch4s3]

Ok, I've tried few things and have it sort of working, but it segfaults on rare occasions. Still a wip.

[parkr]

Ok, I've tried few things and have it sort of working, but it segfaults on rare occasions. Still a wip.

Sweet! When is it segfaulting? Is the C still optional?

[Ch4s3]

It segfaults while transforming the matrix for some inputs. It could be optional.

[jayniz]

Any news here? :)

[Ch4s3]

@jayniz I had to take a break from this for a bit, but I'll try to get back to it soon. Basically I found 1 or 2 promising implementations, but they both segfault for some input. I'm not sure why yet so I haven't gotten past that. I'll do my best to figure it out soon, but I'm open to people helping out.

[jayniz]

Hey @Ch4s3 no problemo :-) The bayes classifier is working and goes into production today.

I played around with LSI locally, and I got the above errors for my inputs - not with the GLS lib and gem though, that worked. But where training the bayes with 6k inputs took ~8 sec, LSI took a couple of minutes for 600 inputs (and with 6k inputs it's still running after 30h) on a recent 13"mbp with 16G ram.

In my benchmarks the bayes classifier performed quite well to detect comment spam though - trained with 6k comments I had it classify ~80k comments and it classified correctly in 94.5% of the cases, with 4.75% false negatives and 0.75% false positives.

Let's see how it behaves in production!