estimate density ratio of large training set and test set

[charliezon]

Hi,

Thank you for sharing this python package for density ratio estimation.
In practical applications, the training sets are often very large.
Is it possible to use this tool to estimate density ratio of large training set and test set? For example, a training set of 20 GB data.

[ameya98]

I haven't experimented with very large datasets. Let me know if you can get anything useful out of densratio_py. One point you should ensure is that you pick the same number of samples from both the training set and the test set. This means it's really the size of the test set that bounds how many samples to take. Another point is that uLSIF/RuLSIF (the underlying algorithm that densratio_py uses) was made for the purpose of change-point detection. If you're hoping to recover the actual densities of the training and test set, I'm not sure how useful this will be.

Still, let me know!

[charliezon]

Dear Ameya,

Thank you very much for your reply.
I don't hope to recover the actual densities of the training and test set. I just want to estimate the density ratio of training set and test set as accurate as possible.
I think if I can take the whole training set for the estimation, the estimation result would be more accurate. In contrast, if the number of samples are bounded by the size of test set, we can only choose a part of samples from training set for estimation, this would introduce more error of estimation. If I have to do so, I need to think about a good re-sampling method to reduce this error.

Do you have any suggestions? Thank you again.

[ameya98]

I believe that if you take a sufficient number of samples from both distributions - not necessarily the entire training/test set - you might still get a good result with RuLSIF.

However, how much is 'sufficient' is really dependent on the distributions themselves. Only experimentation can tell you that. I would start out by splitting an artificial dataset (ie, one for which you know the underlying distribution) into training and test sets, and seeing how many samples you need for a reasonable output.

Thanks!

[charliezon]

Thanks a lot for your valuable suggestion!
Actually, in my case, one of the test sets (only 28 samples) is too small compared with the training set (more than 25000 samples). I've no idea whether is possible to estimate the density ratio with a reasonable accuracy.

Anyway, I will try the method you suggested. Thanks!