Adjust cronjob/batch update logic to match new DB size

[foolo]

Need to investigate how the cronjob should work with a 1M records database after #66

A cronjob may not be the ideal solution. An alternative is to use a constantly running process.

[Divide-By-0]

I noticed that the current job has been shut off, but it might be good to keep that on for just emails that were found in inbox so far, until we resolve this issue, as those are the most important ones.

[foolo]

I noticed that the current job has been shut off, but it might be good to keep that on for just emails that were found in inbox so far, until we resolve this issue, as those are the most important ones.

Okay! we can update the cronjob so that it only checks for records which are not marked as "scraped", etc

It is running though (i double checked in crontab, and in server logs and it's all up and running), but only with 10 DSPs every 1 minute, so with the current database size of 1M, it's basically negligible :) But as you suggested, we can change to only "from inbox" records, and then it will make sense again, when the number of records in scope will be below 10k or so.