leakImporter
LeakImporter is the tool dedicated to import leaks into your mysql/mongodb database. It takes a standardized leak file in input.
python3 leakImporter.py <leakName> <leakFile>
Eg: python3 leakImporter.py Tumblr tumblr.txt
You will need enough disk space to store your leak file twice. Leak Importer creates a copy of the file in your /tmp (not for fun ofc, for processing purposes, in case your file does not fit in memory, which is the case 90% of the time).
LeakImporter DO NOT HANDLE DUPLICATES. Doing so reduces processing time and performances by several orders of magnitude.
If, for any reason, leakImporter cannot import a credential, it will add it in the file '_not_imported.txt'.
LeakImporter will not import emails belonging to private individuals. Email addresses having a domain from a fixed set will not be imported such as mails like "@gmail.com","@hotmail.com" etc. These addresses do not have any value as we want to search specific companies email addresses. Private individuals are not interesting for our use case. If you want to change this behaviour, feel free to edit leakImporter.py, the list is easily modifiable and can be empty.