Handle large number of files

[jtmoon79]

Summary

Running s4.exe C:/Windows/ resulted in most file processing threads returning "Error too many files open", and a failure to process that file.

Suggested behavior

Better handle processing many files. Some suggestions:

• During file preprocessing (filepreprocessor.rs), eliminate more files from consideration by removing files with file name extensions that are likely to fail processing, e.g. *.dll on a Windows platform is very likely a Dynamic Link Library file and will fail to process so do not attempt to process it.
• During file preprocessing (filepreprocessor.rs), eliminate more files from consideration by analyzing the first 1024 bytes during filepreprocessor.rs processing stage. This preprocessor analysis relates to FileType guesstimating needs refactoring #257
• The preprocessing stage is currently single-threaded. It should become a thread pool of preprocessing threads. This could speed initial pre-processing times. However, it would not fix "Error too many files open" and so is outside the scope of this specific Issue.

Workarounds

Limit the number of files opened by s4 using some external mechanism, for example PowerShell cmdlet Get-ChildItem with a -Filter

Get-ChildItem -Filter '*.log' -File -Path "C:\Windows" -Recurse -ErrorAction SilentlyContinue `
  | Select-Object -ExpandProperty FullName `
  | s4.exe -

In Unix, use find

find /var -type f -name '*log' | s4 -