Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extremely slow processing of pcap files #51

Closed
nuschpl opened this issue Dec 12, 2020 · 5 comments · Fixed by #73 · May be fixed by #54
Closed

Extremely slow processing of pcap files #51

nuschpl opened this issue Dec 12, 2020 · 5 comments · Fixed by #73 · May be fixed by #54

Comments

@nuschpl
Copy link

nuschpl commented Dec 12, 2020

I used to process ~15GB pcap files with https://github.com/lgandx/PCredz and I believed it's slow and not optimal tool. But BruteShark comparing to it is extremely slow. Filtering 15GB pcap with wireshark for NTLMSSP packets takes less than a minute on my 16core, 32GB hardware, it results in 3MB file which is processed by BruteShark almost the same ammount of time - something is not optimal here. I've found it utilizes only 1 CPU core and use limited amount of RAM for processing, is there any prefered way to implement threads and caching here ? I can contribute

@odedshimon
Copy link
Owner

Hi @nuschpl !

Thanks for letting me know.

If you used BruteSharkCli it is It makes sense that the run time was long since it is reconstructing all the TCP / UDP sessions by default.

Unfortunately the user-interface is currently does not provide the required capabilities for use a configuration that is appropriate for a particular analysis.. (let's say in your case it is enough to run only the hashes module without building sessions).

BruteSharkDesktop although provide this capabilities (see picture attached), you may prefer to use it for the pcap you described.

In the near future we plan to make significant refactor that will also allow BruteSharkCli to run as a single command line with all the options that the tool provides (this will also solve the other issue that you have opened).

We are welcome contributors, feel free to suggest a design for BruteSharkCli or even implement it.
Also feel free to contribute to any other part of BruteShark.

image

@nuschpl
Copy link
Author

nuschpl commented Dec 14, 2020

Hi, I'm not sure sacrificing some functionality (reconstructing sessions in this case) is the only solution if current implementation is slow. I have at least two suggestions:

  1. Use all CPU cores- in my case one core was 100% , while other 15 cores/threads were idling - it was not I/O issue.
  2. Use moving window for recreating the session - sorry for ignorance - didn't looked into the code yet, but looking at time of processing at very end - 10TCP sessions per second it seems you do some comparisons across entrie PCAP file so processing time increases with every loaded packet. But TCP sessions usually don't last so long - at maximum few minutes when no related packet is transfered - so packes could be loaded into caching queue with:
  • one thread could append loaded packets from multiple PCAPs to the end of queue and up to user defined RAM usage
  • user defined (defaulting to avaiable CPU cores -2 ) amount of threads could process the data with current approach
  • separate thread could remove the oldest packets from caching queue marked as processed and not used since limit (e.g 6 minutes(in relation to PCAP timestamps) and removing those already marked

@odedshimon
Copy link
Owner

It's definitely not the only solution, but a trivial one (and useful for the general user experience as well).
Thank you for suggesting this, I will consider it as possible improvements.
Also, feel free to fork this project and implement any of this solutions, as well as pull-requesting to this repo in case you will decide to do it.

@aviadtamir
Copy link
Collaborator

@nuschpl

there's a branch in this repo in which i'm trying to overcome this issue, that a look if you want to :)

a little glimpse to what i've already accomplished;

Brute.Shark.15_12_2020.19_51_54.mp4

@aviadtamir aviadtamir linked a pull request Dec 19, 2020 that will close this issue
@aviadtamir aviadtamir added enhancement New feature or request and removed enhancement New feature or request labels Jan 2, 2021
@odedshimon odedshimon linked a pull request Jan 23, 2021 that will close this issue
12 tasks
@odedshimon
Copy link
Owner

BruteSharkCli now has two modes: single command and shell mode. The single command mode works by getting all the relevant parameters for the processing and then printing the results to stdout or files. The shell mode allows to perform each step individually.

You can just set the credentials module and configure an output directory and all your hashes will be exported automatically.

Example Video: https://youtu.be/GjGV5vwIGi8

Please see the 'Usage' section for detailed documentation and examples.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants