Add Columns of Info to Asset Inventory

[Sh4d0wHunt3rX]

Hey guys, I was wondering if anyone can add these columns to the asset inventory file?

Status code
Page title
DNS records (A, AAAA, CNAME, SOA, TXT, NS, MX)
Certificates: If any certificate expired, revoked, wildcard, self signed, mismatched
WAF (which kind of waf the subdomain is using)

Thank you : )

[TheTechromancer]

These are all great suggestions. Work has been started on adding these features: #1001.

[TheTechromancer]

@amiremami all these features have been merged into dev, with the exception of ssl certificates. However thanks to nuclei, it is still possible to gather SSL information and store it in the asset inventory:

-m nuclei -c modules.nuclei.tags=ssl --allow-deadly

[Sh4d0wHunt3rX]

@TheTechromancer Please add "internal IP" column to asset inventory file, if it's useful.

[Sh4d0wHunt3rX]

I think it's better to remove "status" column from asset inventory file. Because if there is any port, we know it's active and if there is not any port, we know it's not active.

[Sh4d0wHunt3rX]

Hey @TheTechromancer , sorry to disturb you again. I'm running some flags containing these modules, however, it's not straight forward to get these info from the output files for a noob user like me.

fingerprintx: PROTOCOL

iis_shortnames: URL_HINT

github_codesearch: CODE_REPOSITORY
github_org: CODE_REPOSITORY

emailformat: EMAIL_ADDRESS
hunterio: EMAIL_ADDRESS
pgp: EMAIL_ADDRESS
skymem: EMAIL_ADDRESS
sslcert: EMAIL_ADDRESS
credshed: EMAIL_ADDRESS, HASHED_PASSWORD, PASSWORD, USERNAME
dehashed: HASHED_PASSWORD, PASSWORD, USERNAME

internetdb: VULNERABILITY
ajaxpro: VULNERABILITY
badsecrets: VULNERABILITY
dastardly: VULNERABILITY
generic_ssrf: VULNERABILITY
telerik: VULNERABILITY

bucket_amazon | bucket_azure | bucket_digitalocean | bucket_firebase | bucket_google: STORAGE_BUCKET

I wanted to know if it's possible for you to add some of these events to asset_inventory file.

For example possible to add protocol and VULNERABILITY ?

That would be great if in future it's possible to add specific output for some specific events, for example for CODE_REPOSITORY or EMAIL_ADDRESS , If it's possible to have separate output files so in a txt file or csv, we can see all the emails, password .... extracted and in another one we can see the github urls, in another one we can see all STORAGE_BUCKET.

🙏❤️

[TheTechromancer]

Right now, VULNERABILITY events are already added to the asset inventory. We also already have an output module emails, which outputs all emails to emails.txt. You can easily get any other event type by doing:

cat output.txt | grep PASSWORD | cut -f2

It's important to remember that while it's convenient to have this data in a flat format like a simple text file or CSV, it will never be possible to fully represent the data in this way. Because BBOT is recursive and because it often generates chains of 10 or 20 events, the only true way to represent the data (preserving all the relationships) is in a graph like Neo4j. This is the recommended way to view/search scan results, and it's why we haven't gone to great pains to flatten them.

For example, the asset inventory is deduplicated by host. A single host may have four different open ports, each with a completely different web server. But because the asset inventory has only a single column for HTTP status code, you will only see the best status code between the four. This is just one example of how flattening the data can cause important context to be lost.

[Sh4d0wHunt3rX]

Thank you so much for this valuable information, learned tons of new things from this and I will change the way how I was pulling the data. ❤️

[domwhewell-sage]

Hey @TheTechromancer,
If a host was found on scan 1 (So gets added to the asset_inventory with open ports) and then in scan 2 that host is no longer live (No open ports) it seems to be removed from the asset_inventory entirely.
Am I correct in thinking it it should update the line in the asset inventory to reflect that there is no longer ports open on that host instead of removing it entirely?
This is my command
bbot -t ginandjuice.shop -f subdomain-enum, portscan -n test -om asset_inventory --config output_modules.asset_inventory.use_previous=True

[TheTechromancer]

Yes I agree that would make more sense. Right now the asset inventory is serving as a placeholder until we have a better system for persisting bbot data across scans. Eventually we would like to have a recon-ng-like interface with a sqlite backend that stores data from multiple scans, and from which you can generate reports (like asset inventory) at will.

[domwhewell-sage]

Hi @TheTechromancer, I'm just experimenting with this and it appears if you comment out the ports (Lines 240-241) from the absorb_csv_row() function in the asset inventory you get the desired effect.

This ofc means that only DNS_NAME events are emitted and the nmap module does have to verify ports are still open on hosts emitted from the previous asset_inventory but isn't this desired behavior.

[TheTechromancer]

@domwhewell-sage it's a good point. And this is where our use cases start to differ; internally we use this module for large port scans (i.e. the entire private IP range). These scans can take upwards of 8 hours to finish, so it's important to preserve the open ports so only one port scan is needed. The other use case is just as valid, though.

Personally I've already spent more time than I'd like on this module and it still feels gross (and probably always will, because it's flattening data that doesn't want to be flattened). My focus now is moving us towards a better system like a persistent sqlite database. But if you want to add a new option etc. that changes its behavior, we should be able to accept the PR.