Documentation
This script requires a Windows system to run on that has internet access and allows visual basic scripts to run.
Required for feature functionality:
- API keys are required to query the following services/products:
- VirusTotal (some functionality depends on access to VirusTotal API and will not be available if not enabled)
- ThreatGRID
- CIF
- Malshare.com
- Carbon Black Cb Response
- Proofpoint ET Intelligence
- PassiveTotal
- WhoAPI
- Items such as location and whois names were removed for GDPR
- Freedomainapi.com
- Threatintelligenceaggregator.org
- Optional API for AlienVault OTX (works without API key as well just has lower lookup limits)
- However it's worth using an API key, as it allows 10,000 requests per hour rather than just 1,000 requests per hour.
- Some queries such as NIDS require an API key
- DDNS.dat containing a list of Dynamic DNS domains is required for queries to work. (Included in the git repository)
Whois lookups are used to identify the contact/owner of an IP/domain along with identifying sinkhole domains. Whois retrieval is done via VirusTotal first. VirusTotal doesn't have whois data for every domain so as a backup other Whois functionality exists:
-
Domain whois lookup order
-
- VirusTotal
- Internal Cache Database
- AlienVault OTX
- WhoAPI or freedomainapi.com
- Sysinternals whois tool:
http://live.sysinternals.com/whois.exe
- IP whois lookups are performed against ARIN and RIPE public APIs. If IP is not registered with either of those then whois IP lookups are done through AlientVault OTX. If not in AlienVault OTX the final lookup requires the nirsoft.net whosip utility.
API lookups have limitations such as number of allowed queries within a minute or hour. The script needs to regulate queries for the least common denominator (defaults to 4 queries a minute). If API keys are used elsewhere this can cause errors when performing lookups as the threshold for lookups per minute has been hit. If you receive errors that lookups are failing this could be the cause.
- VirusTotal community page where you can create an account and sign into to get a VirusTotal Public API key:
https://www.virustotal.com/en/community/
- The ThreatGRID API key can be retrieved from your account after logging in:
https://panacea.threatgrid.com
- malshare.com provides a free API key via their website with registration:
http://malshare.com/register.php
- WhoAPI provides a free trial. Free lookup has moved to freedomainapi.com (no longer provides contact/location due to GDPR).
https://whoapi.com/user/signup
- Freedomainapi.com provides a one-time fee API via their website. After registration click the Account info link.
http://freedomainapi.com/members/register.php
- Carbon Black provides an API key for the individual consoles and must be retrieved from the console:
Profile info > API Token
- ThreatIntelligenceAggregator.org provides lookups for virus detection names.
Email: [email protected]
This document assumes familiarity with industry terminology. The script assumes you are not trying to use Private API functions provided by vendors (example: VirusTotal private API). The script was only tested using Public API keys.
Vendor Threat Triage Lookup (VTTL) script can be used to identify detection information and threat intelligence for IP addresses, URLs, domains and hashes. It queries threat intelligence feeds, RBLs, GeoIP, Reverse Lookup and WHOIS to gather, process, sort and present data related to the lookup items. The script outputs the information to a CSV or Microsoft Excel spreadsheet.
Vendor Threat Triage Lookup (VTTL) script provide information related to potential threats you are feeding into the script. Sometimes the output information is simply that a certain source or vendor has more information about that specific lookup item. Other times a score or free form text is used. The output is not overly verbose but tries to convey enough information for initial triage while providing avenues for more detailed information.
When it relates to malware investigations, VTTL provides the most value when used with the VirusTotal public API. To perform lookups an API key will need to be generated for you. This can be done by creating a VirusTotal community account. The more data you feed the script the longer the lookups take (VirusTotal community account has 4 lookups a minute limit). VTTL limits of number of lookups performed in a given time frame to a provided threshold (default of 1 every 15 seconds). Other vendors have similar rate limits and so please be mindful of the limits.
The default configuration is to submit but not scan IP addresses, domains and URLs to VirusTotal. If you don't want that data to be sent to VirusTotal do not add it to the input file. This script does not support uploading files to VirusTotal. If you are not getting scan results back for a hash it will need to be uploaded to VirusTotal for the hash lookup to work (do not submit potential sensitive files to VirusTotal).
Note: URLs are more unique and will likely not return any results without submitting for scanning. It is recommended to only submit the associated domain. URL submission should be considered depreciated.
Warning: Submitting domains to the VirusTotal API could generate an entry in VirusTotal for the domain if one does not already exist. No virus/reputation scanning will occur but that domain may populate with passive DNS and Whois information.
For hash lookups the last scan data is provided. To get more up to date scan results you must rescan/resubmit the file to VirusTotal. However, when using database functionality the last scan date and VirusTotal results will be retrieved from the database cache. VTTL will need to be told to update the cache or to disable reading from cache. To update the cache, use INI setting HashRefresh. This is the number of days from the first time seeing the hash that you want to refresh the cache data (get updated results) for processed items.
VTTL can perform caching for domain and whois results. This is helpful if you are querying the same domain names and want to limit API calls.
Download the project zip file from GitHub and extract to a desired location. VTTL needs to be run from a directory where the user running the script has write access to create files and directories.
VTTL version 7.8.1 or greater has the ability to cache API calls to a SQLite database. This requires an ODBC driver be installed:
http://www.ch-werner.de/sqliteodbc/
Note: I typically install this one for my 64-bit computers:
_ _http://www.ch-werner.de/sqliteodbc/sqliteodbc_w64.exe
Dynamic DNS addresses can be identified by querying a local list provided in DDNS.dat. The DDNS.dat list is provided in the git repro and can be modified if desired.
The script uses VirusTotal to provide whois information but does not contain whois data for everything. The script will natively perform API lookups against RIPE NCC and AIRN IP ranges managed by the two registrars.
To ensure the most complete whois data is obtained the Sysinternals whois and NirSoft WhosIP lookup utilities can be used. They need to be located in the script directory or a directory set in the path variable. A directory can be added to the %path% variable or the stand alone executables copied to an already defined folder.
Open a command prompt and change directory to where the VTTL script is located. Run the whosip and whois commands to verify proper operation:
When first run you will be prompted for API keys:
If you do not enter API keys, you will be prompted if you want to disable future prompts:
Note: To enable API queries after being disabled modify the appropriate setting in the INI or with the GUI application and save.
Once API keys have been loaded the script moves to performing the lookup. VTTL using the input file vtlist.txt to perform lookups. VTTL will create vtlist.txt if it does not already exist. Populate vtlist.txt with the list of hashes or IP addresses and domains that need to be looked up and save the file. VTTL will perform deduplication of input items to prevent unnecessary lookups. IP and Domain addresses can be processed together in the same list. Hashes will need to be processed in a separate list than domains and IP addresses.
_ Note: vtlist.txt must have a carriage return line feed CrLf after each entry. Notepad does this with the enter/return key but not all text editors do._
Note: Editing the vtlist.txt file while it is processing the lookups will alter processing and possibly cause unwanted behavior.
By default, a CSV file will be created in a sub directory named "Reports". Alternatively you can pass the /e parameter and it will output to Microsoft Excel if installed. Once the lookups and queries complete, if written to a CSV file, the script will open the resulting CSV report. CSV output is the recommended output method.
Should VTTL be unable to connect to VirusTotal it will produce an error stating the problem and pause at the message prompt for 30 seconds before attempting the lookup again. Clicking the Ok button continues with the lookups:
"Error # err.number - err.description
Will attempt to submit to VirusTotal again. If problems persist check connectivity"
There may be things that you don't want to have submitted to VTTL for lookups but may somehow wind up in the lookup list. By default the script does not submit items for scanning and only performs lookups. VTTL will not submit RFC 1918 IP addresses to VirusTotal and any text that does not contain at least one period (Excluding hashes).
To prevent scan submissions for internal host names such as internalhost.mydomain.com you simply add .mydomain.com to the list in VTTL_domains.txt which will be in the same directory as the script. Note the period before mydomain is used to identify sub domains and should be included. Excluding the period will excluding what you expect but will also exclude things like fakemydomain.com from lookups. The script will create VTTL_domains.txt by default and add .local to the list.
If there are specific items that the script should not lookup add them to the list in VTTL_NoSubmit.txt. This list file performs direct matches for each item (one per line). It must be an exact match to the line in vtlist.txt for it not to be submitted. This text file is also located in the same directory as the script.
- Hash (md5/sha1/sha256)
- IPv4
- IPv6 (Not fully supported by all vendors)
- Domain
- Associated IP will also be looked up if provided in API results
- URL (usually requires scanning. Is not recommended to use and should be considered depreciated)
- Associated domain will also be looked up
The script uses the VirusTotal API to provide VirusTotal statistics on detections for the provided items. VirusTotal output is in the following format:
Number of positive detections – Scanned Item VirusTotal URL to scan results
Additional outputs based on the VirusTotal API include detection name lookups via threatintelligenceaggregator.org for the below vendors. VTTL also outputs statistics based on the VirusTotal API results.
- Trend Micro
- Microsoft
- McAfee
- Sophos
- Symantec
- ESET-NOD32
- Avira AntiVir
- DrWeb
- Avira
- Panda
- ClamAV
- F-Secure
- Bitdefender
Hash Detection name – Vendor encyclopedia URL
- count number of URL and hash detections returned for IP or domain lookup (unique URLs detected as malicious hosted at IP/domain)
- count number of unique hashes with the domain/IP embedded
- count number of unique hashes with traffic to domain/IP address
- count number of unique domain detections returned for IP lookup
-
ThreatGRID
- API key is required
- Reports if the hash is known to ThreatGRID
-
Collective Intelligence Framework (CIF)
- API key is required to use
-
Dynamic DNS
-
Threat Crowd
-
Cuckoo Sandbox
-
ET Intelligence from Proofpoint
-
PassiveTotal
- Only performs whois organization query for newly identified signers
- Outputs associated domains in spreadsheet output
-
AlienVault
GeoIP information can be retrieved by the AlienVault OTX and freegeoip.app APIs. Note freegeoip.app has 15,000 queries per hour limit. If using SQLite then the IP to Country Lite or IP to Country can be used from https://db-ip.com/db/. This is included in the default database provided in the git repo or can be imported via the Import_GEOIP.vbs script.
The following DNS lookups are done against RBLs:
- IP Address Lookups
- CBL
- Spamhaus ZEN
- Barracuda
- SORBS
- IP Address and Domain Lookups
- uribl.com
- surbl.org
- Domain Lookups
- Spamhaus DBL
- Quad9
An authorized DNS server must be used for Spamhaus ZEN, DBL, Barracuda and URIBL to be utilized. This can be configured via the GUI or INI settings. Otherwise, your default configured DNS server will be used. It is recommended to test your DNS block list configurations before relying on query results.
Parameters can be passed to the script to modify the behavior.
- /s Run silently
- /e Create Spreadsheet in Excel
- /g Perform queries against CSV output from Sysinternals SigCheck and Carbon Black hash dump script.
- /a Perform queries against EnCase tab separated value output or NetAMP CSV
- /dcb Disable Carbon Black
- /dms disable MalShare
- /dtc Disable Threat Crowd lookups
- /det Disable Proofpoint Emerging Threats
- /dtg Disable ThreatGRID
- filepath if you provide a file path that file will be used as sigcheck input. Also "starting lookups" prompt will be bypassed
The script has two spreadsheet modes, outputting to a CSV or to Excel. Passing the /e parameter will output to Microsoft Excel. The script will check the contents of vtlist.txt and identify if it contains hashes or domains and IP addresses to build the appropriate header row.
The spreadsheet has a dynamically generated header row based off of the enabled lookups. The following is an example auto populated header row:
Scanned Item|VTTI Download From|VTTI Referrer|VTTI Callback To|VTTI_Uniq_URLs|VTTI_Uniq_Domains|Barracuda|RBL_Spamhaus ZEN|RBL_Spamhaus DBL|ThreatGRID| CIF|Country Name|Country Code|Region Name|Region Code|City Name|Reverse DNS|WHOIS|Hosted Domains|IP Address|DDNS|Threat Crowd|VT Download|VT Referrer|VT Callback|VT URL|Restricted Domain|Sinkhole
Most cell values are an "X" to indicate a positive hit but others report numbers or even text. VTTI Download From, VTTI Referrer (embedded in strings), VTTI Callback To (direct communication), VTTI_Uniq_URLs, VTTI_Uniq_Domains reflect a numerical value of how many hits each category had (100 max).
VTTI Download From: Number of vendor detected files on VirusTotal that have been downloaded from the source.
VTTI Referrer: Number of vendor detected files on VirusTotal that reference the source in strings found in the file.
VTTI Callback To: Number of vendor detected files on VirusTotal that attempt to connect back to the source.
VTTI_Uniq_URLs: Number of vendor detected URLs on VirusTotal that are found at the source domain/IP address.
VTTI_Uniq_Domains: Number of unique domains listed in the vendor detected unique URLs for an IP address. This gives an idea of is only one domain possibly malicious or many domains.
VT Download, VT Referrer, and VT Callback reflect the greatest number of vendor positive detections for a scanned file. Looking at those provides how much vendor confidence there is for the VTTI Download From, VTTI Referrer VTTI Callback To categories. CIF outputs deduplicated results scoring higher than 50 for confidence level:
Category: Name/source – URL
Validation: Provides items such as "IOC is part of a highly visited website.", "Akamai CDN", and "Whitelist". Highly visited sites can have malicious activity associated with them, but are not malicious themselves.
Detection Name #
Detection Name # is an array of detection names associated with the domain/IP address that was looked up. The number of columns are determined by the number of samples you set to query.
If the contents of vtlist.txt is all hashes then the appropriate header row will be generated. The following is an example auto populated header row:
hash|VT Scan|Mal Score|Generic Score|PUA Score|HKTL Score|Malicious|ThreatGRID|Threat Crowd|Trend Micro|Microsoft|McAfee|Sophos|Symantec|ESET|Avira|DrWeb|Scan Date|Common Name|Detection Type|Cache|MalShare
- VT Scan – Number of positive vendor detections
- Mal Score –Number of detection names that appear to be malware
- Generic Score – Number of detection names that appear to be generic
- PUA Score – Number of detection names that appear to be a PUP/PUA
- Malicious – Adjusted Mal Score based on threat intelligence analysis and correlation
- Scan Date – Last scan date by VirusTotal
- Common Name – The most common vendor name returned in the VirusTotal results
The other vendor detections are based on threatintelligenceaggregator lookups for the detection names. Hash lookups that match are marked with an "X". Matched detection names include the full detection name in the cell contents.
API keys are stored in the same location as the script and have the following names:
- Vt.dat (VirusTotal API key)
- Tg.dat (ThreatGRID API key)
- Cif.dat (CIF)
- Malshare.dat (malshare.com API key)
- Pp.dat (Proofpoint ET Pro API key)
- Pt.dat (PassiveTotal API key)
- Wa.dat (WhoAPI API key)
These files are generated by the script. If your API key has changed you must delete the appropriate key file to input the new key into the script (prompt will appear).
The scan input file vtlist.txt will be created when you run the script if one does not already exist. You must input your list of IP addresses and domain names or a list of hashes into this file and save it for the script to provide results.
Some lookups require dat files or cache results to files.
- DDNS.dat (required for dynamic DNS lookups)
VTTL can process vtlist.que when it has completed vtlist.txt. Before exiting VTTL will check if vtlist.que exists and if so deletes vtlist.txt and replace it with vtlist.que (rename). It then checks if sigcheck.que exists and if so uses that to process the vtlist.que. Note then when processing the vtlist.que the same parameters are used with exception of the sigcheck parameter.
VTTL supports importing CSV files from various tool output for combining file paths and other metadata with the VTTL output:
- Sysinternals sigcheck
- Sysinternals autorunsc
- Rhythm-CB-Scripts CB_Hash_Dump
- Cisco Network AMP
- Crowdstrike Falcon
- EnCase
The PUP.dat cache file is a list of PUP/PUA detection names. This list is built as PUP/PUA files are identified in VirusTotal. The more accurate the PUP cache is then the lower the chance that a PUA/PUP will be missed. The PUP.dat is located in the sub directory "cache".
The hktl.dat cache file is a list of hacker tool detections detection names. This list is built as hacker tool files are identified by VirusTotal lookups. The more accurate the hacker tool and PUP cache is then the lower the chance that grayware will be missed. The hktl.dat is located in the sub directory "cache".
Country code mappings (ex: Afghanistan|AF) are used to populate the spreadsheet where one item in the mapping isn't provided by GeoIP lookups. This mapping is stored in the cc.dat file located in the sub directory "cache". The file is created automatically by the script and should not require modification.
If you have previously identified hashes that you want VTTL to detect then add the hash values to the malhash.dat located in the cache folder. This will increase the adjusted malware score and import the specified note (I typically use a detection name or type). To populate malhash.dat just add the hash value followed by a pipe character "|" followed by your comment (will show up in common name column). Malhash.dat format example:
0123456789abcdef123456789abcdef0|keylogger
If you have a digital publisher/signer that is known to be used to sign malware add them to the ds_mal.dat located in the cache folder. This will increase the adjusted malware score based on what value is set in the ds_mal.dat. To populate malhash.dat add the known bad signer followed by the pipe character "|" and then the value to add to the adjusted malware score. Example:
Open Source Developer, Got Compromised|50
The items above will add 50 to the adjusted malware score for any files signed by the specified publishers.
whitehash.dat can be used to keep track of known good files or files that have been researched. This tracked by hash value and has the following format:
hash|CommonName
When a matching hash is found VTTL uses the paired value to populate the Common Name field in the spreadsheet output. This will also decrease the adjusted mal score a little.
The script performs caching to speed up processing and limit lookups. Caching is performed for threatintelligenceaggregator.org lookups and is stored in the following files within the cache folder:
- Avira.dat
- DrWeb.dat
- ESET.dat
- McAfee.dat
- Microsoft.dat
- Sophos.dat
- Symantec.dat
- TrendMicro.dat
- EncyclopediaN.dat
All of the above files hold detection names that were successfully queried except for EncyclopediaN.dat which holds failed detection name queries. VTTL Currently considers the query failed if a URL is not returned. If caching is enabled, but a SQLite database was not configured, hash lookups and are stored in the following cache sub folders based on the API queried:
- Malshare
- Tc (threat crowd)
- Tg (Threat Grid)
- Vt (VirusTotal)
The data stored in the above folders is simply the return data from the API query stored with the file name of the hash queried. If a cached entry is found an X is placed in the cached column within the spreadsheet to note the data was based on cache. This cache can be configured to update based on age of the cached result which is controlled by intHashCacheThreashold variable within the script.
Watch lists add columns to the spreadsheet output for matches against the watch list.
VirusTotal provides IP addresses associated with a domain and domains associated with IP addresses. To watch for domains or IP addresses during lookups provide the list in IPDwatchlist.txt. Domain lookups that are associated with the IP address will be flagged. IP addresses associated with a flagged domain will be flagged. Example:
zvooogle.info|Vawtrak
wildcardzwincanton.bricks-and-clicks.co.uk|Vawtrak
101.99.20.163|Vawtrak
103.232.215.133|Vawtrak
VirusTotal hash lookup provides vendors' detection names for positive detections against a file. To use this watchlist provide the list of detection names in DNwatchlist.txt. Example:
caphaw
snifula
bedep
vawtrak
neverquest
hancitor
pony
h1n1
shylock
dyre
zeus
zbot
pwszbot
VirusTotal provides the scanned URLs associated with a domain/IP. These can be matched by a substring or RegEx stored in URLwatchlist.txt. Substring example:
/qq/2.txt
/admin.php
RegEx example:
http://([a-fA-F\d]{32})/module/([a-fA-F\d]{32})
http://(?:[0-9]{1,3}.){3}[0-9]{1,3}/module/([a-fA-F\d]{32})
/spreadsheets/\d{8}/96409e35-\d{2}/
This is a watch list that searches AlienVault OTX results for matches to the provided list. The list is located in the current script directory as "\KWordwatchlist.txt". Below is an example list searching for cryptomining related activity:
crypto-mining
cryptomining
crypto mining
cryptocurrency
An error log is written to the same directory as the script named VTTL_Error.log which will note any vendor lookup errors.
Data outputted by VTTL script is logged to files:
- VT_URLs_.txt – Various scan data is logged to this file
Debug:
- VT_IP_.txt – VirusTotal IP address URLs
- VT_TI_.txt – IP and Domain lookup statistics
- VT_Output.txt – Results from the current scan input only
Folders are used to collect debug type information
- Debug folder will contain various debug logs for the web lookups and application debugging.
- Debugging can be turned on via modifying the INI settings
Debugging can be turned on by changing setting BoolDebugTrace = True within the script
- Domain_Reports folder contains report data for VirusTotal domain lookups
- IP_Reports folder contains report data for VirusTotal domain lookups