Merge pull request #110 from krmaxwell/dev

v0.7 release

.gitignore

@@ -52,3 +52,6 @@ maltrieve.out
 archive
 grequests
 *.bak
+*.json
+screenlog*
+README.html

.pre-commit-config.yaml

@@ -0,0 +1,17 @@
+-   repo: git://github.com/pre-commit/pre-commit-hooks
+    sha: master
+    hooks:
+    -   id: autopep8-wrapper
+        args: ['-i', '--ignore=E501']
+    -   id: check-json
+    -   id: check-yaml
+    -   id: end-of-file-fixer
+    -   id: flake8
+        args: [--max-line-length=256]
+    -   id: trailing-whitespace
+
+-   repo: git://github.com/ivanlei/pre-commit-python-sorter
+    sha: master
+    hooks:
+    -   id: python-import-sorter
+        args: [--silent-overwrite, --force_single_line]

README.md

@@ -1,5 +1,9 @@
 [![Stories in Ready](https://badge.waffle.io/krmaxwell/maltrieve.png?label=ready&title=Ready)](https://waffle.io/krmaxwell/maltrieve)
 [![Stories in In Progress](https://badge.waffle.io/krmaxwell/maltrieve.png?label=in%20progress&title=In%20Progress)](https://waffle.io/krmaxwell/maltrieve)
+[![Circle CI](https://circleci.com/gh/krmaxwell/maltrieve/tree/dev.svg?style=svg)](https://circleci.com/gh/krmaxwell/maltrieve/tree/dev)
+[![Coverage Status](https://coveralls.io/repos/krmaxwell/maltrieve/badge.svg?branch=dev)](https://coveralls.io/r/krmaxwell/maltrieve?branch=dev)
+[![Code Health](https://landscape.io/github/krmaxwell/maltrieve/dev/landscape.svg?style=flat)](https://landscape.io/github/krmaxwell/maltrieve/dev)
+
 ```
  _______ _______        _______  ______ _____ _______ _    _ _______
  |  |  | |_____| |         |    |_____/   |   |______  \  /  |______
@@ -12,17 +16,13 @@
 Maltrieve originated as a fork of [mwcrawler](https://github.com/ricardo-dias/mwcrawler). It retrieves malware directly from the sources as listed at a number of sites. Currently we crawl the following:
 
 * [Malc0de](http://malc0de.com/rss)
-* [Malware Black List](http://www.malwareblacklist.com/mbl.xml)
 * [Malware Domain List](http://www.malwaredomainlist.com/hostslist/mdl.xml)
+* [Malware URLs](http://malwareurls.joxeankoret.com/normal.txt)
 * [VX Vault](http://vxvault.siri-urz.net/URL_List.php)
 * [URLquery](http://urlquery.net/)
 * [CleanMX](http://support.clean-mx.de/clean-mx/xmlviruses.php?)
 * [ZeusTracker](https://zeustracker.abuse.ch/monitor.php?urlfeed=binaries)
 
-These lists will be implemented if/when they return to activity.
-
-* [NovCon Minotaur](http://minotauranalysis.com/malwarelist-urls.aspx)
-
 Other improvements include:
 
 * Proxy support
@@ -45,6 +45,8 @@ Maltrieve requires the following dependencies:
 
 With the exception of the Python header files, these can all be found in [requirements.txt](./requirements.txt). On Debian-based distributions, run `sudo apt-get install python-dev`. On Red Hat-based distributions, run `sudo yum install python-devel`. After that, just `pip install -e .`.  You may need to prepend that with ```sudo``` if not running in a virtual environment, but using such an environment is highly encouraged.
 
+Alternately, avoid all of that by using the [Docker image](https://registry.hub.docker.com/u/technoskald/maltrieve/)
+
 ## Usage
 
 __Basic execution:__ `maltrieve` (if installed normally) or ```python maltrieve.py``` (if just downloaded and run)
@@ -63,19 +65,23 @@ optional arguments:
                         Define file for logging progress
   -x, --vxcage          Dump the files to a VxCage instance
   -v, --viper           Dump the files to a Viper instance
+  -r, --crits           Dump the file and domain to a CRITs instance
   -c, --cuckoo          Enable Cuckoo analysis
   -s, --sort_mime       Sort files by MIME type
 
 ```
 
 ### Configuration File
+
 Many of Maltrieve's command line options can be specified in ```maltrieve.cfg```.
 
 ## Automated Execution (Optional)
-Cron can be used to automate the execution of Maltrieve. The following example is provided to help get you started. It will create a cron job that will run Maltrieve every day at 2:01 as a standard user. That said, we recommend enhancing this by creating a custom script for production environments. 
+
+Cron can be used to automate the execution of Maltrieve. The following example is provided to help get you started. It will create a cron job that will run Maltrieve every day at 2:01 as a standard user. That said, we recommend enhancing this by creating a custom script for production environments.
 
 ### Ubuntu
-As a user execute
+
+As a user, execute
 ```
 crontab -e
 ```
@@ -87,22 +93,27 @@ If downloaded to a folder and executed, add the following to the end of the file
 ```
 01 02 * * * cd </folder/location> && /usr/bin/python maltrieve.py <optional flags>
 ```
-### Redhat
-Redhat systems will need to ensure that the user is added to the /etc/cron.allow file. 
+### Red Hat
+
+Red Hat systems will need to ensure that the user is added to the `/etc/cron.allow` file.
+
+
+## Other Tools
+
+Maltrieve doesn't do analysis. In addition to the integrations listed above, we can recommend using [VirusTotalApi](https://github.com/doomedraven/VirusTotalApi) for working with [VirusTotal](https://www.virustotal.com). [Malwr](https://malwr.com) is a similar site based on Cuckoo Sandbox.
 
 ## License
 
 Released under GPL version 3. See the [LICENSE](./LICENSE) file for full details.
 
-
 ## Known bugs
 
 We list all the bugs we know about (plus some things we know we need to add) at the [GitHub issues](https://github.com/krmaxwell/maltrieve/issues) page.
 
 
 ## How you can help
 
-Aside from pull requests, non-developers can open issues on [Github](https://github.com/krmaxwell/maltrieve). Things we'd really appreciate:
+Aside from pull requests, non-developers can open issues on [GitHub](https://github.com/krmaxwell/maltrieve). Things we'd really appreciate:
 
 * Bug reports, preferably with error logs
 * Suggestions of additional sources for malware lists

circle.yml

@@ -0,0 +1,4 @@
+test:
+  override:
+    - py.test --cov maltrieve test.py
+    - coveralls

docker/Dockerfile

@@ -1,47 +1,49 @@
 #
 # This Docker image encapsulates Maltrieve, a tool to retrieve malware
 # directly from the source for security researchers.
-# which was created by Kyle Maxwell (technoskald) and is
-# available at https://github.com/technoskald/maltrieve.
+# which was created by Kyle Maxwell (krmaxwell) and is
+# available at https://github.com/krmaxwell/maltrieve.
 #
 # The file below is based on ideas from Spenser Reinhardt's Dockerfile
 # (https://registry.hub.docker.com/u/sreinhardt/honeynet/dockerfile)
 # and on instructions outlined by M. Fields (@shakey_1).
 #
 # To run this image after installing Docker, use a command like this:
 #
-# sudo docker run --rm -it technoskald/maltrieve bash
-#
-# then run ./maltrieve.py with the desired parameters.
+# sudo docker run --rm -it technoskald/maltrieve
 
 FROM ubuntu:14.04
 MAINTAINER Michael Boman <[email protected]>
 
 USER root
 RUN apt-get update && \
-  apt-get install -y --no-install-recommends \
+  apt-get dist-upgrade -y
+RUN apt-get install -y --no-install-recommends \
     gcc \
     git \
     libpython2.7-stdlib \
     python2.7 \
     python2.7-dev \
     python-pip \
-    python-setuptools && \
-
-  rm -rf /var/lib/apt/lists/* && \
-
+    python-setuptools
+RUN rm -rf /var/lib/apt/lists/* && \
+  pip install --upgrade pip && \
   groupadd -r maltrieve && \
   useradd -r -g maltrieve -d /home/maltrieve -s /sbin/nologin -c "Maltrieve User" maltrieve
 
 WORKDIR /home
-RUN git clone https://github.com/technoskald/maltrieve.git && \
+RUN git clone https://github.com/krmaxwell/maltrieve.git && \
   cd maltrieve && \
-  pip install -r requirements.txt && \
+  git checkout dev && \
+  pip install -e . && \
   chown -R maltrieve:maltrieve /home/maltrieve
 
+RUN mkdir /archive && \
+  chown maltrieve:maltrieve /archive
+
 USER maltrieve
 ENV HOME /home/maltrieve
 ENV USER maltrieve
 WORKDIR /home/maltrieve
-CMD ["./maltrieve.py"]
-
+ENTRYPOINT ["maltrieve"]
+CMD ["-d", "/archive/samples", "-l", "/archive/maltrieve.log"]

maltrieve-test.cfg

@@ -0,0 +1,18 @@
+[Maltrieve]
+dumpdir = archive-test
+logfile = maltrieve-test.log
+logheaders = true
+User-Agent = Test-Agent
+proxy = 127.0.0.1:3128
+
+#viper = http://127.0.0.1:8080
+#cuckoo = http://127.0.0.1:8090
+#vxcage = http://127.0.0.1:8080
+crits = http://127.0.0.1:8080
+crits_user = maltrieve
+crits_key = YOUR_API_KEY_HERE
+crits_source = maltrieve
+
+# Filter Lists are based on mime type NO SPACE BETWEEN ,
+black_list = text/html,text/plain
+white_list = application/pdf,application/x-dosexec

maltrieve.cfg

@@ -7,7 +7,10 @@ User-Agent = Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)
 #viper = http://127.0.0.1:8080
 #cuckoo = http://127.0.0.1:8090
 #vxcage = http://127.0.0.1:8080
-
+#crits = https://127.0.0.1
+#crits_user = maltrieve
+#crits_key = <api_key>
+#crits_source = maltrieve
 
 # Filter Lists are based on mime type NO SPACE BETWEEN ,
 #black_list = text/html,text/plain