Skip to content

Commit

Permalink
Merge branch 'release/2.6.0'
Browse files Browse the repository at this point in the history
  • Loading branch information
fedelemantuano committed Sep 15, 2018
2 parents 8bb98d0 + 620f8e4 commit 66732e6
Show file tree
Hide file tree
Showing 26 changed files with 927 additions and 116 deletions.
3 changes: 1 addition & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@ python:
- "2.7"

env:
- TIKA_VER=1.16
TIKA_APP_JAR=/tmp/tika-app-${TIKA_VER}.jar
- TIKA_APP_JAR=/tmp/tika-app-${TIKA_VER}.jar
FAUP_PATH=/tmp/faup
ZEMANA_PATH=/tmp/zemana
DOCKER_ELASTICSEARCH_PATH=/tmp/docker-elasticsearch
Expand Down
29 changes: 29 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,34 @@ SpamScope has some topologies in [topologies folder](./topologies/), but you can

![Schema topology](docs/images/schema_topology.png?raw=true "Schema topology")

# What Does SpamScope do?
SpamScope gets the raw emails (both RFC822 and Outlook formats) in input and returns an JSON object. Then it extracts urls and attachments (if they are zipped extracts the content files). All informations are saved in JSON objects. This is the first analysis. After that SpamScope runs a _phishing_ module, that gives a _phishing score_ to the emails.

Then you can enable/disable post processing modules, that connect SpamScope with third party tools. There are three main categories:
- raw emails analysis
- attachments analysis
- sender emails analysis

It's possible to add new modules in these three categories, if you want connect SpamScope with others tools.

## Raw emails analysis
These modules (see [here](./src/modules/mails)) analyze the raw emails:
- SMTP dialect
- SpamAssassin

## Attachments analysis
These modules (see [here](./src/modules/attachments)) analyze the attachments of emails:
- Apache Tika
- Store sample on disk (as default SpamScope saves samples in JSON objects)
- Thug
- VirusTotal
- Zemana

## Sender emails analysis
SpamScope can detects the exact sender IP and then it can analyze it (see [here](./src/modules/networks)):
- Shodan
- VirusTotal

# Why should I use SpamScope
- It's very fast: the job is splitted in functionalities that work in parallel.
- It's flexible: you can choose what SpamScope has to do.
Expand Down Expand Up @@ -64,6 +92,7 @@ SpamScope can be downloaded, used, and modified free of charge. It is available
- [The Honeynet Project](http://honeynet.org/node/1329)
- [securityonline.info](http://securityonline.info/pcileech-direct-memory-access-dma-attack-software/)
- [jekil/awesome-hacking](https://github.com/jekil/awesome-hacking)
- [Linux Security Expert](https://linuxsecurity.expert/tools/spamscope/)

# Authors

Expand Down
135 changes: 82 additions & 53 deletions README.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
|PyPI version| |Build Status| |Coverage Status| |BCH compliance|
`PyPI version <https://badge.fury.io/py/SpamScope>`__ `Build
Status <https://travis-ci.org/SpamScope/spamscope>`__ `Coverage
Status <https://coveralls.io/github/SpamScope/spamscope?branch=develop>`__
`BCH compliance <https://bettercodehub.com/>`__

.. figure:: https://raw.githubusercontent.com/SpamScope/spamscope/develop/docs/logo/spamscope.png
:alt: SpamScope
Expand Down Expand Up @@ -26,20 +29,55 @@ folder <./topologies/>`__, but you can make others topologies.

Schema topology

What Does SpamScope do?
=======================

SpamScope gets the raw emails (both RFC822 and Outlook formats) in input
and returns an JSON object. Then it extracts urls and attachments (if
they are zipped extracts the content files). All informations are saved
in JSON objects. This is the first analysis. After that SpamScope runs a
*phishing* module, that gives a *phishing score* to the emails.

Then you can enable/disable post processing modules, that connect
SpamScope with third party tools. There are three main categories: - raw
emails analysis - attachments analysis - sender emails analysis

It’s possible to add new modules in these three categories, if you want
connect SpamScope with others tools.

Raw emails analysis
-------------------

These modules (see `here <./src/modules/mails>`__) analyze the raw
emails: - SMTP dialect - SpamAssassin

Attachments analysis
--------------------

These modules (see `here <./src/modules/attachments>`__) analyze the
attachments of emails: - Apache Tika - Store sample on disk (as default
SpamScope saves samples in JSON objects) - Thug - VirusTotal - Zemana

Sender emails analysis
----------------------

SpamScope can detects the exact sender IP and then it can analyze it
(see `here <./src/modules/networks>`__): - Shodan - VirusTotal

Why should I use SpamScope
==========================

- It's very fast: the job is splitted in functionalities that work in
- Its very fast: the job is splitted in functionalities that work in
parallel.
- It's flexible: you can choose what SpamScope has to do.
- It's distributed: SpamScope uses Apache Storm, free and open source
- Its flexible: you can choose what SpamScope has to do.
- Its distributed: SpamScope uses Apache Storm, free and open source
distributed realtime computation system.
- It makes JSON output that you can save where you want.
- It's easy to setup: there are docker images and docker-compose ready
- Its easy to setup: there are docker images and docker-compose ready
for use.
- It's integrated with Apache Tika, VirusTotal, Thug, Shodan and
- Its integrated with Apache Tika, VirusTotal, Thug, Shodan and
SpamAssassin (for now).
- It's free and open source (for special functions you can contact me).
- Its free and open source (for special functions you can contact me).
- It can analyze Outlook msg.

Distributed
Expand All @@ -58,12 +96,12 @@ SpamScope comes with the following bolts: - **tokenizer** splits mail in
token like headers, body, attachments and it can filter emails,
attachments and ip addresses already seen - **phishing** looks for your
keywords in email and connects email to targets (bank, your customers,
etc.) - **raw\_mail** is for all third party tools that analyze raw
mails like SpamAssassin - **attachments** analyzes all mail attachments
and uses third party tools like VirusTotal - **network** analyzes all
sender ip addresses with third party tools like Shodan - **urls**
extracts all urls in email and attachments - **json\_maker** and
**outputs** make the json report and save it
etc.) - **raw_mail** is for all third party tools that analyze raw mails
like SpamAssassin - **attachments** analyzes all mail attachments and
uses third party tools like VirusTotal - **network** analyzes all sender
ip addresses with third party tools like Shodan - **urls** extracts all
urls in email and attachments - **json_maker** and **outputs** make the
json report and save it

Store where you want
--------------------
Expand All @@ -80,21 +118,21 @@ and/or remove spouts and bolts.
API
---

For now SpamScope doesn't have its own API, because it isn't tied to any
tecnology. If you use ``Redis`` as spout (input), you'll use Redis API
to put mails in topology. If you use ``Elasticsearch`` as output, you'll
For now SpamScope doesnt have its own API, because it isnt tied to any
tecnology. If you use ``Redis`` as spout (input), youll use Redis API
to put mails in topology. If you use ``Elasticsearch`` as output, youll
use Elasticsearch API to get results.

It's possible to develop a middleware API that it talks with input,
output and changes the configuration, but now there isn't.
Its possible to develop a middleware API that it talks with input,
output and changes the configuration, but now there isnt.

Apache 2 Open Source License
============================

SpamScope can be downloaded, used, and modified free of charge. It is
available under the Apache 2 license.

|Donate|
`Donate <https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=VEPXYP745KJF2>`__

SpamScope on Web
================
Expand All @@ -104,6 +142,8 @@ SpamScope on Web
- `The Honeynet Project <http://honeynet.org/node/1329>`__
- `securityonline.info <http://securityonline.info/pcileech-direct-memory-access-dma-attack-software/>`__
- `jekil/awesome-hacking <https://github.com/jekil/awesome-hacking>`__
- `Linux Security
Expert <https://linuxsecurity.expert/tools/spamscope/>`__

Authors
=======
Expand All @@ -124,7 +164,7 @@ For Python requirements you can read: \* `mandatory
requirements <./requirements.txt>`__ \* `optional
requirements <./requirements_optional.txt>`__

*Thug* is another optional requirement, that it's not in requirements.
*Thug* is another optional requirement, that its not in requirements.
See `Thug section <#thug-optional>`__ for more details.

Apache Storm
Expand Down Expand Up @@ -197,32 +237,32 @@ errors in Apache Storm. To avoid any issue set

::

nr. user agents * timeout_thug < supervisor.worker.timeout.secs
nr. user agents * timeout_thug < supervisor.worker.timeout.secs

The best value for ``threshold`` is 1.

VirusTotal (optional)
---------------------

It's possible add to results (for mail attachments and sender ip
Its possible add to results (for mail attachments and sender ip
address) the VirusTotal report. You need a private API key.

Shodan (optional)
-----------------

It's possible add to results the Shodan report for sender ip address.
Its possible add to results the Shodan report for sender ip address.
You need a private API key.

Elasticsearch (optional)
------------------------

It's possible to store the results in Elasticsearch. In this case you
Its possible to store the results in Elasticsearch. In this case you
should install ``elasticsearch`` package.

Redis (optional)
----------------

It's possible to store the results in Redis. In this case you should
Its possible to store the results in Redis. In this case you should
install ``redis`` package.

Configuration
Expand All @@ -231,12 +271,12 @@ Configuration
Read the `example of main configuration
file <./conf/spamscope.example.yml>`__. The default value where
SpamScope will search the configuration file is
``/etc/spamscope/spamscope.yml``, but it's possible to set the
``/etc/spamscope/spamscope.yml``, but its possible to set the
environment variable ``SPAMSCOPE_CONF_FILE``:

::

$ export SPAMSCOPE_CONF_FILE=/etc/spamscope/spamscope.yml
$ export SPAMSCOPE_CONF_FILE=/etc/spamscope/spamscope.yml

When you change the configuration file, SpamScope automatically reloads
the new changes.
Expand All @@ -252,21 +292,21 @@ Topologies
==========

SpamScope comes with three topologies: -
`spamscope\_debug <./topologies/spamscope_debug.py>`__: the output are
`spamscope_debug <./topologies/spamscope_debug.py>`__: the output are
JSON files on file system. -
`spamscope\_elasticsearch <./topologies/spamscope_elasticsearch.py>`__:
`spamscope_elasticsearch <./topologies/spamscope_elasticsearch.py>`__:
the output are stored in Elasticsearch indexes. -
`spamscope\_redis <./topologies/spamscope_redis.py>`__: the output are
`spamscope_redis <./topologies/spamscope_redis.py>`__: the output are
stored in Redis.

If you want submit SpamScope topology use ``spamscope-topology submit``
tool. For more details `see SpamScope cli tools <src/cli/README.md>`__:

::

$ spamscope-topology submit --topology {spamscope_debug,spamscope_elasticsearch,spamscope_redis}
$ spamscope-topology submit --topology {spamscope_debug,spamscope_elasticsearch,spamscope_redis}

It's possible to change the default settings for all Apache Storm
Its possible to change the default settings for all Apache Storm
options. I suggest to change these options:

- **topology.tick.tuple.freq.secs**: reload configuration of all bolts
Expand Down Expand Up @@ -297,16 +337,16 @@ variables:

::

$ export THUG_ENABLED=True
$ export VIRUSTOTAL_ENABLED=True
$ export VIRUSTOTAL_APIKEY="your key"
$ export ZEMANA_ENABLED=True
$ export ZEMANA_APIKEY="your key"
$ export ZEMANA_PARTNERID="your partner id"
$ export ZEMANA_USERID="your userid"
$ export SHODAN_ENABLED=True
$ export SHODAN_APIKEY="your key"
$ export SPAMASSASSIN_ENABLED=True
$ export THUG_ENABLED=True
$ export VIRUSTOTAL_ENABLED=True
$ export VIRUSTOTAL_APIKEY="your key"
$ export ZEMANA_ENABLED=True
$ export ZEMANA_APIKEY="your key"
$ export ZEMANA_PARTNERID="your partner id"
$ export ZEMANA_USERID="your userid"
$ export SHODAN_ENABLED=True
$ export SHODAN_APIKEY="your key"
$ export SPAMASSASSIN_ENABLED=True

Output example
==============
Expand Down Expand Up @@ -338,14 +378,3 @@ Screenshots
:alt: SpamScope Map

SpamScope Map

.. |PyPI version| image:: https://badge.fury.io/py/SpamScope.svg
:target: https://badge.fury.io/py/SpamScope
.. |Build Status| image:: https://travis-ci.org/SpamScope/spamscope.svg?branch=master
:target: https://travis-ci.org/SpamScope/spamscope
.. |Coverage Status| image:: https://coveralls.io/repos/github/SpamScope/spamscope/badge.svg?branch=develop
:target: https://coveralls.io/github/SpamScope/spamscope?branch=develop
.. |BCH compliance| image:: https://bettercodehub.com/edge/badge/SpamScope/spamscope?branch=develop
:target: https://bettercodehub.com/
.. |Donate| image:: https://www.paypal.com/en_US/i/btn/btn_donateCC_LG.gif
:target: https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=VEPXYP745KJF2
68 changes: 68 additions & 0 deletions conf/logstash/conf.d/postfix.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
input {
file {
path => [ "/postfix_logs/maillog" ]
tags => [ "postfix" ]
}
}

filter {
if "postfix" in [tags] {
grok {
match => [ "message", "%{SYSLOGLINE}" ]
overwrite => [ "message" ]
}
date {
match => [ "timestamp8601", "ISO8601" ]
remove_field => [ "timestamp8601" ]
}
if [message] =~ /message-id/ {
grok {
patterns_dir => [ "/usr/share/logstash/patterns" ]
match => ["message", "%{POSTFIX_CODE}"]
add_tag => ["message_id"]
}
} else if [message] =~ /client=/ {
grok {
patterns_dir => [ "/usr/share/logstash/patterns" ]
match => ["message", "%{POSTFIX_CLIENT}"]
add_tag => ["client", "code"]
}
} else if [message] =~ /^> / {
grok {
patterns_dir => [ "/usr/share/logstash/patterns" ]
match => ["message", "%{POSTFIX_DIALECT}"]
add_tag => ["client", "dialect"]
add_field => { "actor" => "server" }
}
} else if [message] =~ /^< / {
grok {
patterns_dir => [ "/usr/share/logstash/patterns" ]
match => ["message", "%{POSTFIX_DIALECT}"]
add_tag => ["client", "dialect"]
add_field => { "actor" => "client" }
}
} else if [message] =~ /^connect / {
grok {
patterns_dir => [ "/usr/share/logstash/patterns" ]
match => ["message", "%{POSTFIX_CONNECT}"]
add_tag => ["client", "connect"]
}
} else if [message] =~ /^disconnect / {
grok {
patterns_dir => [ "/usr/share/logstash/patterns" ]
match => ["message", "%{POSTFIX_DISCONNECT}"]
add_tag => ["client", "disconnect"]
}
}
}
}

output {
if "postfix" in [tags] {
elasticsearch {
hosts => [ "elasticsearch" ]
index => "postfix-%{+YYYY.MM.dd}"
document_type => "_doc"
}
}
}
5 changes: 5 additions & 0 deletions conf/logstash/patterns/postfix
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
POSTFIX_CODE %{NOTSPACE:code}:\s+message-id=%{NOTSPACE:message_id}
POSTFIX_CLIENT %{NOTSPACE:code}:\s+client=%{NOTSPACE:client_name}\[%{NOTSPACE:client_ip}\]
POSTFIX_DIALECT (>|<)\s+%{NOTSPACE:client_name}\[%{NOTSPACE:client_ip}\]:\s+%{GREEDYDATA:dialect}
POSTFIX_CONNECT connect\sfrom\s%{NOTSPACE:client_name}\[%{NOTSPACE:client_ip}\]
POSTFIX_DISCONNECT disconnect\sfrom\s%{NOTSPACE:client_name}\[%{NOTSPACE:client_ip}\]\s%{GREEDYDATA}
Loading

0 comments on commit 66732e6

Please sign in to comment.