diff --git a/README.md b/README.md index c55c649..16d7133 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,8 @@ ThePhish is an automated phishing email analysis tool based on [TheHive](https://github.com/TheHive-Project/TheHive), [Cortex](https://github.com/TheHive-Project/Cortex/) and [MISP](https://github.com/MISP/MISP). It is a web application written in Python 3 and based on Flask that automates the entire analysis process starting from the extraction of the observables from the header and the body of an email to the elaboration of a verdict which is final in most cases. In addition, it allows the analyst to intervene in the analysis process and obtain further details on the email being analyzed if necessary. In order to interact with TheHive and Cortex, it uses [TheHive4py](https://github.com/TheHive-Project/TheHive4py) and [Cortex4py](https://github.com/TheHive-Project/Cortex4py), which are the Python API clients that allow using the REST APIs made available by TheHive and Cortex respectively. -![OS](https://img.shields.io/badge/OS-Linux-red?style=flat&logo=linux) +![Linux](https://img.shields.io/badge/Linux-FCC624?style=for-the-badge&logo=linux&logoColor=black) +![Windows](https://img.shields.io/badge/Windows-0078D6?style=for-the-badge&logo=windows&logoColor=white)
[![made-with-python](https://img.shields.io/badge/Made%20with-Python%203.8-1f425f.svg?logo=python)](https://www.python.org/) [![Docker](https://img.shields.io/badge/Docker-available-green.svg?style=flat&logo=docker)](https://github.com/emalderson/ThePhish/tree/master/docker) [![Maintenance](https://img.shields.io/badge/Maintained-yes-green.svg)](https://github.com/emalderson/ThePhish) @@ -166,7 +167,7 @@ ThePhish interacts with TheHive and Cortex thanks to TheHive4py and Cortex4py. M Since the installation and configuration of TheHive, Cortex and MISP services from scratch for a production environment may not be extremely straightforward, TheHive Project provides Docker images and Docker Compose templates [here](https://github.com/TheHive-Project/Docker-Templates) to facilitate the installation procedure. For the sake of simplicity, the provided templates are made simple, without providing the full configuration options of each docker image. -If you only want to try ThePhish or you want to have it up and running as fast as possible, you can use the provided Docker Template in the `docker` folder, which is a modified version of one of the Docker Templates provided by TheHive Project that also allows creating a ThePhish container. To install ThePhish using Docker and Docker Compose, please refer to [this guide](https://github.com/emalderson/ThePhish/tree/master/docker). I strongly recommend that you install it this way at least the first time you use it so that you can learn the basics and how to configure it with a minimal configuration that should work on the first try. Indeed, the previously linked guide also provides a step-by-step procedure to configure the TheHive, Cortex and MISP instances. +If you only want to try ThePhish or you want to have it up and running as fast as possible, you can use the provided Docker Template in the `docker` folder, which is a modified version of one of the Docker Templates provided by TheHive Project that also allows creating a ThePhish container. To install ThePhish using Docker and Docker Compose, please refer to [this guide](https://github.com/asterictnl-lvdw/ThePhish/tree/master/docker). I strongly recommend that you install it this way at least the first time you use it so that you can learn the basics and how to configure it with a minimal configuration that should work on the first try. Indeed, the previously linked guide also provides a step-by-step procedure to configure the TheHive, Cortex and MISP instances. ### Install it from scratch @@ -175,7 +176,7 @@ This guide refers to the sole installation of ThePhish, which requires: - An up-and-running instance of Cortex - An up-and-running instance of MISP - An email address that users can use to send emails to ThePhish - - A Linux-based OS with Python 3.8+ installed + - A Linux-based OS or a Windows-based OS with Python 3.8+ installed In order to install, configure and integrate TheHive, Cortex and MISP instances, please refer to their official documentation: - [TheHive documentation](https://docs.thehive-project.org/thehive/) @@ -184,13 +185,13 @@ In order to install, configure and integrate TheHive, Cortex and MISP instances, It is advisable that the email address from which ThePhish fetches the emails to analyze be a Gmail address since it is the one with which ThePhish has been tested the most. It is preferable that the account is a newly created one, with the sole purpose of being used by ThePhish. The procedure to activate the app password that is required by ThePhish to connect to the mailbox and fetch the emails is explained [here](https://support.google.com/accounts/answer/185833?hl=en). -This installation procedure has been tested on a VM running Ubuntu 20.04.3 LTS with Python 3.8 installed and the versions of TheHive, Cortex and MISP shown in this [docker-compose.yml](https://github.com/emalderson/ThePhish/blob/master/docker/docker-compose.yml) file. +This installation has been tested with both the Windows and Linux [docker-compose.yml](https://github.com/asterictnl-lvdw/ThePhish/tree/master/docker) file. They have been tested on a Windows 11 machine, and on Kali Linux, Ubuntu and Debian with at least Python 3.8+ installed. Once TheHive, Cortex and MISP are configured and listening at a certain URL and the email address is ready to use, you can install and configure ThePhish. 1. Clone the repository ``` - $ git clone https://github.com/emalderson/ThePhish.git + $ git clone https://github.com/asterictnl-lvdw/ThePhish.git ``` 2. Create a Python virtual environment and activate it (it is good practice but it is not required) ``` @@ -393,7 +394,7 @@ The analyzers emphasized in *italic* are the ones for which the levels have been ### Enable the *MISP* analyzer -In order to integrate Cortex with MISP, you must activate the *MISP_2_1* analyzer and configure it with the authentication key of the user created on MISP that Cortex will use to interact with MISP. This means that an organization and a user with `sync_user` role in that organization must be created on MISP beforehand (you can learn how to do that and obtain the authentication key [here (ThePhish documentation, recommended)](https://github.com/emalderson/ThePhish/tree/master/docker#configure-the-misp-container) or [here (MISP documentation)](https://www.circl.lu/doc/misp/administration/#users). +In order to integrate Cortex with MISP, you must activate the *MISP_2_1* analyzer and configure it with the authentication key of the user created on MISP that Cortex will use to interact with MISP. This means that an organization and a user with `sync_user` role in that organization must be created on MISP beforehand (you can learn how to do that and obtain the authentication key [here (ThePhish documentation, recommended)](https://github.com/asterictnl-lvdw/ThePhish/tree/master/docker#configure-the-misp-container) or [here (MISP documentation)](https://www.circl.lu/doc/misp/administration/#users). ### Enable the *Yara* analyzer diff --git a/docker/cortex/application.conf b/docker/Linux/Latest/cortex/application.conf similarity index 96% rename from docker/cortex/application.conf rename to docker/Linux/Latest/cortex/application.conf index 5c6cac7..6236c81 100644 --- a/docker/cortex/application.conf +++ b/docker/Linux/Latest/cortex/application.conf @@ -1,217 +1,217 @@ -# Sample Cortex application.conf file - -## SECRET KEY -# -# The secret key is used to secure cryptographic functions. -# -# IMPORTANT: If you deploy your application to several instances, make -# sure to use the same key. -play.http.secret.key="msd3232fdn3ofgfbki83ihtzHSD" - -## ElasticSearch -search { - # Name of the index - index = cortex - # ElasticSearch instance address. - # For cluster, join address:port with ',': "http://ip1:9200,ip2:9200,ip3:9200" - uri = "http://elasticsearch:9200" - - ## Advanced configuration - # Scroll keepalive. - #keepalive = 1m - # Scroll page size. - #pagesize = 50 - # Number of shards - #nbshards = 5 - # Number of replicas - #nbreplicas = 1 - # Arbitrary settings - #settings { - # # Maximum number of nested fields - # mapping.nested_fields.limit = 100 - #} - - ## Authentication configuration - #search.username = "" - #search.password = "" - - ## SSL configuration - #search.keyStore { - # path = "/path/to/keystore" - # type = "JKS" # or PKCS12 - # password = "keystore-password" - #} - #search.trustStore { - # path = "/path/to/trustStore" - # type = "JKS" # or PKCS12 - # password = "trustStore-password" - #} -} - -## Cache -# -# If an analyzer is executed against the same observable, the previous report can be returned without re-executing the -# analyzer. The cache is used only if the second job occurs within cache.job (the default is 10 minutes). -cache.job = 10 minutes - -## Authentication -auth { - # "provider" parameter contains the authentication provider(s). It can be multi-valued, which is useful - # for migration. - # The available auth types are: - # - services.LocalAuthSrv : passwords are stored in the user entity within ElasticSearch). No - # configuration are required. - # - ad : use ActiveDirectory to authenticate users. The associated configuration shall be done in - # the "ad" section below. - # - ldap : use LDAP to authenticate users. The associated configuration shall be done in the - # "ldap" section below. - # - oauth2 : use OAuth/OIDC to authenticate users. Configuration is under "auth.oauth2" and "auth.sso" keys - provider = [local] - - ad { - # The Windows domain name in DNS format. This parameter is required if you do not use - # 'serverNames' below. - #domainFQDN = "mydomain.local" - - # Optionally you can specify the host names of the domain controllers instead of using 'domainFQDN - # above. If this parameter is not set, TheHive uses 'domainFQDN'. - #serverNames = [ad1.mydomain.local, ad2.mydomain.local] - - # The Windows domain name using short format. This parameter is required. - #domainName = "MYDOMAIN" - - # If 'true', use SSL to connect to the domain controller. - #useSSL = true - } - - ldap { - # The LDAP server name or address. The port can be specified using the 'host:port' - # syntax. This parameter is required if you don't use 'serverNames' below. - #serverName = "ldap.mydomain.local:389" - - # If you have multiple LDAP servers, use the multi-valued setting 'serverNames' instead. - #serverNames = [ldap1.mydomain.local, ldap2.mydomain.local] - - # Account to use to bind to the LDAP server. This parameter is required. - #bindDN = "cn=thehive,ou=services,dc=mydomain,dc=local" - - # Password of the binding account. This parameter is required. - #bindPW = "***secret*password***" - - # Base DN to search users. This parameter is required. - #baseDN = "ou=users,dc=mydomain,dc=local" - - # Filter to search user in the directory server. Please note that {0} is replaced - # by the actual user name. This parameter is required. - #filter = "(cn={0})" - - # If 'true', use SSL to connect to the LDAP directory server. - #useSSL = true - } - - oauth2 { - # URL of the authorization server - #clientId = "client-id" - #clientSecret = "client-secret" - #redirectUri = "https://my-thehive-instance.example/index.html#!/login" - #responseType = "code" - #grantType = "authorization_code" - - # URL from where to get the access token - #authorizationUrl = "https://auth-site.com/OAuth/Authorize" - #tokenUrl = "https://auth-site.com/OAuth/Token" - - # The endpoint from which to obtain user details using the OAuth token, after successful login - #userUrl = "https://auth-site.com/api/User" - #scope = "openid profile" - # Type of authorization header - #authorizationHeader = "Bearer" # or token - } - - # Single-Sign On - sso { - # Autocreate user in database? - #autocreate = false - - # Autoupdate its profile and roles? - #autoupdate = false - - # Autologin user using SSO? - #autologin = false - - # Attributes mappings - #attributes { - # login = "login" - # name = "name" - # groups = "groups" - # roles = "roles" # list of roles, separated with comma - # organisation = "org" - #} - - # Name of mapping class from user resource to backend user ('simple' or 'group') - #mapper = group - # Default roles for users with no groups mapped ("read", "analyze", "orgadmin") - #defaultRoles = [] - # Default organization - #defaultOrganization = "MyOrga" - - #groups { - # # URL to retreive groups (leave empty if you are using OIDC) - # #url = "https://auth-site.com/api/Groups" - # # Group mappings, you can have multiple roles for each group: they are merged - # mappings { - # admin-profile-name = ["admin"] - # editor-profile-name = ["write"] - # reader-profile-name = ["read"] - # } - #} - } -} - -job { - runner = [docker] -} -## ANALYZERS -# -analyzer { - # analyzer location - # url can be point to: - # - directory where analyzers are installed - # - json file containing the list of analyzer descriptions - urls = [ - "https://download.thehive-project.org/analyzers.json" - #"/absolute/path/of/analyzers" - ] - - # Sane defaults. Do not change unless you know what you are doing. - fork-join-executor { - # Min number of threads available for analysis. - parallelism-min = 2 - # Parallelism (threads) ... ceil(available processors * factor). - parallelism-factor = 2.0 - # Max number of threads available for analysis. - parallelism-max = 4 - } -} - -# RESPONDERS -# -responder { - # responder location (same format as analyzer.urls) - urls = [ - "https://download.thehive-project.org/responders.json" - #"/absolute/path/of/responders" - ] - - # Sane defaults. Do not change unless you know what you are doing. - fork-join-executor { - # Min number of threads available for analysis. - parallelism-min = 2 - # Parallelism (threads) ... ceil(available processors * factor). - parallelism-factor = 2.0 - # Max number of threads available for analysis. - parallelism-max = 4 - } -} - -# It's the end my friend. Happy hunting! +# Sample Cortex application.conf file + +## SECRET KEY +# +# The secret key is used to secure cryptographic functions. +# +# IMPORTANT: If you deploy your application to several instances, make +# sure to use the same key. +play.http.secret.key="msd3232fdn3ofgfbki83ihtzHSD" + +## ElasticSearch +search { + # Name of the index + index = cortex + # ElasticSearch instance address. + # For cluster, join address:port with ',': "http://ip1:9200,ip2:9200,ip3:9200" + uri = "http://elasticsearch:9200" + + ## Advanced configuration + # Scroll keepalive. + #keepalive = 1m + # Scroll page size. + #pagesize = 50 + # Number of shards + #nbshards = 5 + # Number of replicas + #nbreplicas = 1 + # Arbitrary settings + #settings { + # # Maximum number of nested fields + # mapping.nested_fields.limit = 100 + #} + + ## Authentication configuration + #search.username = "" + #search.password = "" + + ## SSL configuration + #search.keyStore { + # path = "/path/to/keystore" + # type = "JKS" # or PKCS12 + # password = "keystore-password" + #} + #search.trustStore { + # path = "/path/to/trustStore" + # type = "JKS" # or PKCS12 + # password = "trustStore-password" + #} +} + +## Cache +# +# If an analyzer is executed against the same observable, the previous report can be returned without re-executing the +# analyzer. The cache is used only if the second job occurs within cache.job (the default is 10 minutes). +cache.job = 10 minutes + +## Authentication +auth { + # "provider" parameter contains the authentication provider(s). It can be multi-valued, which is useful + # for migration. + # The available auth types are: + # - services.LocalAuthSrv : passwords are stored in the user entity within ElasticSearch). No + # configuration are required. + # - ad : use ActiveDirectory to authenticate users. The associated configuration shall be done in + # the "ad" section below. + # - ldap : use LDAP to authenticate users. The associated configuration shall be done in the + # "ldap" section below. + # - oauth2 : use OAuth/OIDC to authenticate users. Configuration is under "auth.oauth2" and "auth.sso" keys + provider = [local] + + ad { + # The Windows domain name in DNS format. This parameter is required if you do not use + # 'serverNames' below. + #domainFQDN = "mydomain.local" + + # Optionally you can specify the host names of the domain controllers instead of using 'domainFQDN + # above. If this parameter is not set, TheHive uses 'domainFQDN'. + #serverNames = [ad1.mydomain.local, ad2.mydomain.local] + + # The Windows domain name using short format. This parameter is required. + #domainName = "MYDOMAIN" + + # If 'true', use SSL to connect to the domain controller. + #useSSL = true + } + + ldap { + # The LDAP server name or address. The port can be specified using the 'host:port' + # syntax. This parameter is required if you don't use 'serverNames' below. + #serverName = "ldap.mydomain.local:389" + + # If you have multiple LDAP servers, use the multi-valued setting 'serverNames' instead. + #serverNames = [ldap1.mydomain.local, ldap2.mydomain.local] + + # Account to use to bind to the LDAP server. This parameter is required. + #bindDN = "cn=thehive,ou=services,dc=mydomain,dc=local" + + # Password of the binding account. This parameter is required. + #bindPW = "***secret*password***" + + # Base DN to search users. This parameter is required. + #baseDN = "ou=users,dc=mydomain,dc=local" + + # Filter to search user in the directory server. Please note that {0} is replaced + # by the actual user name. This parameter is required. + #filter = "(cn={0})" + + # If 'true', use SSL to connect to the LDAP directory server. + #useSSL = true + } + + oauth2 { + # URL of the authorization server + #clientId = "client-id" + #clientSecret = "client-secret" + #redirectUri = "https://my-thehive-instance.example/index.html#!/login" + #responseType = "code" + #grantType = "authorization_code" + + # URL from where to get the access token + #authorizationUrl = "https://auth-site.com/OAuth/Authorize" + #tokenUrl = "https://auth-site.com/OAuth/Token" + + # The endpoint from which to obtain user details using the OAuth token, after successful login + #userUrl = "https://auth-site.com/api/User" + #scope = "openid profile" + # Type of authorization header + #authorizationHeader = "Bearer" # or token + } + + # Single-Sign On + sso { + # Autocreate user in database? + #autocreate = false + + # Autoupdate its profile and roles? + #autoupdate = false + + # Autologin user using SSO? + #autologin = false + + # Attributes mappings + #attributes { + # login = "login" + # name = "name" + # groups = "groups" + # roles = "roles" # list of roles, separated with comma + # organisation = "org" + #} + + # Name of mapping class from user resource to backend user ('simple' or 'group') + #mapper = group + # Default roles for users with no groups mapped ("read", "analyze", "orgadmin") + #defaultRoles = [] + # Default organization + #defaultOrganization = "MyOrga" + + #groups { + # # URL to retreive groups (leave empty if you are using OIDC) + # #url = "https://auth-site.com/api/Groups" + # # Group mappings, you can have multiple roles for each group: they are merged + # mappings { + # admin-profile-name = ["admin"] + # editor-profile-name = ["write"] + # reader-profile-name = ["read"] + # } + #} + } +} + +job { + runner = [docker] +} +## ANALYZERS +# +analyzer { + # analyzer location + # url can be point to: + # - directory where analyzers are installed + # - json file containing the list of analyzer descriptions + urls = [ + "https://download.thehive-project.org/analyzers.json" + #"/absolute/path/of/analyzers" + ] + + # Sane defaults. Do not change unless you know what you are doing. + fork-join-executor { + # Min number of threads available for analysis. + parallelism-min = 2 + # Parallelism (threads) ... ceil(available processors * factor). + parallelism-factor = 2.0 + # Max number of threads available for analysis. + parallelism-max = 4 + } +} + +# RESPONDERS +# +responder { + # responder location (same format as analyzer.urls) + urls = [ + "https://download.thehive-project.org/responders.json" + #"/absolute/path/of/responders" + ] + + # Sane defaults. Do not change unless you know what you are doing. + fork-join-executor { + # Min number of threads available for analysis. + parallelism-min = 2 + # Parallelism (threads) ... ceil(available processors * factor). + parallelism-factor = 2.0 + # Max number of threads available for analysis. + parallelism-max = 4 + } +} + +# It's the end my friend. Happy hunting! diff --git a/docker/docker-compose.yml b/docker/Linux/Latest/docker-compose.yml similarity index 100% rename from docker/docker-compose.yml rename to docker/Linux/Latest/docker-compose.yml diff --git a/docker/thehive/application.conf b/docker/Linux/Latest/thehive/application.conf similarity index 95% rename from docker/thehive/application.conf rename to docker/Linux/Latest/thehive/application.conf index f45f6ca..a991285 100644 --- a/docker/thehive/application.conf +++ b/docker/Linux/Latest/thehive/application.conf @@ -1,89 +1,89 @@ -play.http.secret.key="t5EeDXh2dEtJxohh" - -# JanusGraph -db { - provider: janusgraph - janusgraph { - storage { - backend: cql - hostname: ["cassandra"] - - cql { - cluster-name: thp # cluster name - keyspace: thehive # name of the keyspace - read-consistency-level: ONE - write-consistency-level: ONE - } - } - - ## Index configuration - index { - search { - backend: lucene - directory: /opt/index - } - } - } -} - -storage { - provider: localfs - localfs.location: /opt/data -} - -play.http.parser.maxDiskBuffer: 50MB - -play.modules.enabled += org.thp.thehive.connector.cortex.CortexModule -cortex { - servers = [ - { - name = local - url = "http://cortex:9001" - auth { - type = "bearer" - key = "XXXXXXXXXXXXXXx" - } - # HTTP client configuration (SSL and proxy) - # wsConfig {} - # List TheHive organisation which can use this Cortex server. All ("*") by default - # includedTheHiveOrganisations = ["*"] - # List TheHive organisation which cannot use this Cortex server. None by default - # excludedTheHiveOrganisations = [] - } - ] - # Check job update time intervalcortex - refreshDelay = 5 seconds - # Maximum number of successive errors before give up - maxRetryOnError = 3 - # Check remote Cortex status time interval - statusCheckInterval = 1 minute -} -# MISP configuration -play.modules.enabled += org.thp.thehive.connector.misp.MispModule -misp { - interval: 5 min - servers: [ - { - name = "MISP THP" # MISP name - url = "https://misp/" # URL or MISP - auth { - type = key - key = "XXXXXXXXXXXXXXx" # MISP API key - } - wsConfig { ssl { loose { acceptAnyCertificate: true } } } - } - ] -} - - -notification.webhook.endpoints = [ - { - name: local - url: "http://thehive:5000/" - version: 0 - wsConfig: {} - auth: {type:"none"} - includedTheHiveOrganisations: [] - excludedTheHiveOrganisations: [] - } -] +play.http.secret.key="t5EeDXh2dEtJxohh" + +# JanusGraph +db { + provider: janusgraph + janusgraph { + storage { + backend: cql + hostname: ["cassandra"] + + cql { + cluster-name: thp # cluster name + keyspace: thehive # name of the keyspace + read-consistency-level: ONE + write-consistency-level: ONE + } + } + + ## Index configuration + index { + search { + backend: lucene + directory: /opt/index + } + } + } +} + +storage { + provider: localfs + localfs.location: /opt/data +} + +play.http.parser.maxDiskBuffer: 50MB + +play.modules.enabled += org.thp.thehive.connector.cortex.CortexModule +cortex { + servers = [ + { + name = local + url = "http://cortex:9001" + auth { + type = "bearer" + key = "XXXXXXXXXXXXXXx" + } + # HTTP client configuration (SSL and proxy) + # wsConfig {} + # List TheHive organisation which can use this Cortex server. All ("*") by default + # includedTheHiveOrganisations = ["*"] + # List TheHive organisation which cannot use this Cortex server. None by default + # excludedTheHiveOrganisations = [] + } + ] + # Check job update time intervalcortex + refreshDelay = 5 seconds + # Maximum number of successive errors before give up + maxRetryOnError = 3 + # Check remote Cortex status time interval + statusCheckInterval = 1 minute +} +# MISP configuration +play.modules.enabled += org.thp.thehive.connector.misp.MispModule +misp { + interval: 5 min + servers: [ + { + name = "MISP THP" # MISP name + url = "https://misp/" # URL or MISP + auth { + type = key + key = "XXXXXXXXXXXXXXx" # MISP API key + } + wsConfig { ssl { loose { acceptAnyCertificate: true } } } + } + ] +} + + +notification.webhook.endpoints = [ + { + name: local + url: "http://thehive:5000/" + version: 0 + wsConfig: {} + auth: {type:"none"} + includedTheHiveOrganisations: [] + excludedTheHiveOrganisations: [] + } +] diff --git a/docker/thephish_conf_files/analyzers_level_conf.json b/docker/Linux/Latest/thephish_conf_files/analyzers_level_conf.json similarity index 100% rename from docker/thephish_conf_files/analyzers_level_conf.json rename to docker/Linux/Latest/thephish_conf_files/analyzers_level_conf.json diff --git a/docker/thephish_conf_files/configuration.json b/docker/Linux/Latest/thephish_conf_files/configuration.json similarity index 100% rename from docker/thephish_conf_files/configuration.json rename to docker/Linux/Latest/thephish_conf_files/configuration.json diff --git a/docker/thephish_conf_files/whitelist.json b/docker/Linux/Latest/thephish_conf_files/whitelist.json similarity index 100% rename from docker/thephish_conf_files/whitelist.json rename to docker/Linux/Latest/thephish_conf_files/whitelist.json diff --git a/docker/README.md b/docker/README.md index ced3d19..807c56a 100644 --- a/docker/README.md +++ b/docker/README.md @@ -3,6 +3,12 @@ This folder contains the `docker-compose.yml` file needed to start ThePhish and all the required services with Docker Compose. It uses the following images: + +- Windows +- Linux + +For Linux currently there is only one version that is synced with the main branch containing: + - [cassandra:3.11](https://hub.docker.com/_/cassandra) - [thehiveproject/thehive4:4.1.9-1](https://hub.docker.com/r/thehiveproject/thehive4) - [docker.elastic.co/elasticsearch/elasticsearch:7.11.1](https://www.docker.elastic.co/r/elasticsearch/elasticsearch:7.11.1) @@ -12,11 +18,23 @@ This folder contains the `docker-compose.yml` file needed to start ThePhish and - [coolacid/misp-docker:core-v2.4.148a](https://hub.docker.com/r/coolacid/misp-docker) - [emalderson/thephish:latest](https://hub.docker.com/r/emalderson/thephish) +For Windows there are two versions. One that is synced with the main branch but made compatible to run on a Windows environment with all the versions stated above and one with all containers updated to their latest version. Both docker-compose files have been tested and configured with the instructions down below. The latest version contains: + +- [cassandra:4](https://hub.docker.com/_/cassandra) +- [strangebee/thehive:latest](https://hub.docker.com/r/strangebee/thehive) +- [elasticsearch/elasticsearch:7.17.5](https://hub.docker.com/_/elasticsearch) +- [thehiveproject/cortex:latest](https://hub.docker.com/r/thehiveproject/cortex) +- [redis:6.2.7](https://hub.docker.com/_/redis) +- [mysql:8.0.30](https://hub.docker.com/_/mysql) +- [coolacid/misp-docker:core-latest](https://hub.docker.com/r/coolacid/misp-docker) +- [coolacid/misp-docker:modules-latest](https://hub.docker.com/r/coolacid/misp-docker) +- [emalderson/thephish:latest](https://hub.docker.com/r/emalderson/thephish) + These images are used with a minimal configuration. You can change the default parameters that are provided in the `docker-compose.yml` file. Moreover, it is possible to use additional settings by following the configuration guides available for those images. This folder also contains three subfolders that contain the configuration files used by TheHive, Cortex and ThePhish and that will be mounted as volumes in the respective containers. -In order to use this installation method, it is required to have a Linux-based OS with Docker 19.03.0+ and Docker Compose 1.25.5+ installed. The guide to install Docker can be found [here](https://docs.docker.com/engine/install/), while the guide to install Docker Compose can be found [here](https://docs.docker.com/compose/install/). +For Linux and Windows both installation methods work. For Linux it is required to have at least Docker 19.03.0 or later installed including Docker Compose 1.25.5 or later. The guide to install Docker can be found [here](https://docs.docker.com/engine/install/), while the guide to install Docker Compose can be found [here](https://docs.docker.com/compose/install/). This guide will not only show you how to run the containers but also how to configure them so that you'll be able to analyze your first email. @@ -24,25 +42,27 @@ This guide will not only show you how to run the containers but also how to conf 1. Clone the repository ``` - $ git clone https://github.com/emalderson/ThePhish.git + $ git clone https://github.com/asterictnl-lvdw/ThePhish ``` -2. Run the multi-container application. +2. Navigate to the respective directory based on your OS choice and run ``` - $ cd ThePhish/docker $ docker-compose up ``` -3. If the logs start showing many errors, this is because you need to change the ownership of some folders. Indeed, a new folder called `vol` will be created that will be used by the various containers to store data. You need to change the owner of some of its subfolders to match the user that has run the `docker-compose up` command so that the containers will be able to access their content. +**For Linux:** + +If the logs start showing many errors, this is because you need to change the ownership of some folders. Indeed, a new folder called `vol` will be created that will be used by the various containers to store data. You need to change the owner of some of its subfolders to match the user that has run the `docker-compose up` command so that the containers will be able to access their content. In order to do that, you need to stop the application, apply the change of ownership and then restart the application. + ``` $ docker-compose stop - $ sudo chown -R 1000:1000 vol/index vol/data vol/elastic* + $ sudo chown -R 1000:1000 vol/index vol/data vol/elastic* $ docker-compose up - ``` - This must be done once all the files in those folders have been created. If you face the same errors after having followed this procedure, try waiting some time (minutes) and re-execute the command to change the ownership of those folders recursively. + ``` +This must be done once all the files in those folders have been created. If you face the same errors after having followed this procedure, try waiting some time (minutes) and re-execute the command to change the ownership of those folders recursively. ### Configure the IMAP server diff --git a/docker/Windows/Latest/cortex/application.conf b/docker/Windows/Latest/cortex/application.conf new file mode 100644 index 0000000..4e374ab --- /dev/null +++ b/docker/Windows/Latest/cortex/application.conf @@ -0,0 +1,222 @@ +# Sample Cortex application.conf file + +## SECRET KEY +# +# The secret key is used to secure cryptographic functions. +# +# IMPORTANT: If you deploy your application to several instances, make +# sure to use the same key. +play.http.secret.key="msd3232fdn3ofgfbki83ihtzHSD" + +## ElasticSearch +search { + # Name of the index + index = cortex + # ElasticSearch instance address. + # For cluster, join address:port with ',': "http://ip1:9200,ip2:9200,ip3:9200" + uri = "http://elasticsearch:9200" + + ## Advanced configuration + # Scroll keepalive. + #keepalive = 1m + # Scroll page size. + #pagesize = 50 + # Number of shards + #nbshards = 5 + # Number of replicas + #nbreplicas = 1 + # Arbitrary settings + #settings { + # # Maximum number of nested fields + # mapping.nested_fields.limit = 100 + #} + + ## Authentication configuration + #username = "" + #password = "" + + ## SSL configuration + #keyStore { + # path = "/path/to/keystore" + # type = "JKS" # or PKCS12 + # password = "keystore-password" + #} + #trustStore { + # path = "/path/to/trustStore" + # type = "JKS" # or PKCS12 + # password = "trustStore-password" + #} +} + +## Cache +# +# If an analyzer is executed against the same observable, the previous report can be returned without re-executing the +# analyzer. The cache is used only if the second job occurs within cache.job (the default is 10 minutes). +cache.job = 10 minutes + +## Authentication +auth { + # "provider" parameter contains the authentication provider(s). It can be multi-valued, which is useful + # for migration. + # The available auth types are: + # - services.LocalAuthSrv : passwords are stored in the user entity within ElasticSearch). No + # configuration are required. + # - ad : use ActiveDirectory to authenticate users. The associated configuration shall be done in + # the "ad" section below. + # - ldap : use LDAP to authenticate users. The associated configuration shall be done in the + # "ldap" section below. + # - oauth2 : use OAuth/OIDC to authenticate users. Configuration is under "auth.oauth2" and "auth.sso" keys + provider = [local] + + ad { + # The Windows domain name in DNS format. This parameter is required if you do not use + # 'serverNames' below. + #domainFQDN = "mydomain.local" + + # Optionally you can specify the host names of the domain controllers instead of using 'domainFQDN + # above. If this parameter is not set, TheHive uses 'domainFQDN'. + #serverNames = [ad1.mydomain.local, ad2.mydomain.local] + + # The Windows domain name using short format. This parameter is required. + #domainName = "MYDOMAIN" + + # If 'true', use SSL to connect to the domain controller. + #useSSL = true + } + + ldap { + # The LDAP server name or address. The port can be specified using the 'host:port' + # syntax. This parameter is required if you don't use 'serverNames' below. + #serverName = "ldap.mydomain.local:389" + + # If you have multiple LDAP servers, use the multi-valued setting 'serverNames' instead. + #serverNames = [ldap1.mydomain.local, ldap2.mydomain.local] + + # Account to use to bind to the LDAP server. This parameter is required. + #bindDN = "cn=thehive,ou=services,dc=mydomain,dc=local" + + # Password of the binding account. This parameter is required. + #bindPW = "***secret*password***" + + # Base DN to search users. This parameter is required. + #baseDN = "ou=users,dc=mydomain,dc=local" + + # Filter to search user in the directory server. Please note that {0} is replaced + # by the actual user name. This parameter is required. + #filter = "(cn={0})" + + # If 'true', use SSL to connect to the LDAP directory server. + #useSSL = true + } + + oauth2 { + # URL of the authorization server + #clientId = "client-id" + #clientSecret = "client-secret" + #redirectUri = "https://my-cortex-instance.example/api/ssoLogin" + #responseType = "code" + #grantType = "authorization_code" + + # URL from where to get the access token + #authorizationUrl = "https://auth-site.com/OAuth/Authorize" + #tokenUrl = "https://auth-site.com/OAuth/Token" + + # The endpoint from which to obtain user details using the OAuth token, after successful login + #userUrl = "https://auth-site.com/api/User" + #scope = "openid profile" + } + + # Single-Sign On + sso { + # Autocreate user in database? + #autocreate = false + + # Autoupdate its profile and roles? + #autoupdate = false + + # Autologin user using SSO? + #autologin = false + + # Name of mapping class from user resource to backend user ('simple' or 'group') + #mapper = group + #attributes { + # login = "user" + # name = "name" + # groups = "groups" + # organization = "org" + #} + #defaultRoles = ["read"] + #defaultOrganization = "csirt" + #groups { + # # URL to retreive groups (leave empty if you are using OIDC) + # #url = "https://auth-site.com/api/Groups" + # # Group mappings, you can have multiple roles for each group: they are merged + # mappings { + # admin-profile-name = ["admin"] + # editor-profile-name = ["write"] + # reader-profile-name = ["read"] + # } + #} + + #mapper = simple + #attributes { + # login = "user" + # name = "name" + # roles = "roles" + # organization = "org" + #} + #defaultRoles = ["read"] + #defaultOrganization = "csirt" + } +} + +## ANALYZERS +# +analyzer { + # analyzer location + # url can be point to: + # - directory where analyzers are installed + # - json file containing the list of analyzer descriptions + urls = [ + "https://download.thehive-project.org/analyzers.json" + #"/absolute/path/of/analyzers" + ] + + # Sane defaults. Do not change unless you know what you are doing. + fork-join-executor { + # Min number of threads available for analysis. + parallelism-min = 2 + # Parallelism (threads) ... ceil(available processors * factor). + parallelism-factor = 2.0 + # Max number of threads available for analysis. + parallelism-max = 4 + } +} + +# RESPONDERS +# +responder { + # responder location (same format as analyzer.urls) + urls = [ + "https://download.thehive-project.org/responders.json" + #"/absolute/path/of/responders" + ] + + # Sane defaults. Do not change unless you know what you are doing. + fork-join-executor { + # Min number of threads available for analysis. + parallelism-min = 2 + # Parallelism (threads) ... ceil(available processors * factor). + parallelism-factor = 2.0 + # Max number of threads available for analysis. + parallelism-max = 4 + } +} + +# Proxy configuration to retrieve catalogs +# play.ws.proxy { +# host = proxy.example.com +# port = 3128 +# } + +# It's the end my friend. Happy hunting! diff --git a/docker/Windows/Latest/docker-compose.yml b/docker/Windows/Latest/docker-compose.yml new file mode 100644 index 0000000..96b908e --- /dev/null +++ b/docker/Windows/Latest/docker-compose.yml @@ -0,0 +1,129 @@ +version: "3.8" +services: + + thehive: + image: strangebee/thehive:latest + container_name: thehive + depends_on: + - cassandra + - elasticsearch + mem_limit: 1500m + volumes: + - ./thehive/application.conf:/etc/thehive/application.conf + ports: + - "9000:9000" + environment: + - JVM_OPTS="-Xms1024M -Xmx1024M" + command: --no-config --no-config-secret + + cassandra: + image: 'cassandra:4' + container_name: cassandra + mem_limit: 1000m + ports: + - "9042:9042" + environment: + - CASSANDRA_CLUSTER_NAME=TheHive + - MAX_HEAP_SIZE=1G + - HEAP_NEWSIZE=1G + - num_tokens=256 + volumes: + - cassandradata:/var/lib/cassandra + + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:7.17.5 + container_name: elasticsearch + mem_limit: 512m + ports: + - "9200:9200" + environment: + - http.host=0.0.0.0 + - discovery.type=single-node + - script.allowed_types=inline,stored + - thread_pool.search.queue_size=100000 + - thread_pool.write.queue_size=100000 + - xpack.security.enabled=false + - "ES_JAVA_OPTS=-Xms256m -Xmx256m" + volumes: + - elasticsearchdata:/usr/share/elasticsearch/data + + mysql: + image: mysql:8.0.30 + container_name: mysql + restart: unless-stopped + command: --default-authentication-plugin=mysql_native_password + environment: + - "MYSQL_USER=misp" + - "MYSQL_PASSWORD=example" + - "MYSQL_ROOT_PASSWORD=password" + - "MYSQL_DATABASE=misp" + volumes: + - ./vol/mysql:/var/lib/mysql + + redis: + image: redis:6.2.7 + container_name: redis + restart: unless-stopped + + misp-modules: + image: coolacid/misp-docker:modules-latest + container_name: misp_modules + environment: + - "REDIS_BACKEND=redis" + depends_on: + - redis + - mysql + + misp: + image: coolacid/misp-docker:core-latest + container_name: misp + restart: unless-stopped + depends_on: + - redis + - mysql + ports: + - '80:80' + - '443:443' + environment: + - "MYSQL_HOST=mysql" + - "HOSTNAME=https://localhost" + - "REDIS_FQDN=redis" + - TIMEZONE=Europe/Amsterdam + - "INIT=true" + - "CRON_USER_ID=1" + - "DISIPV6=true" + + cortex: + image: thehiveproject/cortex:latest + container_name: cortex + volumes: + - ./cortex/application.conf:/etc/cortex/application.conf + - //var/run/docker.sock:/var/run/docker.sock + - /var/run/cortex/jobs:/tmp/cortex-jobs + depends_on: + - thehive + ports: + - "9001:9001" + command: + - "--no-config" + - "--no-config-es" + + thephish: + image: emalderson/thephish:latest + container_name: thephish + restart: unless-stopped + depends_on: + - thehive + - cortex + - misp + ports: + - '0.0.0.0:8080:8080' + volumes: + - ./thephish_conf_files/analyzers_level_conf.json:/root/thephish/analyzers_level_conf.json + - ./thephish_conf_files/configuration.json:/root/thephish/configuration.json + - ./thephish_conf_files/whitelist.json:/root/thephish/whitelist.json + +volumes: + cassandradata: + elasticsearchdata: + mysql_data: \ No newline at end of file diff --git a/docker/Windows/Latest/thehive/application.conf b/docker/Windows/Latest/thehive/application.conf new file mode 100644 index 0000000..0a89d26 --- /dev/null +++ b/docker/Windows/Latest/thehive/application.conf @@ -0,0 +1,97 @@ +# TheHive configuration - application.conf +# +# +# This is the default configuration file. +# This is prepared to run with all services locally: +# - Cassandra for the database +# - Elasticsearch for index engine +# - File storage is local in /opt/thp/thehive/files +# +# If this is not your setup, please refer to the documentation at: +# https://docs.strangebee.com/thehive/ +# +# +# Secret key - used by Play Framework +# If TheHive is installed with DEB/RPM package, this is automatically generated +# If TheHive is not installed from DEB or RPM packages run the following +# command before starting thehive: +# cat > /etc/thehive/secret.conf << _EOF_ +# play.http.secret.key="$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 |# head -n 1)" +# _EOF_ +play.http.secret.key = "t5EeDXh2dEtJxohh" + + +# Database and index configuration +# By default, TheHive is configured to connect to local Cassandra 4.x and a +# local Elasticsearch services without authentication. +db.janusgraph { + storage { + backend = cql + hostname = ["cassandra"] + # Cassandra authentication (if configured) + # username = "thehive" + # password = "password" + cql { + cluster-name = thp + keyspace = thehive + } + } + index.search { + backend = elasticsearch + hostname = ["elasticsearch"] + index-name = thehive + mode = http + http-urls = ["http://elasticsearch:9200"] + } +} + +# Attachment storage configuration +# By default, TheHive is configured to store files locally in the folder. +# The path can be updated and should belong to the user/group running thehive service. (by default: thehive:thehive) +storage { + provider = localfs + localfs.location = /opt/thp/thehive/ +} + +# Define the maximum size for an attachment accepted by TheHive +play.http.parser.maxDiskBuffer = 1GB +# Define maximum size of http request (except attachment) +play.http.parser.maxMemoryBuffer = 10M + +# Service configuration +application.baseUrl = "http://localhost:9000" +play.http.context = "/" + +# Additional modules +# +# TheHive is strongly integrated with Cortex and MISP. +# Both modules are enabled by default. If not used, each one can be disabled by +# commenting the configuration line. +scalligraph.modules += org.thp.thehive.connector.misp.MispModule +misp{ + interval: 5 min + servers: [ + { + name = "MISP THP" + url = "http://misp" + auth{ + type = key + key = "0" + } + wsConfig { ssl { loose { acceptAnyCertificate: true } } } + includedTheHiveOrganisations = ["*"] + excludedTheHiveOrganisations = [] + } +] +} +scalligraph.modules += org.thp.thehive.connector.cortex.CortexModule +cortex.servers = [ +{ + name = local + url = "http://cortex:9001" + auth { + type = "bearer" + key = "0" + } +} +] diff --git a/docker/Windows/Latest/thephish_conf_files/analyzers_level_conf.json b/docker/Windows/Latest/thephish_conf_files/analyzers_level_conf.json new file mode 100644 index 0000000..ff51357 --- /dev/null +++ b/docker/Windows/Latest/thephish_conf_files/analyzers_level_conf.json @@ -0,0 +1,110 @@ +{ + "DomainMailSPFDMARC_Analyzer_1_1" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "suspicious", + "suspicious" : "suspicious", + "safe" : "safe", + "info" : "info" + } + }, + "Malwares_GetReport_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Malwares_Scan_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "MISP_2_1" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "malicious", + "suspicious" : "malicious", + "safe" : "safe", + "info" : "info" + } + }, + "Onyphe_Summary_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "PassiveTotal_Malware_2_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Robtex_Reverse_PDNS_Query_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Robtex_Forward_PDNS_Query_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Robtex_IP_Query_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Threatcrowd_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "malicious", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Urlscan_io_Search_0_1_1" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "VirusTotal_GetReport_3_0" : { + "dataType" : ["ip", "domain"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + } +} \ No newline at end of file diff --git a/docker/Windows/Latest/thephish_conf_files/configuration.json b/docker/Windows/Latest/thephish_conf_files/configuration.json new file mode 100644 index 0000000..63855f7 --- /dev/null +++ b/docker/Windows/Latest/thephish_conf_files/configuration.json @@ -0,0 +1,26 @@ +{ + "imap" : { + "host" : "imap.gmail.com", + "port" : "993", + "user" : "example@gmail.com", + "password" : "0", + "folder" : "inbox" + }, + "thehive" : { + "url" : "http://thehive:9000", + "apikey" : "0" + }, + "cortex" : { + "url" : "http://cortex:9001", + "apikey" : "0", + "id" : "local" + }, + "misp" : { + "id" : "MISP THP" + }, + "case" : { + "tlp" : "2", + "pap" : "2", + "tags" : ["email", "ThePhish"] + } +} \ No newline at end of file diff --git a/docker/Windows/Latest/thephish_conf_files/whitelist.json b/docker/Windows/Latest/thephish_conf_files/whitelist.json new file mode 100644 index 0000000..df2f1e4 --- /dev/null +++ b/docker/Windows/Latest/thephish_conf_files/whitelist.json @@ -0,0 +1,1385 @@ +{ + "exactMatching": { + "mail" : [], + "ip" : [ + "127.0.0.1", + "8.8.8.8", + "8.8.4.4" + ], + "url" : [], + "domain" : [ + "bit.ly", + "adf.ly", + "goo.gl", + "tinyurl", + "ow.ly", + "rebrandly", + "is.gd", + "buff.ly", + "bit.do", + "su.pr", + "4chan.org", + "4shared.com", + "9gag.com", + "aa.com", + "accuweather.com", + "adidas.com", + "adidas.it", + "adobe.com", + "airbnb.com", + "alibaba.com", + "aliexpress.com", + "alipay.com", + "allegro.pl", + "amazon.ca", + "amazon.cn", + "amazon.co.jp", + "amazon.co.uk", + "amazon.com", + "amazon.de", + "amazon.es", + "amazon.fr", + "amazon.in", + "amazon.it", + "amazonaws.com", + "americanexpress.com", + "android.com", + "apple.com", + "archive.org", + "audible.com", + "avg.com", + "badoo.com", + "bankofamerica.com", + "bbc.co.uk", + "bbc.com", + "bing.com", + "bloomberg.com", + "bodybuilding.com", + "booking.com", + "buzzfeed.com", + "cisco.com", + "cnn.com", + "contactlab.com", + "contactlab.it", + "corriere.it", + "dailymotion.com", + "dell.com", + "delta.com", + "deviantart.com", + "dictionary.com", + "disqus.com", + "dropbox.com", + "duckduckgo.com", + "duolingo.com", + "e-hentai.org", + "ebates.com", + "ebay-kleinanzeigen.de", + "ebay.co.uk", + "ebay.com", + "ebay.com.au", + "ebay.de", + "ebay.fr", + "ebay.in", + "ebay.it", + "etsy.com", + "europa.eu", + "eventbrite.com", + "evernote.com", + "expedia.com", + "facebook.com", + "fbcdn.net", + "fedex.com", + "fitbit.com", + "fiverr.com", + "flickr.com", + "foodnetwork.com", + "forbes.com", + "foxnews.com", + "gameforge.com", + "gamepedia.com", + "gap.com", + "garmin.com", + "gazzetta.it", + "gearbest.com", + "genius.com", + "getpocket.com", + "gfycat.com", + "giphy.com", + "github.com", + "gizmodo.com", + "glassdoor.com", + "gmail.com", + "godaddy.com", + "gofundme.com", + "goodreads.com", + "google.ac", + "google.ad", + "google.ae", + "google.al", + "google.am", + "google.as", + "google.at", + "google.az", + "google.ba", + "google.be", + "google.bf", + "google.bg", + "google.bi", + "google.bj", + "google.bs", + "google.bt", + "google.by", + "google.ca", + "google.cat", + "google.cc", + "google.cd", + "google.cf", + "google.cg", + "google.ch", + "google.ci", + "google.cl", + "google.cm", + "google.cn", + "google.co.ao", + "google.co.bw", + "google.co.ck", + "google.co.cr", + "google.co.id", + "google.co.il", + "google.co.in", + "google.co.jp", + "google.co.ke", + "google.co.kr", + "google.co.ls", + "google.co.ma", + "google.co.mz", + "google.co.nz", + "google.co.pn", + "google.co.th", + "google.co.tz", + "google.co.ug", + "google.co.uk", + "google.co.uz", + "google.co.ve", + "google.co.vi", + "google.co.za", + "google.co.zm", + "google.co.zw", + "google.com", + "google.com.af", + "google.com.ag", + "google.com.ai", + "google.com.ar", + "google.com.au", + "google.com.bd", + "google.com.bh", + "google.com.bn", + "google.com.bo", + "google.com.br", + "google.com.bz", + "google.com.co", + "google.com.cu", + "google.com.cy", + "google.com.do", + "google.com.ec", + "google.com.eg", + "google.com.et", + "google.com.fj", + "google.com.gh", + "google.com.gi", + "google.com.gt", + "google.com.hk", + "google.com.jm", + "google.com.kh", + "google.com.kw", + "google.com.lb", + "google.com.lc", + "google.com.ly", + "google.com.mm", + "google.com.mt", + "google.com.mx", + "google.com.my", + "google.com.na", + "google.com.nf", + "google.com.ng", + "google.com.ni", + "google.com.np", + "google.com.om", + "google.com.pa", + "google.com.pe", + "google.com.pg", + "google.com.ph", + "google.com.pk", + "google.com.pr", + "google.com.py", + "google.com.qa", + "google.com.sa", + "google.com.sb", + "google.com.sg", + "google.com.sl", + "google.com.sv", + "google.com.tj", + "google.com.tr", + "google.com.tw", + "google.com.ua", + "google.com.uy", + "google.com.vc", + "google.com.vn", + "google.cv", + "google.cz", + "google.de", + "google.dj", + "google.dk", + "google.dm", + "google.dz", + "google.ee", + "google.es", + "google.fi", + "google.fm", + "google.fr", + "google.ga", + "google.ge", + "google.gf", + "google.gg", + "google.gl", + "google.gm", + "google.gp", + "google.gr", + "google.gy", + "google.hn", + "google.hr", + "google.ht", + "google.hu", + "google.ie", + "google.im", + "google.io", + "google.iq", + "google.is", + "google.it", + "google.je", + "google.jo", + "google.kg", + "google.ki", + "google.kz", + "google.la", + "google.li", + "google.lk", + "google.lt", + "google.lu", + "google.lv", + "google.md", + "google.me", + "google.mg", + "google.mk", + "google.ml", + "google.mn", + "google.ms", + "google.mu", + "google.mv", + "google.mw", + "google.ne", + "google.nl", + "google.no", + "google.nr", + "google.nu", + "google.pl", + "google.pn", + "google.ps", + "google.pt", + "google.ro", + "google.rs", + "google.ru", + "google.rw", + "google.sc", + "google.se", + "google.sh", + "google.si", + "google.sk", + "google.sm", + "google.sn", + "google.so", + "google.sr", + "google.st", + "google.td", + "google.tg", + "google.tk", + "google.tl", + "google.tm", + "google.tn", + "google.to", + "google.tt", + "google.vg", + "google.vu", + "google.ws", + "googleapis.com", + "googleusercontent.com", + "gov.uk", + "gstatic.com", + "groupon.com", + "hdblog.it", + "hilton.com", + "hootsuite.com", + "hotels.com", + "hotmail.com", + "hotmail.it", + "huffingtonpost.com", + "ibm.com", + "icloud.com", + "ign.com", + "ikea.com", + "imdb.com", + "imgur.com", + "instagram.com", + "intel.com", + "intesasanpaolo.com", + "java.com", + "kayak.com", + "leagueoflegends.com", + "line.me", + "linkedin.com", + "live.com", + "live.it", + "mail.ru", + "mcafee.com", + "microsoft.com", + "minecraft.net", + "mozilla.org", + "msn.com", + "myanimelist.net", + "myfitnesspal.com", + "netflix.com", + "netflix.net", + "nflxext.com", + "nflximg.net", + "nflxvideo.net", + "nike.com", + "nikkei.com", + "norton.com", + "office.com", + "office365.com", + "okcupid.com", + "oracle.com", + "outlook.com", + "pandora.com", + "paypal.com", + "pinterest.com", + "qq.com", + "quora.com", + "quoracdn.net", + "reddit.com", + "repubblica.it", + "researchgate.net", + "ryanair.com", + "salesforce.com", + "samsung.com", + "schema.org", + "sciencedirect.com", + "scribd.com", + "sendgrid.com", + "sendgrid.net", + "shopify.com", + "sky.com", + "skype.com", + "slideshare.net", + "snapchat.com", + "soundcloud.com", + "speedtest.net", + "spotify.com", + "ssl-images-amazon.com", + "stackexchange.com", + "stackoverflow.com", + "t-mobile.com", + "telegram.org", + "thatviralfeed.com", + "thedailybeast.com", + "thefreedictionary.com", + "theguardian.com", + "thehindu.com", + "thekitchn.com", + "theladbible.com", + "themeforest.net", + "thesaurus.com", + "thesportbible.com", + "theverge.com", + "ticketmaster.com", + "tim.it", + "time.com", + "trello.com", + "tribunnews.com", + "tripadvisor.co.uk", + "tripadvisor.com", + "trulia.com", + "tumblr.com", + "tutorialspoint.com", + "twitch.tv", + "twitter.com", + "uber.com", + "udemy.com", + "uefa.com", + "ultimate-guitar.com", + "unina.it", + "united.com", + "ups.com", + "urbandictionary.com", + "usaa.com", + "usatoday.com", + "usbank.com", + "usps.com", + "verizon.com", + "vice.com", + "vimeo.com", + "vk.com", + "w3.org", + "w3schools.com", + "walgreens.com", + "walmart.com", + "washingtonpost.com", + "webex.com", + "wellsfargo.com", + "westernjournalism.com", + "whatsapp.com", + "whitepages.com", + "wikia.com", + "wikihow.com", + "wikimedia.org", + "wikipedia.org", + "wiktionary.org", + "wiley.com", + "wittyfeed.com", + "wix.com", + "wordpress.com", + "wordpress.org", + "wordreference.com", + "wp.com", + "xbox.com", + "xda-developers.com", + "xe.com", + "yahoo.com", + "yandex.by", + "yandex.com", + "yandex.com.tr", + "yandex.kz", + "yandex.ru", + "yandex.ua", + "yandex.uz", + "yelp.com", + "youtube.com", + "zara.com" + ], + "filename" : [], + "filetype" : [], + "hash" : [] + }, + "domainsInSubdomains" : [ + "4chan.org", + "4shared.com", + "9gag.com", + "aa.com", + "accuweather.com", + "adidas.com", + "adidas.it", + "adobe.com", + "airbnb.com", + "alibaba.com", + "aliexpress.com", + "alipay.com", + "allegro.pl", + "amazon.ca", + "amazon.cn", + "amazon.co.jp", + "amazon.co.uk", + "amazon.com", + "amazon.de", + "amazon.es", + "amazon.fr", + "amazon.in", + "amazon.it", + "amazonaws.com", + "americanexpress.com", + "android.com", + "apple.com", + "archive.org", + "audible.com", + "avg.com", + "badoo.com", + "bankofamerica.com", + "bbc.co.uk", + "bbc.com", + "bing.com", + "bloomberg.com", + "bodybuilding.com", + "booking.com", + "buzzfeed.com", + "cisco.com", + "cnn.com", + "contactlab.com", + "contactlab.it", + "corriere.it", + "dailymotion.com", + "dell.com", + "delta.com", + "deviantart.com", + "dictionary.com", + "disqus.com", + "dropbox.com", + "duckduckgo.com", + "duolingo.com", + "e-hentai.org", + "ebates.com", + "ebay-kleinanzeigen.de", + "ebay.co.uk", + "ebay.com", + "ebay.com.au", + "ebay.de", + "ebay.fr", + "ebay.in", + "ebay.it", + "etsy.com", + "europa.eu", + "eventbrite.com", + "evernote.com", + "expedia.com", + "facebook.com", + "fbcdn.net", + "fedex.com", + "fitbit.com", + "fiverr.com", + "flickr.com", + "foodnetwork.com", + "forbes.com", + "foxnews.com", + "gameforge.com", + "gamepedia.com", + "gap.com", + "garmin.com", + "gazzetta.it", + "gearbest.com", + "genius.com", + "getpocket.com", + "gfycat.com", + "giphy.com", + "github.com", + "gizmodo.com", + "glassdoor.com", + "gmail.com", + "godaddy.com", + "gofundme.com", + "goodreads.com", + "google.ac", + "google.ad", + "google.ae", + "google.al", + "google.am", + "google.as", + "google.at", + "google.az", + "google.ba", + "google.be", + "google.bf", + "google.bg", + "google.bi", + "google.bj", + "google.bs", + "google.bt", + "google.by", + "google.ca", + "google.cat", + "google.cc", + "google.cd", + "google.cf", + "google.cg", + "google.ch", + "google.ci", + "google.cl", + "google.cm", + "google.cn", + "google.co.ao", + "google.co.bw", + "google.co.ck", + "google.co.cr", + "google.co.id", + "google.co.il", + "google.co.in", + "google.co.jp", + "google.co.ke", + "google.co.kr", + "google.co.ls", + "google.co.ma", + "google.co.mz", + "google.co.nz", + "google.co.pn", + "google.co.th", + "google.co.tz", + "google.co.ug", + "google.co.uk", + "google.co.uz", + "google.co.ve", + "google.co.vi", + "google.co.za", + "google.co.zm", + "google.co.zw", + "google.com", + "google.com.af", + "google.com.ag", + "google.com.ai", + "google.com.ar", + "google.com.au", + "google.com.bd", + "google.com.bh", + "google.com.bn", + "google.com.bo", + "google.com.br", + "google.com.bz", + "google.com.co", + "google.com.cu", + "google.com.cy", + "google.com.do", + "google.com.ec", + "google.com.eg", + "google.com.et", + "google.com.fj", + "google.com.gh", + "google.com.gi", + "google.com.gt", + "google.com.hk", + "google.com.jm", + "google.com.kh", + "google.com.kw", + "google.com.lb", + "google.com.lc", + "google.com.ly", + "google.com.mm", + "google.com.mt", + "google.com.mx", + "google.com.my", + "google.com.na", + "google.com.nf", + "google.com.ng", + "google.com.ni", + "google.com.np", + "google.com.om", + "google.com.pa", + "google.com.pe", + "google.com.pg", + "google.com.ph", + "google.com.pk", + "google.com.pr", + "google.com.py", + "google.com.qa", + "google.com.sa", + "google.com.sb", + "google.com.sg", + "google.com.sl", + "google.com.sv", + "google.com.tj", + "google.com.tr", + "google.com.tw", + "google.com.ua", + "google.com.uy", + "google.com.vc", + "google.com.vn", + "google.cv", + "google.cz", + "google.de", + "google.dj", + "google.dk", + "google.dm", + "google.dz", + "google.ee", + "google.es", + "google.fi", + "google.fm", + "google.fr", + "google.ga", + "google.ge", + "google.gf", + "google.gg", + "google.gl", + "google.gm", + "google.gp", + "google.gr", + "google.gy", + "google.hn", + "google.hr", + "google.ht", + "google.hu", + "google.ie", + "google.im", + "google.io", + "google.iq", + "google.is", + "google.it", + "google.je", + "google.jo", + "google.kg", + "google.ki", + "google.kz", + "google.la", + "google.li", + "google.lk", + "google.lt", + "google.lu", + "google.lv", + "google.md", + "google.me", + "google.mg", + "google.mk", + "google.ml", + "google.mn", + "google.ms", + "google.mu", + "google.mv", + "google.mw", + "google.ne", + "google.nl", + "google.no", + "google.nr", + "google.nu", + "google.pl", + "google.pn", + "google.ps", + "google.pt", + "google.ro", + "google.rs", + "google.ru", + "google.rw", + "google.sc", + "google.se", + "google.sh", + "google.si", + "google.sk", + "google.sm", + "google.sn", + "google.so", + "google.sr", + "google.st", + "google.td", + "google.tg", + "google.tk", + "google.tl", + "google.tm", + "google.tn", + "google.to", + "google.tt", + "google.vg", + "google.vu", + "google.ws", + "googleapis.com", + "googleusercontent.com", + "gov.uk", + "gstatic.com", + "groupon.com", + "hdblog.it", + "hilton.com", + "hootsuite.com", + "hotels.com", + "hotmail.com", + "hotmail.it", + "huffingtonpost.com", + "ibm.com", + "icloud.com", + "ign.com", + "ikea.com", + "imdb.com", + "imgur.com", + "instagram.com", + "intel.com", + "intesasanpaolo.com", + "java.com", + "kayak.com", + "leagueoflegends.com", + "line.me", + "linkedin.com", + "live.com", + "live.it", + "mail.ru", + "mcafee.com", + "microsoft.com", + "minecraft.net", + "mozilla.org", + "msn.com", + "myanimelist.net", + "myfitnesspal.com", + "netflix.com", + "netflix.net", + "nflxext.com", + "nflximg.net", + "nflxvideo.net", + "nike.com", + "nikkei.com", + "norton.com", + "office.com", + "office365.com", + "okcupid.com", + "oracle.com", + "outlook.com", + "pandora.com", + "paypal.com", + "pinterest.com", + "qq.com", + "quora.com", + "quoracdn.net", + "reddit.com", + "repubblica.it", + "researchgate.net", + "ryanair.com", + "salesforce.com", + "samsung.com", + "schema.org", + "sciencedirect.com", + "scribd.com", + "sendgrid.com", + "sendgrid.net", + "shopify.com", + "sky.com", + "skype.com", + "slideshare.net", + "snapchat.com", + "soundcloud.com", + "speedtest.net", + "spotify.com", + "ssl-images-amazon.com", + "stackexchange.com", + "stackoverflow.com", + "t-mobile.com", + "telegram.org", + "thatviralfeed.com", + "thedailybeast.com", + "thefreedictionary.com", + "theguardian.com", + "thehindu.com", + "thekitchn.com", + "theladbible.com", + "themeforest.net", + "thesaurus.com", + "thesportbible.com", + "theverge.com", + "ticketmaster.com", + "tim.it", + "time.com", + "trello.com", + "tribunnews.com", + "tripadvisor.co.uk", + "tripadvisor.com", + "trulia.com", + "tumblr.com", + "tutorialspoint.com", + "twitch.tv", + "twitter.com", + "uber.com", + "udemy.com", + "uefa.com", + "ultimate-guitar.com", + "unina.it", + "united.com", + "ups.com", + "urbandictionary.com", + "usaa.com", + "usatoday.com", + "usbank.com", + "usps.com", + "verizon.com", + "vice.com", + "vimeo.com", + "vk.com", + "w3.org", + "w3schools.com", + "walgreens.com", + "walmart.com", + "washingtonpost.com", + "webex.com", + "wellsfargo.com", + "westernjournalism.com", + "whatsapp.com", + "whitepages.com", + "wikia.com", + "wikihow.com", + "wikimedia.org", + "wikipedia.org", + "wiktionary.org", + "wiley.com", + "wittyfeed.com", + "wix.com", + "wordpress.com", + "wordpress.org", + "wordreference.com", + "wp.com", + "xbox.com", + "xda-developers.com", + "xe.com", + "yahoo.com", + "yandex.by", + "yandex.com", + "yandex.com.tr", + "yandex.kz", + "yandex.ru", + "yandex.ua", + "yandex.uz", + "yelp.com", + "youtube.com", + "zara.com" + ], + "domainsInURLs" : [ + "4chan.org", + "4shared.com", + "9gag.com", + "aa.com", + "accuweather.com", + "adidas.com", + "adidas.it", + "adobe.com", + "airbnb.com", + "alibaba.com", + "aliexpress.com", + "alipay.com", + "allegro.pl", + "amazon.ca", + "amazon.cn", + "amazon.co.jp", + "amazon.co.uk", + "amazon.com", + "amazon.de", + "amazon.es", + "amazon.fr", + "amazon.in", + "amazon.it", + "amazonaws.com", + "americanexpress.com", + "android.com", + "apple.com", + "archive.org", + "audible.com", + "avg.com", + "badoo.com", + "bankofamerica.com", + "bbc.co.uk", + "bbc.com", + "bing.com", + "bloomberg.com", + "bodybuilding.com", + "booking.com", + "buzzfeed.com", + "cisco.com", + "cnn.com", + "contactlab.com", + "contactlab.it", + "corriere.it", + "dailymotion.com", + "dell.com", + "delta.com", + "deviantart.com", + "dictionary.com", + "disqus.com", + "dropbox.com", + "duckduckgo.com", + "duolingo.com", + "e-hentai.org", + "ebates.com", + "ebay-kleinanzeigen.de", + "ebay.co.uk", + "ebay.com", + "ebay.com.au", + "ebay.de", + "ebay.fr", + "ebay.in", + "ebay.it", + "etsy.com", + "europa.eu", + "eventbrite.com", + "evernote.com", + "expedia.com", + "facebook.com", + "fbcdn.net", + "fedex.com", + "fitbit.com", + "fiverr.com", + "flickr.com", + "foodnetwork.com", + "forbes.com", + "foxnews.com", + "gameforge.com", + "gamepedia.com", + "gap.com", + "garmin.com", + "gazzetta.it", + "gearbest.com", + "genius.com", + "getpocket.com", + "gfycat.com", + "giphy.com", + "github.com", + "gizmodo.com", + "glassdoor.com", + "gmail.com", + "godaddy.com", + "gofundme.com", + "goodreads.com", + "google.ac", + "google.ad", + "google.ae", + "google.al", + "google.am", + "google.as", + "google.at", + "google.az", + "google.ba", + "google.be", + "google.bf", + "google.bg", + "google.bi", + "google.bj", + "google.bs", + "google.bt", + "google.by", + "google.ca", + "google.cat", + "google.cc", + "google.cd", + "google.cf", + "google.cg", + "google.ch", + "google.ci", + "google.cl", + "google.cm", + "google.cn", + "google.co.ao", + "google.co.bw", + "google.co.ck", + "google.co.cr", + "google.co.id", + "google.co.il", + "google.co.in", + "google.co.jp", + "google.co.ke", + "google.co.kr", + "google.co.ls", + "google.co.ma", + "google.co.mz", + "google.co.nz", + "google.co.pn", + "google.co.th", + "google.co.tz", + "google.co.ug", + "google.co.uk", + "google.co.uz", + "google.co.ve", + "google.co.vi", + "google.co.za", + "google.co.zm", + "google.co.zw", + "google.com", + "google.com.af", + "google.com.ag", + "google.com.ai", + "google.com.ar", + "google.com.au", + "google.com.bd", + "google.com.bh", + "google.com.bn", + "google.com.bo", + "google.com.br", + "google.com.bz", + "google.com.co", + "google.com.cu", + "google.com.cy", + "google.com.do", + "google.com.ec", + "google.com.eg", + "google.com.et", + "google.com.fj", + "google.com.gh", + "google.com.gi", + "google.com.gt", + "google.com.hk", + "google.com.jm", + "google.com.kh", + "google.com.kw", + "google.com.lb", + "google.com.lc", + "google.com.ly", + "google.com.mm", + "google.com.mt", + "google.com.mx", + "google.com.my", + "google.com.na", + "google.com.nf", + "google.com.ng", + "google.com.ni", + "google.com.np", + "google.com.om", + "google.com.pa", + "google.com.pe", + "google.com.pg", + "google.com.ph", + "google.com.pk", + "google.com.pr", + "google.com.py", + "google.com.qa", + "google.com.sa", + "google.com.sb", + "google.com.sg", + "google.com.sl", + "google.com.sv", + "google.com.tj", + "google.com.tr", + "google.com.tw", + "google.com.ua", + "google.com.uy", + "google.com.vc", + "google.com.vn", + "google.cv", + "google.cz", + "google.de", + "google.dj", + "google.dk", + "google.dm", + "google.dz", + "google.ee", + "google.es", + "google.fi", + "google.fm", + "google.fr", + "google.ga", + "google.ge", + "google.gf", + "google.gg", + "google.gl", + "google.gm", + "google.gp", + "google.gr", + "google.gy", + "google.hn", + "google.hr", + "google.ht", + "google.hu", + "google.ie", + "google.im", + "google.io", + "google.iq", + "google.is", + "google.it", + "google.je", + "google.jo", + "google.kg", + "google.ki", + "google.kz", + "google.la", + "google.li", + "google.lk", + "google.lt", + "google.lu", + "google.lv", + "google.md", + "google.me", + "google.mg", + "google.mk", + "google.ml", + "google.mn", + "google.ms", + "google.mu", + "google.mv", + "google.mw", + "google.ne", + "google.nl", + "google.no", + "google.nr", + "google.nu", + "google.pl", + "google.pn", + "google.ps", + "google.pt", + "google.ro", + "google.rs", + "google.ru", + "google.rw", + "google.sc", + "google.se", + "google.sh", + "google.si", + "google.sk", + "google.sm", + "google.sn", + "google.so", + "google.sr", + "google.st", + "google.td", + "google.tg", + "google.tk", + "google.tl", + "google.tm", + "google.tn", + "google.to", + "google.tt", + "google.vg", + "google.vu", + "google.ws", + "googleapis.com", + "googleusercontent.com", + "gov.uk", + "gstatic.com", + "groupon.com", + "hdblog.it", + "hilton.com", + "hootsuite.com", + "hotels.com", + "hotmail.com", + "hotmail.it", + "huffingtonpost.com", + "ibm.com", + "icloud.com", + "ign.com", + "ikea.com", + "imdb.com", + "imgur.com", + "instagram.com", + "intel.com", + "intesasanpaolo.com", + "java.com", + "kayak.com", + "leagueoflegends.com", + "line.me", + "linkedin.com", + "live.com", + "live.it", + "mail.ru", + "mcafee.com", + "microsoft.com", + "minecraft.net", + "mozilla.org", + "msn.com", + "myanimelist.net", + "myfitnesspal.com", + "netflix.com", + "netflix.net", + "nflxext.com", + "nflximg.net", + "nflxvideo.net", + "nike.com", + "nikkei.com", + "norton.com", + "office.com", + "office365.com", + "okcupid.com", + "oracle.com", + "outlook.com", + "pandora.com", + "paypal.com", + "pinterest.com", + "qq.com", + "quora.com", + "quoracdn.net", + "reddit.com", + "repubblica.it", + "researchgate.net", + "ryanair.com", + "salesforce.com", + "samsung.com", + "schema.org", + "sciencedirect.com", + "scribd.com", + "sendgrid.com", + "sendgrid.net", + "shopify.com", + "sky.com", + "skype.com", + "slideshare.net", + "snapchat.com", + "soundcloud.com", + "speedtest.net", + "spotify.com", + "ssl-images-amazon.com", + "stackexchange.com", + "stackoverflow.com", + "t-mobile.com", + "telegram.org", + "thatviralfeed.com", + "thedailybeast.com", + "thefreedictionary.com", + "theguardian.com", + "thehindu.com", + "thekitchn.com", + "theladbible.com", + "themeforest.net", + "thesaurus.com", + "thesportbible.com", + "theverge.com", + "ticketmaster.com", + "tim.it", + "time.com", + "trello.com", + "tribunnews.com", + "tripadvisor.co.uk", + "tripadvisor.com", + "trulia.com", + "tumblr.com", + "tutorialspoint.com", + "twitch.tv", + "twitter.com", + "uber.com", + "udemy.com", + "uefa.com", + "ultimate-guitar.com", + "unina.it", + "united.com", + "ups.com", + "urbandictionary.com", + "usaa.com", + "usatoday.com", + "usbank.com", + "usps.com", + "verizon.com", + "vice.com", + "vimeo.com", + "vk.com", + "w3.org", + "w3schools.com", + "walgreens.com", + "walmart.com", + "washingtonpost.com", + "webex.com", + "wellsfargo.com", + "westernjournalism.com", + "whatsapp.com", + "whitepages.com", + "wikia.com", + "wikihow.com", + "wikimedia.org", + "wikipedia.org", + "wiktionary.org", + "wiley.com", + "wittyfeed.com", + "wix.com", + "wordpress.com", + "wordpress.org", + "wordreference.com", + "wp.com", + "xbox.com", + "xda-developers.com", + "xe.com", + "yahoo.com", + "yandex.by", + "yandex.com", + "yandex.com.tr", + "yandex.kz", + "yandex.ru", + "yandex.ua", + "yandex.uz", + "yelp.com", + "youtube.com", + "zara.com" + ], + "domainsInEmails" : [ + "accountprotection.microsoft.com" + ], + "regexMatching" : { + "mail" : [], + "ip" : [ + "10\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}", + "172\\.16\\.\\d{1,3}\\.\\d{1,3}", + "192\\.168\\.\\d{1,3}\\.\\d{1,3}" + ], + "url" : [], + "domain" : [], + "filename" : [] + } +} \ No newline at end of file diff --git a/docker/Windows/Old/cortex/application.conf b/docker/Windows/Old/cortex/application.conf new file mode 100644 index 0000000..6236c81 --- /dev/null +++ b/docker/Windows/Old/cortex/application.conf @@ -0,0 +1,217 @@ +# Sample Cortex application.conf file + +## SECRET KEY +# +# The secret key is used to secure cryptographic functions. +# +# IMPORTANT: If you deploy your application to several instances, make +# sure to use the same key. +play.http.secret.key="msd3232fdn3ofgfbki83ihtzHSD" + +## ElasticSearch +search { + # Name of the index + index = cortex + # ElasticSearch instance address. + # For cluster, join address:port with ',': "http://ip1:9200,ip2:9200,ip3:9200" + uri = "http://elasticsearch:9200" + + ## Advanced configuration + # Scroll keepalive. + #keepalive = 1m + # Scroll page size. + #pagesize = 50 + # Number of shards + #nbshards = 5 + # Number of replicas + #nbreplicas = 1 + # Arbitrary settings + #settings { + # # Maximum number of nested fields + # mapping.nested_fields.limit = 100 + #} + + ## Authentication configuration + #search.username = "" + #search.password = "" + + ## SSL configuration + #search.keyStore { + # path = "/path/to/keystore" + # type = "JKS" # or PKCS12 + # password = "keystore-password" + #} + #search.trustStore { + # path = "/path/to/trustStore" + # type = "JKS" # or PKCS12 + # password = "trustStore-password" + #} +} + +## Cache +# +# If an analyzer is executed against the same observable, the previous report can be returned without re-executing the +# analyzer. The cache is used only if the second job occurs within cache.job (the default is 10 minutes). +cache.job = 10 minutes + +## Authentication +auth { + # "provider" parameter contains the authentication provider(s). It can be multi-valued, which is useful + # for migration. + # The available auth types are: + # - services.LocalAuthSrv : passwords are stored in the user entity within ElasticSearch). No + # configuration are required. + # - ad : use ActiveDirectory to authenticate users. The associated configuration shall be done in + # the "ad" section below. + # - ldap : use LDAP to authenticate users. The associated configuration shall be done in the + # "ldap" section below. + # - oauth2 : use OAuth/OIDC to authenticate users. Configuration is under "auth.oauth2" and "auth.sso" keys + provider = [local] + + ad { + # The Windows domain name in DNS format. This parameter is required if you do not use + # 'serverNames' below. + #domainFQDN = "mydomain.local" + + # Optionally you can specify the host names of the domain controllers instead of using 'domainFQDN + # above. If this parameter is not set, TheHive uses 'domainFQDN'. + #serverNames = [ad1.mydomain.local, ad2.mydomain.local] + + # The Windows domain name using short format. This parameter is required. + #domainName = "MYDOMAIN" + + # If 'true', use SSL to connect to the domain controller. + #useSSL = true + } + + ldap { + # The LDAP server name or address. The port can be specified using the 'host:port' + # syntax. This parameter is required if you don't use 'serverNames' below. + #serverName = "ldap.mydomain.local:389" + + # If you have multiple LDAP servers, use the multi-valued setting 'serverNames' instead. + #serverNames = [ldap1.mydomain.local, ldap2.mydomain.local] + + # Account to use to bind to the LDAP server. This parameter is required. + #bindDN = "cn=thehive,ou=services,dc=mydomain,dc=local" + + # Password of the binding account. This parameter is required. + #bindPW = "***secret*password***" + + # Base DN to search users. This parameter is required. + #baseDN = "ou=users,dc=mydomain,dc=local" + + # Filter to search user in the directory server. Please note that {0} is replaced + # by the actual user name. This parameter is required. + #filter = "(cn={0})" + + # If 'true', use SSL to connect to the LDAP directory server. + #useSSL = true + } + + oauth2 { + # URL of the authorization server + #clientId = "client-id" + #clientSecret = "client-secret" + #redirectUri = "https://my-thehive-instance.example/index.html#!/login" + #responseType = "code" + #grantType = "authorization_code" + + # URL from where to get the access token + #authorizationUrl = "https://auth-site.com/OAuth/Authorize" + #tokenUrl = "https://auth-site.com/OAuth/Token" + + # The endpoint from which to obtain user details using the OAuth token, after successful login + #userUrl = "https://auth-site.com/api/User" + #scope = "openid profile" + # Type of authorization header + #authorizationHeader = "Bearer" # or token + } + + # Single-Sign On + sso { + # Autocreate user in database? + #autocreate = false + + # Autoupdate its profile and roles? + #autoupdate = false + + # Autologin user using SSO? + #autologin = false + + # Attributes mappings + #attributes { + # login = "login" + # name = "name" + # groups = "groups" + # roles = "roles" # list of roles, separated with comma + # organisation = "org" + #} + + # Name of mapping class from user resource to backend user ('simple' or 'group') + #mapper = group + # Default roles for users with no groups mapped ("read", "analyze", "orgadmin") + #defaultRoles = [] + # Default organization + #defaultOrganization = "MyOrga" + + #groups { + # # URL to retreive groups (leave empty if you are using OIDC) + # #url = "https://auth-site.com/api/Groups" + # # Group mappings, you can have multiple roles for each group: they are merged + # mappings { + # admin-profile-name = ["admin"] + # editor-profile-name = ["write"] + # reader-profile-name = ["read"] + # } + #} + } +} + +job { + runner = [docker] +} +## ANALYZERS +# +analyzer { + # analyzer location + # url can be point to: + # - directory where analyzers are installed + # - json file containing the list of analyzer descriptions + urls = [ + "https://download.thehive-project.org/analyzers.json" + #"/absolute/path/of/analyzers" + ] + + # Sane defaults. Do not change unless you know what you are doing. + fork-join-executor { + # Min number of threads available for analysis. + parallelism-min = 2 + # Parallelism (threads) ... ceil(available processors * factor). + parallelism-factor = 2.0 + # Max number of threads available for analysis. + parallelism-max = 4 + } +} + +# RESPONDERS +# +responder { + # responder location (same format as analyzer.urls) + urls = [ + "https://download.thehive-project.org/responders.json" + #"/absolute/path/of/responders" + ] + + # Sane defaults. Do not change unless you know what you are doing. + fork-join-executor { + # Min number of threads available for analysis. + parallelism-min = 2 + # Parallelism (threads) ... ceil(available processors * factor). + parallelism-factor = 2.0 + # Max number of threads available for analysis. + parallelism-max = 4 + } +} + +# It's the end my friend. Happy hunting! diff --git a/docker/Windows/Old/docker-compose.yml b/docker/Windows/Old/docker-compose.yml new file mode 100644 index 0000000..2227c55 --- /dev/null +++ b/docker/Windows/Old/docker-compose.yml @@ -0,0 +1,132 @@ +version: "3.8" +services: + + cassandra: + image: cassandra:3.11 + container_name: cassandra + restart: unless-stopped + hostname: cassandra + environment: + - MAX_HEAP_SIZE=1G + - HEAP_NEWSIZE=1G + - CASSANDRA_CLUSTER_NAME=thp + volumes: + - ./vol/cassandra-data:/var/lib/cassandra/data + + thehive: + image: thehiveproject/thehive4:4.1.9-1 + container_name: thehive + restart: unless-stopped + depends_on: + - cassandra + - elasticsearch + - cortex + ports: + - '0.0.0.0:9000:9000' + volumes: + - ./thehive/application.conf:/etc/thehive/application.conf + - ./vol/data:/opt/data + - ./vol/index:/opt/index + command: '--no-config --no-config-secret' + + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:7.11.1 + container_name: elasticsearch + restart: unless-stopped + ports: + - '0.0.0.0:9200:9200' + environment: + - http.host=0.0.0.0 + - discovery.type=single-node + - cluster.name=hive + - script.allowed_types= inline + - thread_pool.search.queue_size=100000 + - thread_pool.write.queue_size=10000 + - gateway.recover_after_nodes=1 + - xpack.security.enabled=false + - bootstrap.memory_lock=true + - 'ES_JAVA_OPTS=-Xms256m -Xmx256m' + ulimits: + nofile: + soft: 65536 + hard: 65536 + volumes: + - ./vol/elasticsearch_data:/usr/share/elasticsearch/data + - ./vol/elasticsearch_logs:/usr/share/elasticsearch/logs + + cortex: + image: thehiveproject/cortex:3.1.1-1 + container_name: cortex + restart: unless-stopped + depends_on: + - elasticsearch + volumes: + - ./cortex/application.conf:/etc/cortex/application.conf + - '//var/run/docker.sock:/var/run/docker.sock' + - '/var/run/cortex/jobs:/tmp/cortex-jobs' + environment: + - http_proxy=${http_proxy} + - https_proxy=${https_proxy} + ports: + - '0.0.0.0:9001:9001' + + redis: + image: redis:6.2.5 + container_name: redis + restart: unless-stopped + + mysql: + image: mysql:8.0.30 + container_name: mysql + restart: unless-stopped + command: --default-authentication-plugin=mysql_native_password + environment: + - "MYSQL_USER=misp" + - "MYSQL_PASSWORD=example" + - "MYSQL_ROOT_PASSWORD=password" + - "MYSQL_DATABASE=misp" + volumes: + - ./vol/mysql:/var/lib/mysql + + misp: + image: coolacid/misp-docker:core-v2.4.148a + container_name: misp + restart: unless-stopped + depends_on: + - redis + - mysql + ports: + - '0.0.0.0:80:80' + - '0.0.0.0:443:443' + environment: + - "MYSQL_HOST=mysql" + - "HOSTNAME=https://localhost" + - "REDIS_FQDN=redis" + - TIMEZONE=Europe/Amsterdam + - "INIT=true" + - "CRON_USER_ID=1" + - "DISIPV6=true" + + misp-modules: + image: coolacid/misp-docker:modules-latest + container_name: misp-modules + environment: + - "REDIS_BACKEND=redis" + depends_on: + - redis + - mysql + + thephish: + image: emalderson/thephish:latest + container_name: thephish + restart: unless-stopped + depends_on: + - thehive + - cortex + - misp + ports: + - '0.0.0.0:8080:8080' + volumes: + - ./thephish_conf_files/analyzers_level_conf.json:/root/thephish/analyzers_level_conf.json + - ./thephish_conf_files/configuration.json:/root/thephish/configuration.json + - ./thephish_conf_files/whitelist.json:/root/thephish/whitelist.json \ No newline at end of file diff --git a/docker/Windows/Old/thehive/application.conf b/docker/Windows/Old/thehive/application.conf new file mode 100644 index 0000000..a26dfb4 --- /dev/null +++ b/docker/Windows/Old/thehive/application.conf @@ -0,0 +1,86 @@ +play.http.secret.key="t5EeDXh2dEtJxohh" + +# JanusGraph +db { + provider: janusgraph + janusgraph { + storage { + backend: cql + hostname: ["cassandra"] + + cql { + cluster-name: thp # cluster name + keyspace: thehive # name of the keyspace + read-consistency-level: ONE + write-consistency-level: ONE + } + } + + ## Index configuration + index { + search { + backend: lucene + directory: /opt/index + } + } + } +} + +storage { + provider: localfs + localfs.location: /opt/data +} + +play.http.parser.maxDiskBuffer: 50MB + +play.modules.enabled += org.thp.thehive.connector.cortex.CortexModule +cortex { + servers = [ + { + name = local + url = "http://cortex:9001" + auth { + type = "bearer" + key = "XXXXXXXXXXXXx" + } + wsConfig {} + includedTheHiveOrganisations = ["*"] + excludedTheHiveOrganisations = [] + } + ] + refreshDelay = 5 seconds + maxRetryOnError = 3 + statusCheckInterval = 1 minute +} + +# MISP configuration +play.modules.enabled += org.thp.thehive.connector.misp.MispModule +misp { + interval: 5 min + servers: [ + { + name = "MISP THP" + url = "http://misp/" + auth { + type = key + key = "XXXXXXXXXXXXx" + } + wsConfig { ssl { loose { acceptAnyCertificate: true } } } + includedTheHiveOrganisations = ["*"] + excludedTheHiveOrganisations = [] + } + ] +} + + +notification.webhook.endpoints = [ + { + name: local + url: "http://thehive:5000/" + version: 0 + wsConfig: {} + auth: {type:"none"} + includedTheHiveOrganisations: [] + excludedTheHiveOrganisations: [] + } +] diff --git a/docker/Windows/Old/thephish_conf_files/analyzers_level_conf.json b/docker/Windows/Old/thephish_conf_files/analyzers_level_conf.json new file mode 100644 index 0000000..ff51357 --- /dev/null +++ b/docker/Windows/Old/thephish_conf_files/analyzers_level_conf.json @@ -0,0 +1,110 @@ +{ + "DomainMailSPFDMARC_Analyzer_1_1" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "suspicious", + "suspicious" : "suspicious", + "safe" : "safe", + "info" : "info" + } + }, + "Malwares_GetReport_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Malwares_Scan_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "MISP_2_1" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "malicious", + "suspicious" : "malicious", + "safe" : "safe", + "info" : "info" + } + }, + "Onyphe_Summary_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "PassiveTotal_Malware_2_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Robtex_Reverse_PDNS_Query_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Robtex_Forward_PDNS_Query_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Robtex_IP_Query_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Threatcrowd_1_0" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "malicious", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "Urlscan_io_Search_0_1_1" : { + "dataType" : ["url", "ip", "domain", "mail"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + }, + "VirusTotal_GetReport_3_0" : { + "dataType" : ["ip", "domain"], + "levelMapping" : { + "malicious" : "info", + "suspicious" : "info", + "safe" : "safe", + "info" : "info" + } + } +} \ No newline at end of file diff --git a/docker/Windows/Old/thephish_conf_files/configuration.json b/docker/Windows/Old/thephish_conf_files/configuration.json new file mode 100644 index 0000000..85f1bbc --- /dev/null +++ b/docker/Windows/Old/thephish_conf_files/configuration.json @@ -0,0 +1,26 @@ +{ + "imap" : { + "host" : "imap.gmail.com", + "port" : "993", + "user" : "example@gmail.com", + "password" : "", + "folder" : "inbox" + }, + "thehive" : { + "url" : "http://thehive:9000", + "apikey" : "xXxXxXxXxXxXxXxXxXxXxXx" + }, + "cortex" : { + "url" : "http://cortex:9001", + "apikey" : "xXxXxXxXxXxXxXxXxXxXxXx", + "id" : "local" + }, + "misp" : { + "id" : "MISP THP" + }, + "case" : { + "tlp" : "2", + "pap" : "2", + "tags" : ["email", "ThePhish"] + } +} \ No newline at end of file diff --git a/docker/Windows/Old/thephish_conf_files/whitelist.json b/docker/Windows/Old/thephish_conf_files/whitelist.json new file mode 100644 index 0000000..df2f1e4 --- /dev/null +++ b/docker/Windows/Old/thephish_conf_files/whitelist.json @@ -0,0 +1,1385 @@ +{ + "exactMatching": { + "mail" : [], + "ip" : [ + "127.0.0.1", + "8.8.8.8", + "8.8.4.4" + ], + "url" : [], + "domain" : [ + "bit.ly", + "adf.ly", + "goo.gl", + "tinyurl", + "ow.ly", + "rebrandly", + "is.gd", + "buff.ly", + "bit.do", + "su.pr", + "4chan.org", + "4shared.com", + "9gag.com", + "aa.com", + "accuweather.com", + "adidas.com", + "adidas.it", + "adobe.com", + "airbnb.com", + "alibaba.com", + "aliexpress.com", + "alipay.com", + "allegro.pl", + "amazon.ca", + "amazon.cn", + "amazon.co.jp", + "amazon.co.uk", + "amazon.com", + "amazon.de", + "amazon.es", + "amazon.fr", + "amazon.in", + "amazon.it", + "amazonaws.com", + "americanexpress.com", + "android.com", + "apple.com", + "archive.org", + "audible.com", + "avg.com", + "badoo.com", + "bankofamerica.com", + "bbc.co.uk", + "bbc.com", + "bing.com", + "bloomberg.com", + "bodybuilding.com", + "booking.com", + "buzzfeed.com", + "cisco.com", + "cnn.com", + "contactlab.com", + "contactlab.it", + "corriere.it", + "dailymotion.com", + "dell.com", + "delta.com", + "deviantart.com", + "dictionary.com", + "disqus.com", + "dropbox.com", + "duckduckgo.com", + "duolingo.com", + "e-hentai.org", + "ebates.com", + "ebay-kleinanzeigen.de", + "ebay.co.uk", + "ebay.com", + "ebay.com.au", + "ebay.de", + "ebay.fr", + "ebay.in", + "ebay.it", + "etsy.com", + "europa.eu", + "eventbrite.com", + "evernote.com", + "expedia.com", + "facebook.com", + "fbcdn.net", + "fedex.com", + "fitbit.com", + "fiverr.com", + "flickr.com", + "foodnetwork.com", + "forbes.com", + "foxnews.com", + "gameforge.com", + "gamepedia.com", + "gap.com", + "garmin.com", + "gazzetta.it", + "gearbest.com", + "genius.com", + "getpocket.com", + "gfycat.com", + "giphy.com", + "github.com", + "gizmodo.com", + "glassdoor.com", + "gmail.com", + "godaddy.com", + "gofundme.com", + "goodreads.com", + "google.ac", + "google.ad", + "google.ae", + "google.al", + "google.am", + "google.as", + "google.at", + "google.az", + "google.ba", + "google.be", + "google.bf", + "google.bg", + "google.bi", + "google.bj", + "google.bs", + "google.bt", + "google.by", + "google.ca", + "google.cat", + "google.cc", + "google.cd", + "google.cf", + "google.cg", + "google.ch", + "google.ci", + "google.cl", + "google.cm", + "google.cn", + "google.co.ao", + "google.co.bw", + "google.co.ck", + "google.co.cr", + "google.co.id", + "google.co.il", + "google.co.in", + "google.co.jp", + "google.co.ke", + "google.co.kr", + "google.co.ls", + "google.co.ma", + "google.co.mz", + "google.co.nz", + "google.co.pn", + "google.co.th", + "google.co.tz", + "google.co.ug", + "google.co.uk", + "google.co.uz", + "google.co.ve", + "google.co.vi", + "google.co.za", + "google.co.zm", + "google.co.zw", + "google.com", + "google.com.af", + "google.com.ag", + "google.com.ai", + "google.com.ar", + "google.com.au", + "google.com.bd", + "google.com.bh", + "google.com.bn", + "google.com.bo", + "google.com.br", + "google.com.bz", + "google.com.co", + "google.com.cu", + "google.com.cy", + "google.com.do", + "google.com.ec", + "google.com.eg", + "google.com.et", + "google.com.fj", + "google.com.gh", + "google.com.gi", + "google.com.gt", + "google.com.hk", + "google.com.jm", + "google.com.kh", + "google.com.kw", + "google.com.lb", + "google.com.lc", + "google.com.ly", + "google.com.mm", + "google.com.mt", + "google.com.mx", + "google.com.my", + "google.com.na", + "google.com.nf", + "google.com.ng", + "google.com.ni", + "google.com.np", + "google.com.om", + "google.com.pa", + "google.com.pe", + "google.com.pg", + "google.com.ph", + "google.com.pk", + "google.com.pr", + "google.com.py", + "google.com.qa", + "google.com.sa", + "google.com.sb", + "google.com.sg", + "google.com.sl", + "google.com.sv", + "google.com.tj", + "google.com.tr", + "google.com.tw", + "google.com.ua", + "google.com.uy", + "google.com.vc", + "google.com.vn", + "google.cv", + "google.cz", + "google.de", + "google.dj", + "google.dk", + "google.dm", + "google.dz", + "google.ee", + "google.es", + "google.fi", + "google.fm", + "google.fr", + "google.ga", + "google.ge", + "google.gf", + "google.gg", + "google.gl", + "google.gm", + "google.gp", + "google.gr", + "google.gy", + "google.hn", + "google.hr", + "google.ht", + "google.hu", + "google.ie", + "google.im", + "google.io", + "google.iq", + "google.is", + "google.it", + "google.je", + "google.jo", + "google.kg", + "google.ki", + "google.kz", + "google.la", + "google.li", + "google.lk", + "google.lt", + "google.lu", + "google.lv", + "google.md", + "google.me", + "google.mg", + "google.mk", + "google.ml", + "google.mn", + "google.ms", + "google.mu", + "google.mv", + "google.mw", + "google.ne", + "google.nl", + "google.no", + "google.nr", + "google.nu", + "google.pl", + "google.pn", + "google.ps", + "google.pt", + "google.ro", + "google.rs", + "google.ru", + "google.rw", + "google.sc", + "google.se", + "google.sh", + "google.si", + "google.sk", + "google.sm", + "google.sn", + "google.so", + "google.sr", + "google.st", + "google.td", + "google.tg", + "google.tk", + "google.tl", + "google.tm", + "google.tn", + "google.to", + "google.tt", + "google.vg", + "google.vu", + "google.ws", + "googleapis.com", + "googleusercontent.com", + "gov.uk", + "gstatic.com", + "groupon.com", + "hdblog.it", + "hilton.com", + "hootsuite.com", + "hotels.com", + "hotmail.com", + "hotmail.it", + "huffingtonpost.com", + "ibm.com", + "icloud.com", + "ign.com", + "ikea.com", + "imdb.com", + "imgur.com", + "instagram.com", + "intel.com", + "intesasanpaolo.com", + "java.com", + "kayak.com", + "leagueoflegends.com", + "line.me", + "linkedin.com", + "live.com", + "live.it", + "mail.ru", + "mcafee.com", + "microsoft.com", + "minecraft.net", + "mozilla.org", + "msn.com", + "myanimelist.net", + "myfitnesspal.com", + "netflix.com", + "netflix.net", + "nflxext.com", + "nflximg.net", + "nflxvideo.net", + "nike.com", + "nikkei.com", + "norton.com", + "office.com", + "office365.com", + "okcupid.com", + "oracle.com", + "outlook.com", + "pandora.com", + "paypal.com", + "pinterest.com", + "qq.com", + "quora.com", + "quoracdn.net", + "reddit.com", + "repubblica.it", + "researchgate.net", + "ryanair.com", + "salesforce.com", + "samsung.com", + "schema.org", + "sciencedirect.com", + "scribd.com", + "sendgrid.com", + "sendgrid.net", + "shopify.com", + "sky.com", + "skype.com", + "slideshare.net", + "snapchat.com", + "soundcloud.com", + "speedtest.net", + "spotify.com", + "ssl-images-amazon.com", + "stackexchange.com", + "stackoverflow.com", + "t-mobile.com", + "telegram.org", + "thatviralfeed.com", + "thedailybeast.com", + "thefreedictionary.com", + "theguardian.com", + "thehindu.com", + "thekitchn.com", + "theladbible.com", + "themeforest.net", + "thesaurus.com", + "thesportbible.com", + "theverge.com", + "ticketmaster.com", + "tim.it", + "time.com", + "trello.com", + "tribunnews.com", + "tripadvisor.co.uk", + "tripadvisor.com", + "trulia.com", + "tumblr.com", + "tutorialspoint.com", + "twitch.tv", + "twitter.com", + "uber.com", + "udemy.com", + "uefa.com", + "ultimate-guitar.com", + "unina.it", + "united.com", + "ups.com", + "urbandictionary.com", + "usaa.com", + "usatoday.com", + "usbank.com", + "usps.com", + "verizon.com", + "vice.com", + "vimeo.com", + "vk.com", + "w3.org", + "w3schools.com", + "walgreens.com", + "walmart.com", + "washingtonpost.com", + "webex.com", + "wellsfargo.com", + "westernjournalism.com", + "whatsapp.com", + "whitepages.com", + "wikia.com", + "wikihow.com", + "wikimedia.org", + "wikipedia.org", + "wiktionary.org", + "wiley.com", + "wittyfeed.com", + "wix.com", + "wordpress.com", + "wordpress.org", + "wordreference.com", + "wp.com", + "xbox.com", + "xda-developers.com", + "xe.com", + "yahoo.com", + "yandex.by", + "yandex.com", + "yandex.com.tr", + "yandex.kz", + "yandex.ru", + "yandex.ua", + "yandex.uz", + "yelp.com", + "youtube.com", + "zara.com" + ], + "filename" : [], + "filetype" : [], + "hash" : [] + }, + "domainsInSubdomains" : [ + "4chan.org", + "4shared.com", + "9gag.com", + "aa.com", + "accuweather.com", + "adidas.com", + "adidas.it", + "adobe.com", + "airbnb.com", + "alibaba.com", + "aliexpress.com", + "alipay.com", + "allegro.pl", + "amazon.ca", + "amazon.cn", + "amazon.co.jp", + "amazon.co.uk", + "amazon.com", + "amazon.de", + "amazon.es", + "amazon.fr", + "amazon.in", + "amazon.it", + "amazonaws.com", + "americanexpress.com", + "android.com", + "apple.com", + "archive.org", + "audible.com", + "avg.com", + "badoo.com", + "bankofamerica.com", + "bbc.co.uk", + "bbc.com", + "bing.com", + "bloomberg.com", + "bodybuilding.com", + "booking.com", + "buzzfeed.com", + "cisco.com", + "cnn.com", + "contactlab.com", + "contactlab.it", + "corriere.it", + "dailymotion.com", + "dell.com", + "delta.com", + "deviantart.com", + "dictionary.com", + "disqus.com", + "dropbox.com", + "duckduckgo.com", + "duolingo.com", + "e-hentai.org", + "ebates.com", + "ebay-kleinanzeigen.de", + "ebay.co.uk", + "ebay.com", + "ebay.com.au", + "ebay.de", + "ebay.fr", + "ebay.in", + "ebay.it", + "etsy.com", + "europa.eu", + "eventbrite.com", + "evernote.com", + "expedia.com", + "facebook.com", + "fbcdn.net", + "fedex.com", + "fitbit.com", + "fiverr.com", + "flickr.com", + "foodnetwork.com", + "forbes.com", + "foxnews.com", + "gameforge.com", + "gamepedia.com", + "gap.com", + "garmin.com", + "gazzetta.it", + "gearbest.com", + "genius.com", + "getpocket.com", + "gfycat.com", + "giphy.com", + "github.com", + "gizmodo.com", + "glassdoor.com", + "gmail.com", + "godaddy.com", + "gofundme.com", + "goodreads.com", + "google.ac", + "google.ad", + "google.ae", + "google.al", + "google.am", + "google.as", + "google.at", + "google.az", + "google.ba", + "google.be", + "google.bf", + "google.bg", + "google.bi", + "google.bj", + "google.bs", + "google.bt", + "google.by", + "google.ca", + "google.cat", + "google.cc", + "google.cd", + "google.cf", + "google.cg", + "google.ch", + "google.ci", + "google.cl", + "google.cm", + "google.cn", + "google.co.ao", + "google.co.bw", + "google.co.ck", + "google.co.cr", + "google.co.id", + "google.co.il", + "google.co.in", + "google.co.jp", + "google.co.ke", + "google.co.kr", + "google.co.ls", + "google.co.ma", + "google.co.mz", + "google.co.nz", + "google.co.pn", + "google.co.th", + "google.co.tz", + "google.co.ug", + "google.co.uk", + "google.co.uz", + "google.co.ve", + "google.co.vi", + "google.co.za", + "google.co.zm", + "google.co.zw", + "google.com", + "google.com.af", + "google.com.ag", + "google.com.ai", + "google.com.ar", + "google.com.au", + "google.com.bd", + "google.com.bh", + "google.com.bn", + "google.com.bo", + "google.com.br", + "google.com.bz", + "google.com.co", + "google.com.cu", + "google.com.cy", + "google.com.do", + "google.com.ec", + "google.com.eg", + "google.com.et", + "google.com.fj", + "google.com.gh", + "google.com.gi", + "google.com.gt", + "google.com.hk", + "google.com.jm", + "google.com.kh", + "google.com.kw", + "google.com.lb", + "google.com.lc", + "google.com.ly", + "google.com.mm", + "google.com.mt", + "google.com.mx", + "google.com.my", + "google.com.na", + "google.com.nf", + "google.com.ng", + "google.com.ni", + "google.com.np", + "google.com.om", + "google.com.pa", + "google.com.pe", + "google.com.pg", + "google.com.ph", + "google.com.pk", + "google.com.pr", + "google.com.py", + "google.com.qa", + "google.com.sa", + "google.com.sb", + "google.com.sg", + "google.com.sl", + "google.com.sv", + "google.com.tj", + "google.com.tr", + "google.com.tw", + "google.com.ua", + "google.com.uy", + "google.com.vc", + "google.com.vn", + "google.cv", + "google.cz", + "google.de", + "google.dj", + "google.dk", + "google.dm", + "google.dz", + "google.ee", + "google.es", + "google.fi", + "google.fm", + "google.fr", + "google.ga", + "google.ge", + "google.gf", + "google.gg", + "google.gl", + "google.gm", + "google.gp", + "google.gr", + "google.gy", + "google.hn", + "google.hr", + "google.ht", + "google.hu", + "google.ie", + "google.im", + "google.io", + "google.iq", + "google.is", + "google.it", + "google.je", + "google.jo", + "google.kg", + "google.ki", + "google.kz", + "google.la", + "google.li", + "google.lk", + "google.lt", + "google.lu", + "google.lv", + "google.md", + "google.me", + "google.mg", + "google.mk", + "google.ml", + "google.mn", + "google.ms", + "google.mu", + "google.mv", + "google.mw", + "google.ne", + "google.nl", + "google.no", + "google.nr", + "google.nu", + "google.pl", + "google.pn", + "google.ps", + "google.pt", + "google.ro", + "google.rs", + "google.ru", + "google.rw", + "google.sc", + "google.se", + "google.sh", + "google.si", + "google.sk", + "google.sm", + "google.sn", + "google.so", + "google.sr", + "google.st", + "google.td", + "google.tg", + "google.tk", + "google.tl", + "google.tm", + "google.tn", + "google.to", + "google.tt", + "google.vg", + "google.vu", + "google.ws", + "googleapis.com", + "googleusercontent.com", + "gov.uk", + "gstatic.com", + "groupon.com", + "hdblog.it", + "hilton.com", + "hootsuite.com", + "hotels.com", + "hotmail.com", + "hotmail.it", + "huffingtonpost.com", + "ibm.com", + "icloud.com", + "ign.com", + "ikea.com", + "imdb.com", + "imgur.com", + "instagram.com", + "intel.com", + "intesasanpaolo.com", + "java.com", + "kayak.com", + "leagueoflegends.com", + "line.me", + "linkedin.com", + "live.com", + "live.it", + "mail.ru", + "mcafee.com", + "microsoft.com", + "minecraft.net", + "mozilla.org", + "msn.com", + "myanimelist.net", + "myfitnesspal.com", + "netflix.com", + "netflix.net", + "nflxext.com", + "nflximg.net", + "nflxvideo.net", + "nike.com", + "nikkei.com", + "norton.com", + "office.com", + "office365.com", + "okcupid.com", + "oracle.com", + "outlook.com", + "pandora.com", + "paypal.com", + "pinterest.com", + "qq.com", + "quora.com", + "quoracdn.net", + "reddit.com", + "repubblica.it", + "researchgate.net", + "ryanair.com", + "salesforce.com", + "samsung.com", + "schema.org", + "sciencedirect.com", + "scribd.com", + "sendgrid.com", + "sendgrid.net", + "shopify.com", + "sky.com", + "skype.com", + "slideshare.net", + "snapchat.com", + "soundcloud.com", + "speedtest.net", + "spotify.com", + "ssl-images-amazon.com", + "stackexchange.com", + "stackoverflow.com", + "t-mobile.com", + "telegram.org", + "thatviralfeed.com", + "thedailybeast.com", + "thefreedictionary.com", + "theguardian.com", + "thehindu.com", + "thekitchn.com", + "theladbible.com", + "themeforest.net", + "thesaurus.com", + "thesportbible.com", + "theverge.com", + "ticketmaster.com", + "tim.it", + "time.com", + "trello.com", + "tribunnews.com", + "tripadvisor.co.uk", + "tripadvisor.com", + "trulia.com", + "tumblr.com", + "tutorialspoint.com", + "twitch.tv", + "twitter.com", + "uber.com", + "udemy.com", + "uefa.com", + "ultimate-guitar.com", + "unina.it", + "united.com", + "ups.com", + "urbandictionary.com", + "usaa.com", + "usatoday.com", + "usbank.com", + "usps.com", + "verizon.com", + "vice.com", + "vimeo.com", + "vk.com", + "w3.org", + "w3schools.com", + "walgreens.com", + "walmart.com", + "washingtonpost.com", + "webex.com", + "wellsfargo.com", + "westernjournalism.com", + "whatsapp.com", + "whitepages.com", + "wikia.com", + "wikihow.com", + "wikimedia.org", + "wikipedia.org", + "wiktionary.org", + "wiley.com", + "wittyfeed.com", + "wix.com", + "wordpress.com", + "wordpress.org", + "wordreference.com", + "wp.com", + "xbox.com", + "xda-developers.com", + "xe.com", + "yahoo.com", + "yandex.by", + "yandex.com", + "yandex.com.tr", + "yandex.kz", + "yandex.ru", + "yandex.ua", + "yandex.uz", + "yelp.com", + "youtube.com", + "zara.com" + ], + "domainsInURLs" : [ + "4chan.org", + "4shared.com", + "9gag.com", + "aa.com", + "accuweather.com", + "adidas.com", + "adidas.it", + "adobe.com", + "airbnb.com", + "alibaba.com", + "aliexpress.com", + "alipay.com", + "allegro.pl", + "amazon.ca", + "amazon.cn", + "amazon.co.jp", + "amazon.co.uk", + "amazon.com", + "amazon.de", + "amazon.es", + "amazon.fr", + "amazon.in", + "amazon.it", + "amazonaws.com", + "americanexpress.com", + "android.com", + "apple.com", + "archive.org", + "audible.com", + "avg.com", + "badoo.com", + "bankofamerica.com", + "bbc.co.uk", + "bbc.com", + "bing.com", + "bloomberg.com", + "bodybuilding.com", + "booking.com", + "buzzfeed.com", + "cisco.com", + "cnn.com", + "contactlab.com", + "contactlab.it", + "corriere.it", + "dailymotion.com", + "dell.com", + "delta.com", + "deviantart.com", + "dictionary.com", + "disqus.com", + "dropbox.com", + "duckduckgo.com", + "duolingo.com", + "e-hentai.org", + "ebates.com", + "ebay-kleinanzeigen.de", + "ebay.co.uk", + "ebay.com", + "ebay.com.au", + "ebay.de", + "ebay.fr", + "ebay.in", + "ebay.it", + "etsy.com", + "europa.eu", + "eventbrite.com", + "evernote.com", + "expedia.com", + "facebook.com", + "fbcdn.net", + "fedex.com", + "fitbit.com", + "fiverr.com", + "flickr.com", + "foodnetwork.com", + "forbes.com", + "foxnews.com", + "gameforge.com", + "gamepedia.com", + "gap.com", + "garmin.com", + "gazzetta.it", + "gearbest.com", + "genius.com", + "getpocket.com", + "gfycat.com", + "giphy.com", + "github.com", + "gizmodo.com", + "glassdoor.com", + "gmail.com", + "godaddy.com", + "gofundme.com", + "goodreads.com", + "google.ac", + "google.ad", + "google.ae", + "google.al", + "google.am", + "google.as", + "google.at", + "google.az", + "google.ba", + "google.be", + "google.bf", + "google.bg", + "google.bi", + "google.bj", + "google.bs", + "google.bt", + "google.by", + "google.ca", + "google.cat", + "google.cc", + "google.cd", + "google.cf", + "google.cg", + "google.ch", + "google.ci", + "google.cl", + "google.cm", + "google.cn", + "google.co.ao", + "google.co.bw", + "google.co.ck", + "google.co.cr", + "google.co.id", + "google.co.il", + "google.co.in", + "google.co.jp", + "google.co.ke", + "google.co.kr", + "google.co.ls", + "google.co.ma", + "google.co.mz", + "google.co.nz", + "google.co.pn", + "google.co.th", + "google.co.tz", + "google.co.ug", + "google.co.uk", + "google.co.uz", + "google.co.ve", + "google.co.vi", + "google.co.za", + "google.co.zm", + "google.co.zw", + "google.com", + "google.com.af", + "google.com.ag", + "google.com.ai", + "google.com.ar", + "google.com.au", + "google.com.bd", + "google.com.bh", + "google.com.bn", + "google.com.bo", + "google.com.br", + "google.com.bz", + "google.com.co", + "google.com.cu", + "google.com.cy", + "google.com.do", + "google.com.ec", + "google.com.eg", + "google.com.et", + "google.com.fj", + "google.com.gh", + "google.com.gi", + "google.com.gt", + "google.com.hk", + "google.com.jm", + "google.com.kh", + "google.com.kw", + "google.com.lb", + "google.com.lc", + "google.com.ly", + "google.com.mm", + "google.com.mt", + "google.com.mx", + "google.com.my", + "google.com.na", + "google.com.nf", + "google.com.ng", + "google.com.ni", + "google.com.np", + "google.com.om", + "google.com.pa", + "google.com.pe", + "google.com.pg", + "google.com.ph", + "google.com.pk", + "google.com.pr", + "google.com.py", + "google.com.qa", + "google.com.sa", + "google.com.sb", + "google.com.sg", + "google.com.sl", + "google.com.sv", + "google.com.tj", + "google.com.tr", + "google.com.tw", + "google.com.ua", + "google.com.uy", + "google.com.vc", + "google.com.vn", + "google.cv", + "google.cz", + "google.de", + "google.dj", + "google.dk", + "google.dm", + "google.dz", + "google.ee", + "google.es", + "google.fi", + "google.fm", + "google.fr", + "google.ga", + "google.ge", + "google.gf", + "google.gg", + "google.gl", + "google.gm", + "google.gp", + "google.gr", + "google.gy", + "google.hn", + "google.hr", + "google.ht", + "google.hu", + "google.ie", + "google.im", + "google.io", + "google.iq", + "google.is", + "google.it", + "google.je", + "google.jo", + "google.kg", + "google.ki", + "google.kz", + "google.la", + "google.li", + "google.lk", + "google.lt", + "google.lu", + "google.lv", + "google.md", + "google.me", + "google.mg", + "google.mk", + "google.ml", + "google.mn", + "google.ms", + "google.mu", + "google.mv", + "google.mw", + "google.ne", + "google.nl", + "google.no", + "google.nr", + "google.nu", + "google.pl", + "google.pn", + "google.ps", + "google.pt", + "google.ro", + "google.rs", + "google.ru", + "google.rw", + "google.sc", + "google.se", + "google.sh", + "google.si", + "google.sk", + "google.sm", + "google.sn", + "google.so", + "google.sr", + "google.st", + "google.td", + "google.tg", + "google.tk", + "google.tl", + "google.tm", + "google.tn", + "google.to", + "google.tt", + "google.vg", + "google.vu", + "google.ws", + "googleapis.com", + "googleusercontent.com", + "gov.uk", + "gstatic.com", + "groupon.com", + "hdblog.it", + "hilton.com", + "hootsuite.com", + "hotels.com", + "hotmail.com", + "hotmail.it", + "huffingtonpost.com", + "ibm.com", + "icloud.com", + "ign.com", + "ikea.com", + "imdb.com", + "imgur.com", + "instagram.com", + "intel.com", + "intesasanpaolo.com", + "java.com", + "kayak.com", + "leagueoflegends.com", + "line.me", + "linkedin.com", + "live.com", + "live.it", + "mail.ru", + "mcafee.com", + "microsoft.com", + "minecraft.net", + "mozilla.org", + "msn.com", + "myanimelist.net", + "myfitnesspal.com", + "netflix.com", + "netflix.net", + "nflxext.com", + "nflximg.net", + "nflxvideo.net", + "nike.com", + "nikkei.com", + "norton.com", + "office.com", + "office365.com", + "okcupid.com", + "oracle.com", + "outlook.com", + "pandora.com", + "paypal.com", + "pinterest.com", + "qq.com", + "quora.com", + "quoracdn.net", + "reddit.com", + "repubblica.it", + "researchgate.net", + "ryanair.com", + "salesforce.com", + "samsung.com", + "schema.org", + "sciencedirect.com", + "scribd.com", + "sendgrid.com", + "sendgrid.net", + "shopify.com", + "sky.com", + "skype.com", + "slideshare.net", + "snapchat.com", + "soundcloud.com", + "speedtest.net", + "spotify.com", + "ssl-images-amazon.com", + "stackexchange.com", + "stackoverflow.com", + "t-mobile.com", + "telegram.org", + "thatviralfeed.com", + "thedailybeast.com", + "thefreedictionary.com", + "theguardian.com", + "thehindu.com", + "thekitchn.com", + "theladbible.com", + "themeforest.net", + "thesaurus.com", + "thesportbible.com", + "theverge.com", + "ticketmaster.com", + "tim.it", + "time.com", + "trello.com", + "tribunnews.com", + "tripadvisor.co.uk", + "tripadvisor.com", + "trulia.com", + "tumblr.com", + "tutorialspoint.com", + "twitch.tv", + "twitter.com", + "uber.com", + "udemy.com", + "uefa.com", + "ultimate-guitar.com", + "unina.it", + "united.com", + "ups.com", + "urbandictionary.com", + "usaa.com", + "usatoday.com", + "usbank.com", + "usps.com", + "verizon.com", + "vice.com", + "vimeo.com", + "vk.com", + "w3.org", + "w3schools.com", + "walgreens.com", + "walmart.com", + "washingtonpost.com", + "webex.com", + "wellsfargo.com", + "westernjournalism.com", + "whatsapp.com", + "whitepages.com", + "wikia.com", + "wikihow.com", + "wikimedia.org", + "wikipedia.org", + "wiktionary.org", + "wiley.com", + "wittyfeed.com", + "wix.com", + "wordpress.com", + "wordpress.org", + "wordreference.com", + "wp.com", + "xbox.com", + "xda-developers.com", + "xe.com", + "yahoo.com", + "yandex.by", + "yandex.com", + "yandex.com.tr", + "yandex.kz", + "yandex.ru", + "yandex.ua", + "yandex.uz", + "yelp.com", + "youtube.com", + "zara.com" + ], + "domainsInEmails" : [ + "accountprotection.microsoft.com" + ], + "regexMatching" : { + "mail" : [], + "ip" : [ + "10\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}", + "172\\.16\\.\\d{1,3}\\.\\d{1,3}", + "192\\.168\\.\\d{1,3}\\.\\d{1,3}" + ], + "url" : [], + "domain" : [], + "filename" : [] + } +} \ No newline at end of file