AIL framework - Framework for Analysis of Information Leaks
AIL is a modular framework to analyse potential information leaks from unstructured data sources like pastes from Pastebin or similar services or unstructured data streams. AIL framework is flexible and can be extended to support other functionalities to mine or process sensitive information.
- Modular architecture to handle streams of unstructured or structured information
- Default support for external ZMQ feeds, such as provided by CIRCL or other providers
- Multiple feed support
- Each module can process and reprocess the information already processed by AIL
- Detecting and extracting URLs including their geographical location (e.g. IP address location)
- Extracting and validating potential leak of credit cards numbers, credentials, ...
- Extracting and validating email addresses leaked including DNS MX validation
- Module for extracting Tor .onion addresses (to be further processed for analysis)
- Keep tracks of duplicates (and diffing between each duplicate found)
- Extracting and validating potential hostnames (e.g. to feed Passive DNS systems)
- A full-text indexer module to index unstructured information
- Statistics on modules and web
- Real-time modules manager in terminal
- Global sentiment analysis for each providers based on nltk vader module
- Terms, Set of terms and Regex tracking and occurrence
- Many more modules for extracting phone numbers, credentials and others
- Alerting to MISP to share found leaks within a threat intelligence platform using MISP standard
Type these command lines for a fully automated installation and start AIL framework:
git clone https://github.com/CIRCL/AIL-framework.git
cd AIL-framework
./installing_deps.sh
cd var/www/
./update_thirdparty.sh
cd ~/AIL-framework/
. ./AILENV/bin/activate
cd bin/
./LAUNCH.shThe default installing_deps.sh is for Debian and Ubuntu based distributions. For Arch linux based distributions, you can replace it with installing_deps_archlinux.sh.
There is also a Travis file used for automating the installation that can be used to build and install AIL on other systems.
- Install Docker
sudo su
apt-get install -y curl
curl https://get.docker.com | /bin/bash- Type these commands to build the Docker image:
git clone https://github.com/CIRCL/ail-framework
cd AIL-framework
docker build -t ail-framework .- To start AIL on port 7000, type the following command below:
docker run -p 7000:7000 ail-framework
- To debug the running container, type the following command and note the container name or identifier:
docker psAfter getting the name or identifier type the following commands:
docker exec -it CONTAINER_NAME_OR_IDENTIFIER bash
cd /opt/ailTo start the web interface, you first need to fetch the required JavaScript/CSS files:
cd $AILENV
cd var/www/
bash update_thirdparty.shand then you can start the web interface python script:
cd $AILENV
cd var/www/
Flask_server.pyEventually you can browse the status of the AIL framework website at the following URL:
http://localhost:7000/
HOWTO are available in HOWTO.md
Copyright (C) 2014 Jules Debra
Copyright (C) 2014-2017 CIRCL - Computer Incident Response Center Luxembourg (c/o smile, security made in Lëtzebuerg, Groupement d'Intérêt Economique)
Copyright (c) 2014-2017 Raphaël Vinot
Copyright (c) 2014-2017 Alexandre Dulaunoy
Copyright (c) 2016-2017 Sami Mokaddem
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.