diff --git a/.github/CODE_OF_CONDUCT.md b/.github/CODE_OF_CONDUCT.md new file mode 100644 index 00000000000..6a2d845c89d --- /dev/null +++ b/.github/CODE_OF_CONDUCT.md @@ -0,0 +1,128 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +maltrail.dev@gmail.com. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. + +Community Impact Guidelines were inspired by [Mozilla's code of conduct +enforcement ladder](https://github.com/mozilla/diversity). + +[homepage]: https://www.contributor-covenant.org + +For answers to common questions about this code of conduct, see the FAQ at +https://www.contributor-covenant.org/faq. Translations are available at +https://www.contributor-covenant.org/translations. diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md new file mode 100644 index 00000000000..6b9d2e3f4ba --- /dev/null +++ b/.github/CONTRIBUTING.md @@ -0,0 +1,50 @@ +# Contributing to Maltrail + +## Reporting bugs + +**Bug reports are welcome**! +Please report all bugs on the [issue tracker](https://github.com/stamparm/maltrail/issues). + +If you have a security related report, take a time to read [Reporting Maltrail Security Vulnerability](https://github.com/stamparm/maltrail/blob/master/SECURITY.md) policy. + +### Guidelines + +* Before you submit a bug report, search both [open](https://github.com/stamparm/maltrail/issues?q=is%3Aopen+is%3Aissue) and [closed](https://github.com/stamparm/maltrail/issues?q=is%3Aissue+is%3Aclosed) issues to make sure the issue has not come up before. +* Make sure you can reproduce the bug with the latest release version of Maltrail. +* Your report should give detailed instructions on how to reproduce the problem. If Maltrail raises an unhandled exception, the entire traceback is needed. Details of the unexpected behaviour are welcome too. A small test case is ideal to have. +* If you are making an enhancement request (RFE, feature request), lay out the rationale for the feature you are requesting. Describe why would proposed feature be useful. + +## Submitting code changes + +All code contributions are greatly appreciated. First off, clone the [Git repository](https://github.com/stamparm/maltrail), read the [User's manual](https://github.com/stamparm/maltrail/blob/master/README.md) and the [Wiki pages](https://github.com/stamparm/maltrail/wiki) carefully, go through the code yourself and [drop us an email](mailto:maltrail.dev@gmail.com) if you are having a hard time grasping its structure and meaning. + +Our preferred method of patch submission is via a Git [pull request](https://help.github.com/articles/using-pull-requests). + +Many [people](https://github.com/stamparm/maltrail/graphs/contributors) have contributed in different ways to the Maltrail development. See also the Maltrail's ["Thank you" list](https://github.com/stamparm/maltrail#thank-you). + +### Guidelines + +In order to maintain consistency and readability throughout the code, we ask that you adhere to the following instructions: + +* Each patch should make one logical change. +* Avoid tabbing, use four blank spaces instead. +* Before you put time into a non-trivial patch, it is worth discussing it privately by [email](mailto:maltrail.dev@gmail.com). +* Do not change style on numerous files in one single pull request, we can [discuss](mailto:maltrail.dev@gmail.com) about those before doing any major restyling, but be sure that personal preferences not having a strong support in [PEP 8](http://www.python.org/dev/peps/pep-0008/) will likely to be rejected. +* Make changes on less than five files per single pull request - there is rarely a good reason to have more than five files changed on one pull request, as this dramatically increases the review time required to land (commit) any of those pull requests. +* Style that is too different from main branch will be ''adapted'' by the developers side. +* Do not touch anything inside `thirdparty/` folder. + +## Maltrail trails contribution + +All contributions to static trails (adding new Maltrail detections, fixing false positives, updating whitelist, etc) are greatly appreciated. Before you submit a contribution to Maltrail detection trails database, take a time to read respective auxiliary articles in Maltrail's Wiki: + +* [Trail classes](https://github.com/stamparm/maltrail/wiki/Trail-classes) - Information about different classes of trails. +* [Specific detections](https://github.com/stamparm/maltrail/wiki/Specific-detections) - Information about Maltrail specific detections. +* [Maltrail trails structure](https://github.com/stamparm/maltrail/wiki/Maltrail-trails-structure) - Information about Maltrail trails structure. +* [Maltrail trails base format](https://github.com/stamparm/maltrail/wiki/Maltrail-trails-base-format) - Information about Maltrail trails base format. +* [Maltrail detection nuances](https://github.com/stamparm/maltrail/wiki/Maltrail-detection-nuances) - Information about Maltrail detection nuances. +* [Maltrail trails contribution](https://github.com/stamparm/maltrail/wiki/Maltrail-trails-contribution) - Information about Maltrail trails contribution. + +## Licensing + +By submitting code contributions to the Maltrail developers or via Git pull request, checking them into the Maltrail source code repository, it is understood (unless you specify otherwise) that you are offering the Maltrail copyright holders the unlimited, non-exclusive right to reuse, modify, and relicense the code. This is important because the inability to relicense code has caused devastating problems for other software projects. If you wish to specify special license conditions of your contributions, just say so when you send them. diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000000..968c049771b --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,35 @@ +--- +name: Bug report +about: Create a report to help us improve Maltrail +title: "[BUG]" +labels: '' +assignees: '' + +--- + +**Describe the bug** +A clear and concise description of what the bug is. + +**How To Reproduce** +Steps to reproduce the behavior: +1. Go to '...' +2. Click on '....' +3. Scroll down to '....' +4. See error + +**Expected behavior** +A clear and concise description of what you expected to happen. + +**Screenshots** +If applicable, add screenshots to help explain your problem. + +**Environment:** + - Device: [e.g. Linux-based device, OPNSense plugin] + - OS: [Linux, *BSD] + - Type of Maltrail installation: [e.g. ```git clone``` command] + - Problematic Maltrail component: [e.g. server, sensor, web-interface] + - Maltrail version: [e.g. 0.59] + - ```python-pcapy-ng``` version: [e.g. 1.0.9] + +**Additional context** +Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 00000000000..b196bfa35ff --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,20 @@ +--- +name: Feature request +about: Suggest an idea for Maltrail project +title: "[Feature Request]" +labels: '' +assignees: '' + +--- + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] + +**Describe the solution you'd like** +A clear and concise description of what you want to happen. + +**Describe alternatives you've considered** +A clear and concise description of any alternative solutions or features you've considered. + +**Additional context** +Add any other context or screenshots about the feature request here. diff --git a/.github/ISSUE_TEMPLATE/questions-and-support.md b/.github/ISSUE_TEMPLATE/questions-and-support.md new file mode 100644 index 00000000000..53341a7cfa4 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/questions-and-support.md @@ -0,0 +1,14 @@ +--- +name: Questions and Support +about: General topics. Questions and Support. +title: "[Questions and Support]" +labels: '' +assignees: '' + +--- + +**Question** +Put your question on Maltrail's functionality. + +**Support** +Put descrption of an issue you have with Maltrail settings up. diff --git a/.gitignore b/.gitignore index e7afa1b6ccc..4e75ef9d327 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ *.py[cod] *~ +Pipfile* diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index a0e40735a70..00000000000 --- a/.travis.yml +++ /dev/null @@ -1,11 +0,0 @@ -language: python -python: - - "2.6" - - "2.7" -before_install: - - sudo apt-get update -qq - - sudo apt-get install -qq libpcap-dev -install: - - pip install pcapy -script: - - python -c "import server; import sensor" diff --git a/CHANGELOG b/CHANGELOG new file mode 100644 index 00000000000..a80d58a2e12 --- /dev/null +++ b/CHANGELOG @@ -0,0 +1,366 @@ +# Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) +# See the file 'LICENSE' for copying permission + +################################################################# +# Maltrail Changelog File # +################################################################# + +[+] Added functionality +[-] Deleted functionality +[!] Bug fixing +[=] Minor update or changed functionality + +################################################################# + + + +- Version 0.73 -> 0.74 (Upcoming release) + + + +- Version 0.72 -> 0.73 (01 Sep 2024) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.71 -> 0.72 (01 Aug 2024) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.70 -> 0.71 (01 Jul 2024) + +[=] Maltrail docker container run is improved (Issue #19260) +[=] php-inj detection is improved (Issue #19262) +[=] Python 3.12 compability is improved (Issue #19257) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + +- Version 0.69 -> 0.70 (01 Jun 2024) + +[=] cruzit feed URL changed (Issue #19253) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.68 -> 0.69 (01 May 2024) + +[+] Support of simpleton IPv6 bogon address handling was added +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + + + +- Version 0.68 -> 0.69 (01 May 2024) + +[+] Support of simpleton IPv6 bogon address handling was added +[=] Multiple updates and optimizations for regular static trails and the whitelist + + +- Version 0.67 -> 0.68 (01 Apr 2024) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.66 -> 0.67 (01 Mar 2024) + +[=] Handling usage of pcapy lib instead of pcapy-ng is improved (Issue #19242) +[=] Fixed /server.py and /sensor.py restart in docker container (Issue #19243) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.65 -> 0.66 (01 Feb 2024) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.64 -> 0.65 (01 Jan 2024) + +[+] Customisable blacklists via BLACKLIST option in /maltrail.conf file (Issue #19230) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + + +- Version 0.63 -> 0.64 (01 Dec 2023) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + + +- Version 0.62 -> 0.63 (01 Nov 2023) + +[=] FAIL2BAN_REGEX and REMOTE_SEVERITY_REGEX options were updated to handle "potential iot-malware download" heur (Issue #19207) +[=] Abuseipdb feed was updated (Issue #19208) +[=] "potential remote code execution" heur for CVE-2016-0545 detection is updated (Issue #19210) +[=] "potential remote code execution" heur is updated for MacOS process list tracking in HTTP POST-req (Issue #19214) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + + +- Version 0.61 -> 0.62 (01 Oct 2023) + +[=] Multiple updates and optimizations for regular static trails and the whitelist +[=] Updates for mass_scanner and worst_asns trails + + + + +- Version 0.60 -> 0.61 (01 Sep 2023) + +[!] Workaround to have working searx server (Issue #19199) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + +- Version 0.59 -> 0.60 (01 Aug 2023) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.58 -> 0.59 (01 Jul 2023) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.57 -> 0.58 (01 Jun 2023) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.56 -> 0.57 (01 May 2023) + +[!] Fixed login page GUI issue for mobile devices (Issue #19153) +[!] Fixed incorrect parsing of ViriBack feed (Issue #19154) +[=] Added new descriptions in "Specific detections" Wiki chapter +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.55 -> 0.56 (01 Apr 2023) + +[=] Minor update for /feeds/emergingthreatsdns.py (Issue #19147) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.54 -> 0.55 (01 Mar 2023) + +[!] Fixed unauthenticated OS command injection vulnerability in http.py (Issue #19146) +[=] Minor update for _process_packet func in sensor (Issue #19129) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + + +- Version 0.53 -> 0.54 (01 Feb 2023) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.52 -> 0.53 (01 Jan 2023) + +[-] Defunct 360-netlab feeds were deleted (Issue #19138) +[=] "potential data leakage" heur is improved +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.51 -> 0.52 (01 Dec 2022) + +[=] "potential iot-malware download" heur is improved +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.50 -> 0.51 (01 Nov 2022) + +[+] New Wiki pages are added +[!] Fixed deadlock of Docker output to stdout (Issue #19121) +[!] Definition of network interfaces is improved (Issue #19123) +[!] Fixed regex for /360bigviktor.py feed (Issue #19124) +[!] Fixed syscalls handling (Issue #19125) +[=] "potential remote code execution" heuristic is improved +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.49 -> 0.50 (01 Oct 2022) + +[=] "potential remote code execution" heur for CVE-2022-30190 detection is updated +[=] "Maltrail detection nuances" wiki-page is updated +[=] "Trail classes" wiki-page is updated +[=] Multiple updates and optimizations for regular static trails and the whitelist + +- Version 0.48 -> 0.49 (01 Sep 2022) + +[!] Fixed row rendering in UI (Issue #19109) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.47 -> 0.48 (01 Aug 2022) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.46 -> 0.47 (01 Jul 2022) + +[+] "potential ssti injection" heuristic is added (CVE-2022-26134) +[=] "potential data leak" heuristic is improved +[=] "Trail-classes" wiki page is updated +[=] /requirements.txt file is updated (pcapy-ng) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.45 -> 0.46 (01 Jun 2022) + +[+] New Wiki page is added +[=] "potential remote code execution" heuristic is improved (CVE-2022-1388) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.44 -> 0.45 (01 May 2022) + +[+] systemd-based realization for Maltrail sensor.py, server.py and ipset/iptables ban-list (dedicated repo) have added +[+] New Wiki pages are added +[=] "potential remote code execution" heuristic is improved (detection for Java-related RCE stuff) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.43 -> 0.44 (01 Apr 2022) + +[=] "potential remote code execution" heuristic is improved +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.42 -> 0.43 (01 Mar 2022) + +[=] "potential remote code execution" heuristic is improved +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.41 -> 0.42 (01 Feb 2022) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.40 -> 0.41 (01 Jan 2022) + +[+] "potential remote code execution" heuristic is extended for log4j/log4shell (CVE-2021-44228) vulnerability detection +[+] "generic_log4shell.txt" and "hacked_log4j.txt" trails were added for log4j/log4shell (CVE-2021-44228) vulnerability static detection +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.39 -> 0.40 (01 Dec 2021) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.38 -> 0.39 (01 Nov 2021) + +[=] "potential directory traversal" heuristic is extended +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.37 -> 0.38 (03 Oct 2021) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.36 -> 0.37 (02 Sep 2021) + + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.35 -> 0.36 (02 Aug 2021) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.34 -> 0.35 (04 Jul 2021) + +[+] Added the prototype of heur for potential web scanning attempts +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.33 -> 0.34 (10 Jun 2021) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.32 -> 0.33 (10 Jun 2021) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.31 -> 0.32 (10 May 2021) + +[!] Fixed PR_END_OF_FILE_ERROR bug, when using HTTPS for Maltrail's server (Issue #16217) +[!] Fixed bug with TLSv1_2_METHOD (Issue #16250) +[+] Added displaying real IP behind Cloudflare's one (Issue #20) +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.30 -> 0.31 (01 Apr 2021) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.29 -> 0.30 (01 Mar 2021) + +[=] Multiple updates and optimizations for regular static trails and the whitelist + + + +- Version 0.28 -> 0.29 (01 Feb 2021) + +[+] Two new UI features (hide threat and report false positive options) +[+] Auto-refresh for Maltrail web-page (/?refresh=N, where N in seconds. Issue #624) +[+] Maltrail demo pages are released: maltraildemo.github.io +[=] Multiple updates and optimizations for regular static trails and the whitelist +[=] Potential DNS changer heur is improved +[+] Implemented colorized console output +[=] Minor style revamp and improved look and feel on mobile phones +[-] Memory check is removed +[+] Added info for proper Maltrail citation (/CITATION.cff) +[=] Added starting and ending times to console output + + + +- Version 0.27 -> 0.28 (01 Jan 2021) + +[+] Implementing support for LOGSTASH_SERVER (Logs in JSON format) +[+] Implementing REMOTE_SEVERITY_REGEX (Issue #13251) +[=] Sensor is able to get started without server (Issue #6020) +[=] Multiple updates and optimizations for regular static trails and the whitelist diff --git a/CITATION.cff b/CITATION.cff new file mode 100644 index 00000000000..9255b05c0bf --- /dev/null +++ b/CITATION.cff @@ -0,0 +1,19 @@ +# YAML 1.2 +--- +cff-version: "1.1.0" +message: "If you use this software, please cite it using these metadata." +doi: 10.23721/100/1503924 +title: "Maltrail - Malicious traffic detection system" +authors: + - + family-names: Stampar + given-names: Miroslav + orcid: "https://orcid.org/0000-0002-2662-5469" + - + family-names: Kasimov + given-names: Mikhail +abstract: "Maltrail is a malicious traffic detection system, utilizing publicly available (black)lists containing malicious and/or generally suspicious trails, along with static trails compiled from various AV reports and custom user defined lists" +date-released: 2014-12-04 +repository-code: "https://github.com/stamparm/maltrail" +license: MIT +... diff --git a/LICENSE b/LICENSE index 956f2151d15..4637822908b 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ The MIT License (MIT) -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal @@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE -SOFTWARE. \ No newline at end of file +SOFTWARE. diff --git a/README.md b/README.md index 254ad38fafa..7ffc616460a 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,13 @@ ![Maltrail](https://i.imgur.com/3xjInOD.png) -[![Build Status](https://api.travis-ci.org/stamparm/maltrail.svg?branch=master)](https://api.travis-ci.org/stamparm/maltrail) [![Python 2.6|2.7](https://img.shields.io/badge/python-2.6|2.7-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-MIT-red.svg)](https://github.com/stamparm/maltrail#license) [![Twitter](https://img.shields.io/badge/twitter-@maltrail-blue.svg)](https://twitter.com/maltrail) +[![Python 2.6|2.7|3.x](https://img.shields.io/badge/python-2.6|2.7|3.x-yellow.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-MIT-red.svg)](https://github.com/stamparm/maltrail#license) [![Malware families](https://img.shields.io/badge/malware_families-1494-orange.svg)](https://github.com/stamparm/maltrail/tree/master/trails/static/malware) [![Malware sinkholes](https://img.shields.io/badge/malware_sinkholes-1354-green.svg)](https://github.com/stamparm/maltrail/tree/master/trails/static/malware) [![Twitter](https://img.shields.io/badge/twitter-@maltrail-blue.svg)](https://twitter.com/maltrail) ## Content - [Introduction](#introduction) - [Architecture](#architecture) +- [Demo pages](#demo-pages) +- [Requirements](#requirements) - [Quick start](#quick-start) - [Administrator's guide](#administrators-guide) - [Sensor](#sensor) @@ -25,120 +27,87 @@ - [DNS resource exhaustion](#dns-resource-exhaustion) - [Data leakage](#data-leakage) - [False positives](#false-positives) -- [Requirements](#requirements) +- [Best practice(s)](#best-practices) - [License](#license) -- [Donations](#donations) +- [Sponsors](#sponsors) +- [Developers](#developers) +- [Presentations](#presentations) +- [Publications](#publications) +- [Blacklist](#blacklist) +- [Thank you](#thank-you) +- [Third-party integrations](#third-party-integrations) ## Introduction -**Maltrail** is a malicious traffic detection system, utilizing publicly available (black)lists containing malicious and/or generally suspicious trails, along with static trails compiled from various AV reports and custom user defined lists, where trail can be anything from domain name (e.g. `zvpprsensinaix.com` for [Banjori](http://www.johannesbader.ch/2015/02/the-dga-of-banjori/) malware), URL (e.g. `http://109.162.38.120/harsh02.exe` for known malicious [executable](https://www.virustotal.com/en/file/61f56f71b0b04b36d3ef0c14bbbc0df431290d93592d5dd6e3fffcc583ec1e12/analysis/)), IP address (e.g. `185.130.5.231` for known attacker) or HTTP User-Agent header value (e.g. `sqlmap` for automatic SQL injection and database takeover tool). Also, it uses (optional) advanced heuristic mechanisms that can help in discovery of unknown threats (e.g. new malware). +**Maltrail** is a malicious traffic detection system, utilizing publicly available (black)lists containing malicious and/or generally suspicious trails, along with static trails compiled from various AV reports and custom user defined lists, where trail can be anything from domain name (e.g. `zvpprsensinaix.com` for [Banjori](http://www.johannesbader.ch/2015/02/the-dga-of-banjori/) malware), URL (e.g. `hXXp://109.162.38.120/harsh02.exe` for known malicious [executable](https://www.virustotal.com/en/file/61f56f71b0b04b36d3ef0c14bbbc0df431290d93592d5dd6e3fffcc583ec1e12/analysis/)), IP address (e.g. `185.130.5.231` for known attacker) or HTTP User-Agent header value (e.g. `sqlmap` for automatic SQL injection and database takeover tool). Also, it uses (optional) advanced heuristic mechanisms that can help in discovery of unknown threats (e.g. new malware). ![Reporting tool](https://i.imgur.com/Sd9eqoa.png) The following (black)lists (i.e. feeds) are being utilized: ``` -alienvault, autoshun, badips, bambenekconsultingc2dns, -bambenekconsultingc2ip, bambenekconsultingdga, bitcoinnodes, -blocklist, botscout, bruteforceblocker, ciarmy, cruzit, -cybercrimetracker, deepviz, dataplanesipinvitation, -dataplanesipquery, dataplane, dshielddns, dshieldip, -emergingthreatsbot, emergingthreatscip, emergingthreatsdns, -feodotrackerdns, malwaredomainlist, malwaredomains, malwarepatrol, -maxmind, myip, nothink, openbl, openphish, packetmailcarisirt, -packetmailramnode, palevotracker, policeman, proxylists, proxyrss, -proxy, ransomwaretrackerdns, ransomwaretrackerip, -ransomwaretrackerurl, riproxies, rutgers, sblam, securityresearch, -snort, socksproxy, sslipbl, sslproxies, torproject, torstatus, -turris, urlvir, voipbl, vxvault, zeustrackerdns, zeustrackerip, -zeustrackermonitor, zeustrackerurl, etc. +360bigviktor, 360chinad, 360conficker, 360cryptolocker, 360gameover, +360locky, 360necurs, 360suppobox, 360tofsee, 360virut, abuseipdb, alienvault, +atmos, badips, bitcoinnodes, blackbook, blocklist, botscout, +bruteforceblocker, ciarmy, cobaltstrike, cruzit, cybercrimetracker, +dataplane, dshieldip, emergingthreatsbot, emergingthreatscip, +emergingthreatsdns, feodotrackerip, gpfcomics, greensnow, ipnoise, +kriskinteldns, kriskintelip, malc0de, malwaredomainlistdns, malwaredomains, +maxmind, minerchk, myip, openphish, palevotracker, policeman, pony, +proxylists, proxyrss, proxyspy, ransomwaretrackerdns, ransomwaretrackerip, +ransomwaretrackerurl, riproxies, rutgers, sblam, socksproxy, sslbl, +sslproxies, talosintelligence, torproject, trickbot, turris, urlhaus, +viriback, vxvault, zeustrackermonitor, zeustrackerurl, etc. ``` As of static entries, the trails for the following malicious entities (e.g. malware C&Cs or sinkholes) have been manually included (from various AV reports and personal research): ``` -aboc, adwind, alienspy, almalocker, alureon, android_acecard, -android_adrd, android_alienspy, android_arspam, -android_backflash, android_basebridge, android_boxer, -android_chuli, android_claco, android_coolreaper, -android_counterclank, android_cyberwurx, android_dendoroid, -android_dougalek, android_droidjack, android_droidkungfu, -android_enesoluty, android_ewalls, android_exprespam, -android_fakebanco, android_fakedown, android_fakeinst, -android_fakelog, android_fakemart, android_fakemrat, -android_fakeneflic, android_fakesecsuit, android_feabme, -android_flexispy, android_frogonal, android_geinimi, -android_ghostpush, android_ginmaster, android_gmaster, -android_godwon, android_golddream, android_gonesixty, -android_ibanking, android_kemoge, android_lockdroid, -android_lovetrap, android_maistealer, android_maxit, -android_oneclickfraud, android_opfake, android_ozotshielder, -android_pikspam, android_pjapps, android_qdplugin, -android_repane, android_roidsec, android_samsapo, -android_sandorat, android_selfmite, android_simplocker, -android_skullkey, android_sndapps, android_spytekcell, -android_stealer, android_stels, android_teelog, android_tetus, -android_tonclank, android_torec, android_uracto, -android_usbcleaver, android_walkinwat, android_windseeker, -android_zertsecurity, androm, andromem, angler, anuna, arec, -aridviper, artro, autoit, avalanche, avrecon, axpergle, babar, -bachosens, badblock, balamid, bamital, bankapol, bankpatch, -banloa, banprox, bayrob, bedep, blackenergy, blackvine, -blockbuster, bredolab, bubnix, bucriv, buterat, camerashy, -carbanak, carberp, careto, casper, cerber, changeup, chanitor, -chekua, cheshire, chewbacca, chisbur, cleaver, cloud_atlas, -conficker, contopee, copykittens, corebot, cosmicduke, -couponarific, criakl, cridex, crilock, cryakl, cryptinfinite, -cryptodefense, cryptolocker, cryptowall, ctblocker, cutwail, -darkhotel, defru, desertfalcon, destory, dnschanger, -dnsmessenger, dnstrojan, dorifel, dorkbot, drapion, dridex, -dukes, dursg, dyreza, elf_aidra, elf_billgates, elf_darlloz, -elf_ekoms, elf_fysbis, elf_groundhog, elf_hacked_mint, -elf_mayhem, elf_mokes, elf_pinscan, elf_rekoobe, elf_shelldos, -elf_sshscan, elf_themoon, elf_turla, elf_xnote, elf_xorddos, -elpman, emogen, emotet, equation, evilbunny, ewind, expiro, -fakeav, fakeran, fantom, fareit, fbi_ransomware, fiexp, -fignotok, fin4, finfisher, fraudload, fynloski, fysna, gamarue, -gauss, gbot, generic, golroted, gozi, groundbait, harnig, -hawkeye, helompy, hiloti, hinired, htran, immortal, injecto, -ios_keyraider, ios_muda, ios_oneclickfraud, ios_specter, -ismdoor, jenxcus, kegotip, keydnap, kingslayer, kolab, -koobface, korgo, korplug, kovter, kradellsh, kulekmoko, -lazarus, locky, lollipop, lotus_blossom, luckycat, majikpos, -malwaremustdie, marsjoke, mdrop, mebroot, mestep, mhretriev, -miniduke, misogow, modpos, morto, nanocor, nbot, necurs, -nemeot, neshuta, nettraveler, netwire, neurevt, nexlogger, -nivdort, nonbolqu, nuqel, nwt, nymaim, odcodc, oficla, onkods, -optima, osx_keranger, osx_keydnap, osx_salgorea, -osx_wirelurker, palevo, pdfjsc, pegasus, pepperat, phytob, -picgoo, pift, plagent, plugx, ponmocup, poshcoder, potao, -powelike, proslikefan, pushdo, pykspa, qakbot, quasar, ramnit, -ransirac, reactorbot, redoctober, redsip, remcos, renocide, -reveton, revetrat, rovnix, runforestrun, russian_doll, rustock, -sakurel, sality, satana, sathurbot, scarcruft, scarletmimic, -scieron, seaduke, sednit, sefnit, selfdel, shifu, shylock, -siesta, silentbrute, silly, simda, sinkhole_abuse, -sinkhole_anubis, sinkhole_arbor, sinkhole_bitdefender, -sinkhole_blacklab, sinkhole_botnethunter, sinkhole_certgovau, -sinkhole_certpl, sinkhole_checkpoint, sinkhole_cirtdk, -sinkhole_conficker, sinkhole_cryptolocker, sinkhole_drweb, -sinkhole_dynadot, sinkhole_dyre, sinkhole_farsight, -sinkhole_fbizeus, sinkhole_fitsec, sinkhole_fnord, -sinkhole_gameoverzeus, sinkhole_georgiatech, sinkhole_gladtech, -sinkhole_honeybot, sinkhole_kaspersky, sinkhole_microsoft, -sinkhole_rsa, sinkhole_secureworks, sinkhole_shadowserver, -sinkhole_sidnlabs, sinkhole_sinkdns, sinkhole_sofacy, -sinkhole_sugarbucket, sinkhole_tech, sinkhole_unknown, -sinkhole_virustracker, sinkhole_wapacklabs, sinkhole_zinkhole, -skeeyah, skynet, skyper, smsfakesky, snake, snifula, snort, -sockrat, sofacy, sohanad, spyeye, stabuniq, stonedrill, -stuxnet, synolocker, tdss, teamspy, teerac, teslacrypt, -themida, tibet, tinba, torpig, torrentlocker, troldesh, turla, -unruy, upatre, utoti, vawtrak, vbcheman, vinderuf, virtum, -virut, vittalia, vobfus, volatilecedar, vundo, waledac, -waterbug, wecorl, wndred, xadupi, xcodeghost, xtrat, yenibot, -yimfoca, zaletelly, zcrypt, zemot, zeroaccess, zeus, zherotee, -zlader, zlob, zombrari, zxshell, etc. +1ms0rry, 404, 9002, aboc, absent, ab, acbackdoor, acridrain, activeagent, +adrozek, advisorbot, adwind, adylkuzz, adzok, afrodita, agaadex, agenttesla, +aldibot, alina, allakore, almalocker, almashreq, alpha, alureon, amadey, +amavaldo, amend_miner, ammyyrat, android_acecard, android_actionspy, +android_adrd, android_ahmythrat, android_alienspy, android_andichap, +android_androrat, android_anubis, android_arspam, android_asacub, +android_backflash, android_bankbot, android_bankun, android_basbanke, +android_basebridge, android_besyria, android_blackrock, android_boxer, +android_buhsam, android_busygasper, android_calibar, android_callerspy, +android_camscanner, android_cerberus, android_chuli, android_circle, +android_claco, android_clickfraud, android_cometbot, android_cookiethief, +android_coolreaper, android_copycat, android_counterclank, android_cyberwurx, +android_darkshades, android_dendoroid, android_dougalek, android_droidjack, +android_droidkungfu, android_enesoluty, android_eventbot, android_ewalls, +android_ewind, android_exodus, android_exprespam, android_fakeapp, +android_fakebanco, android_fakedown, android_fakeinst, android_fakelog, +android_fakemart, android_fakemrat, android_fakeneflic, android_fakesecsuit, +android_fanta, android_feabme, android_flexispy, android_fobus, +android_fraudbot, android_friend, android_frogonal, android_funkybot, +android_gabas, android_geinimi, android_generic, android_geost, +android_ghostpush, android_ginmaster, android_ginp, android_gmaster, +android_gnews, android_godwon, android_golddream, android_goldencup, +android_golfspy, android_gonesixty, android_goontact, android_gplayed, +android_gustuff, android_gypte, android_henbox, android_hiddad, +android_hydra, android_ibanking, android_joker, android_jsmshider, +android_kbuster, android_kemoge, android_ligarat, android_lockdroid, +android_lotoor, android_lovetrap, android_malbus, android_mandrake, +android_maxit, android_mobok, android_mobstspy, android_monokle, +android_notcompatible, android_oneclickfraud, android_opfake, +android_ozotshielder, android_parcel, android_phonespy, android_pikspam, +android_pjapps, android_qdplugin, android_raddex, android_ransomware, +android_redalert, android_regon, android_remotecode, android_repane, +android_riltok, android_roamingmantis, android_roidsec, android_rotexy, +android_samsapo, android_sandrorat, android_selfmite, android_shadowvoice, +android_shopper, android_simbad, android_simplocker, android_skullkey, +android_sndapps, android_spynote, android_spytekcell, android_stels, +android_svpeng, android_swanalitics, android_teelog, android_telerat, +android_tetus, android_thiefbot, android_tonclank, android_torec, +android_triada, android_uracto, android_usbcleaver, android_viceleaker, +android_vmvol, android_walkinwat, android_windseeker, android_wirex, +android_wolfrat, android_xavirad, android_xbot007, android_xerxes, +android_xhelper, android_xploitspy, android_z3core, android_zertsecurity, +android_ztorg, andromeda, antefrigus, antibot, anubis, anuna, apocalypse, +apt_12, apt_17, apt_18, apt_23, apt_27, apt_30, apt_33, apt_37, apt_38, +apt_aridviper, apt_babar, apt_bahamut, etc. ``` ## Architecture @@ -151,23 +120,52 @@ Maltrail is based on the **Traffic** -> **Sensor** <-> **Server** <- Note: **Server** component can be skipped altogether, and just use the standalone **Sensor**. In such case, all events would be stored in the local logging directory, while the log entries could be examined either manually or by some CSV reading application. +## Demo pages + +Fully functional demo pages with collected real-life threats can be found [here](https://maltraildemo.github.io/). + +## Requirements + +To run Maltrail properly, [Python](http://www.python.org/download/) **2.6**, **2.7** or **3.x** is required on \*nix/BSD system, together with installed [pcapy-ng](https://pypi.org/project/pcapy-ng/) package. + +**NOTE:** Using of ```pcapy``` lib instead of ```pcapy-ng``` can lead to incorrect work of Maltrail, especially on **Python 3.x** environments. [Examples](https://github.com/stamparm/maltrail/issues?q=label%3Apcapy-ng-related+is%3Aclosed). + +- **Sensor** component requires at least 1GB of RAM to run in single-process mode or more if run in multiprocessing mode, depending on the value used for option `CAPTURE_BUFFER`. Additionally, **Sensor** component (in general case) requires administrative/root privileges. + +- **Server** component does not have any special requirements. + ## Quick start The following set of commands should get your Maltrail **Sensor** up and running (out of the box with default settings and monitoring interface "any"): +- For **Ubuntu/Debian** + +```sh +sudo apt-get install git python3 python3-dev python3-pip python-is-python3 libpcap-dev build-essential procps schedtool +sudo pip3 install pcapy-ng +git clone --depth 1 https://github.com/stamparm/maltrail.git +cd maltrail +sudo python3 sensor.py ``` -sudo apt-get install git python-pcapy -git clone https://github.com/stamparm/maltrail.git + +- For **SUSE/openSUSE** + +```sh +sudo zypper install gcc gcc-c++ git libpcap-devel python3-devel python3-pip procps schedtool +sudo pip3 install pcapy-ng +git clone --depth 1 https://github.com/stamparm/maltrail.git cd maltrail -sudo python sensor.py +sudo python3 sensor.py ``` +- For **Docker** environment instructions can be found [here](docker). + ![Sensor](https://i.imgur.com/E9tt2ek.png) To start the (optional) **Server** on same machine, open a new terminal and execute the following: -``` -[[ -d maltrail ]] || git clone https://github.com/stamparm/maltrail.git +```sh +[[ -d maltrail ]] || git clone --depth 1 https://github.com/stamparm/maltrail.git cd maltrail python server.py ``` @@ -176,16 +174,25 @@ python server.py To test that everything is up and running execute the following: -``` +```sh ping -c 1 136.161.101.53 cat /var/log/maltrail/$(date +"%Y-%m-%d").log ``` ![Test](https://i.imgur.com/NYJg6Kl.png) -To stop **Sensor** and **Server** instances (if running in background) execute the following: +Also, to test the capturing of DNS traffic you can try the following: +```sh +nslookup morphed.ru +cat /var/log/maltrail/$(date +"%Y-%m-%d").log ``` + +![Test2](https://i.imgur.com/62oafEe.png) + +To stop **Sensor** and **Server** instances (if running in background) execute the following: + +```sh sudo pkill -f sensor.py pkill -f server.py ``` @@ -203,8 +210,21 @@ Sensor's configuration can be found inside the `maltrail.conf` file's section `[ ![Sensor's configuration](https://i.imgur.com/8yZKH14.png) If option `USE_MULTIPROCESSING` is set to `true` then all CPU cores will be used. One core will be used only for packet capture (with appropriate affinity, IO priority and nice level settings), while other cores will be used for packet processing. Otherwise, everything will be run on a single core. Option `USE_FEED_UPDATES` can be used to turn off the trail updates from feeds altogether (and just use the provided static ones). Option `UPDATE_PERIOD` contains the number of seconds between each automatic trails update (Note: default value is set to `86400` (i.e. one day)) by using definitions inside the `trails` directory (Note: both **Sensor** and **Server** take care of the trails update). Option `CUSTOM_TRAILS_DIR` can be used by user to provide location of directory containing the custom trails (`*.txt`) files. + Option `USE_HEURISTICS` turns on heuristic mechanisms (e.g. `long domain name (suspicious)`, `excessive no such domain name (suspicious)`, `direct .exe download (suspicious)`, etc.), potentially introducing false positives. Option `CAPTURE_BUFFER` presents a total memory (in bytes of percentage of total physical memory) to be used in case of multiprocessing mode for storing packet capture in a ring buffer for further processing by non-capturing processes. Option `MONITOR_INTERFACE` should contain the name of the capturing interface. Use value `any` to capture from all interfaces (if OS supports this). Option `CAPTURE_FILTER` should contain the network capture (`tcpdump`) filter to skip the uninteresting packets and ease the capturing process. Option `SENSOR_NAME` contains the name that should be appearing inside the events `sensor_name` value, so the event from one sensor could be distinguished from the other. If option `LOG_SERVER` is set, then all events are being sent remotely to the **Server**, otherwise they are stored directly into the logging directory set with option `LOG_DIR`, which can be found inside the `maltrail.conf` file's section `[All]`. In case that the option `UPDATE_SERVER` is set, then all the trails are being pulled from the given location, otherwise they are being updated from trails definitions located inside the installation itself. +Options `SYSLOG_SERVER` and/or `LOGSTASH_SERVER` can be used to send sensor events (i.e. log data) to non-Maltrail servers. In case of `SYSLOG_SERVER`, event data will be sent in CEF (*Common Event Format*) format to UDP (e.g. Syslog) service listening at the given address (e.g. `192.168.2.107:514`), while in case of `LOGSTASH_SERVER` event data will be sent in JSON format to UDP (e.g. Logstash) service listening at the given address (e.g. `192.168.2.107:5000`). + +Example of event data being sent over UDP is as follows: + +- For option `SYSLOG_SERVER` (Note: `LogSeverity` values are 0 (for low), 1 (for medium) and 2 (for high)): + +```Dec 24 15:05:55 beast CEF:0|Maltrail|sensor|0.27.68|2020-12-24|andromeda (malware)|2|src=192.168.5.137 spt=60453 dst=8.8.8.8 dpt=53 trail=morphed.ru ref=(static)``` + +- For option `LOGSTASH_SERVER`: + +```{"timestamp": 1608818692, "sensor": "beast", "severity": "high", "src_ip": "192.168.5.137", "src_port": 48949, "dst_ip": "8.8.8.8", "dst_port": 53, "proto": "UDP", "type": "DNS", "trail": "morphed.ru", "info": "andromeda (malware)", "reference": "(static)"}``` + When running the sensor (e.g. `sudo python sensor.py`) for the first time and/or after a longer period of non-running, it will automatically update the trails from trail definitions (Note: stored inside the `trails` directory). After the initialization, it will start monitoring the configured interface (option `MONITOR_INTERFACE` inside the `maltrail.conf`) and write the events to either the configured log directory (option `LOG_DIR` inside the `maltrail.conf` file's section `[All]`) or send them remotely to the logging/reporting **Server** (option `LOG_SERVER`). ![Sensor run](https://i.imgur.com/A0qROp8.png) @@ -227,6 +247,36 @@ Subsection `USERS` contains user's configuration settings. Each user entry consi Option `UDP_ADDRESS` contains the server's log collecting listening address (Note: use `0.0.0.0` to listen on all interfaces), while option `UDP_PORT` contains listening port value. If turned on, when used in combination with option `LOG_SERVER`, it can be used for distinct (multiple) **Sensor** <-> **Server** architecture. +Option `FAIL2BAN_REGEX` contains the regular expression (e.g. `attacker|reputation|potential[^"]*(web scan|directory traversal|injection|remote code|iot-malware download|spammer|mass scanner`) to be used in `/fail2ban` web calls for extraction of today's attacker source IPs. This allows the usage of IP blocking mechanisms (e.g. `fail2ban`, `iptables` or `ipset`) by periodic pulling of blacklisted IP addresses from remote location. Example usage would be the following script (e.g. run as a `root` cronjob on a minute basis): + +```sh +#!/bin/bash +ipset -q flush maltrail +ipset -q create maltrail hash:net +for ip in $(curl http://127.0.0.1:8338/fail2ban 2>/dev/null | grep -P '^[0-9.]+$'); do ipset add maltrail $ip; done +iptables -I INPUT -m set --match-set maltrail src -j DROP +``` + +Option `BLACKLIST` allows to build regular expressions to apply on one field. For each rule, the syntax is : ` ` where : +* `field` indicates the field to compage, it can be: `src_ip`,`src_port`,`dst_ip`,`dst_port`,`protocol`,`type`,`trail` or `filter`. +* `control` can be either `~` for *matches* or `!~` for *doesn't match* +* `regexp` is the regular expression to apply to the field. +Chain another rule with the `and` keyword (the `or` keyword is not supported, just add a line for this). + +You can use the keyword `BLACKLIST` alone or add a name : `BLACKLIST_NAME`. In the latter case, the url will be : `/blacklist/name` + +For example, the following will build an out blacklist for all traffic from another source than `192.168.0.0/16` to destination port `SSH` or matching the filters `scan` or `known attacker` +``` +BLACKLIST_OUT + src_ip !~ ^192.168. and dst_port ~ ^22$ + src_ip !~ ^192.168. and filter ~ scan + src_ip !~ ^192.168. and filter ~ known attacker + +BLACKLIST_IN + src_ip ~ ^192.168. and filter ~ malware +``` +The way to build ipset blacklist is the same (see above) excepted that URLs will be `/blacklist/in` and `/blacklist/out` in our example. + Same as for **Sensor**, when running the **Server** (e.g. `python server.py`) for the first time and/or after a longer period of non-running, if option `USE_SERVER_UPDATE_TRAILS` is set to `true`, it will automatically update the trails from trail definitions (Note: stored inside the `trails` directory). Its basic function is to store the log entries inside the logging directory (i.e. option `LOG_DIR` inside the `maltrail.conf` file's section `[All]`) and provide the web reporting interface for presenting those same entries to the end-user (Note: there is no need install the 3rd party web server packages like Apache): ![Server run](https://i.imgur.com/GHdGPw7.png) @@ -265,17 +315,17 @@ When moving mouse over `src_ip` and `dst_ip` table entries, information tooltip ![On mouse over IP](https://i.imgur.com/BgKchAX.png) -Event details (e.g. `src_port`, `dst_port`, `proto`, etc.) that differ inside same threat entry are condensed in form of a cloud icon (i.e. ![Cloud ellipsis](https://raw.githubusercontent.com/stamparm/maltrail/master/html/images/ellipsis.png)). This is performed to get an usable reporting interface with as less rows as possible. Moving mouse over such icon will result in a display of an information tooltip with all items held (e.g. all port numbers being scanned by `attacker`): +Event details (e.g. `src_port`, `dst_port`, `proto`, etc.) that differ inside same threat entry are condensed in form of a bubble icon (i.e. ![Ellipsis](https://raw.githubusercontent.com/stamparm/maltrail/master/html/images/ellipsis.png)). This is performed to get an usable reporting interface with as less rows as possible. Moving mouse over such icon will result in a display of an information tooltip with all items held (e.g. all port numbers being scanned by `attacker`): -![On mouse over cloud](https://i.imgur.com/BfYT2u7.png) +![On mouse over bubble](https://i.imgur.com/BfYT2u7.png) Clicking on one such icon will open a new dialog containing all stored items (Note: in their uncondensed form) ready to be Copy-Paste(d) for further analysis: ![Ctrl-C dialog](https://i.imgur.com/9pgMpiR.png) -When hovering mouse pointer over the threat's trail for couple of seconds it will result in a frame consisted of results using the trail as a search term performed against [DuckDuckGo](https://duckduckgo.com/) search engine. In lots of cases, this provides basic information about the threat itself, eliminating the need for user to do the manual search for it. In upper right corner of the opened frame window there are two extra buttons. By clicking the first one (i.e. ![New tab icon](https://raw.githubusercontent.com/stamparm/maltrail/master/html/images/newtab.png)), the resulting frame will be opened inside the new browser's tab (or window), while by clicking the second one (i.e. ![Close icon](https://raw.githubusercontent.com/stamparm/maltrail/master/html/images/close.png)) will immediately close the frame (Note: the same action is achieved by moving the mouse pointer outside the frame borders): +When hovering mouse pointer over the threat's trail for couple of seconds it will result in a frame consisted of results using the trail as a search term performed against ~~[Search Encrypt](https://www.searchencrypt.com/)~~ [searX](https://searx.nixnet.services/) search engine. In lots of cases, this provides basic information about the threat itself, eliminating the need for user to do the manual search for it. In upper right corner of the opened frame window there are two extra buttons. By clicking the first one (i.e. ![New tab icon](https://raw.githubusercontent.com/stamparm/maltrail/master/html/images/newtab.png)), the resulting frame will be opened inside the new browser's tab (or window), while by clicking the second one (i.e. ![Close icon](https://raw.githubusercontent.com/stamparm/maltrail/master/html/images/close.png)) will immediately close the frame (Note: the same action is achieved by moving the mouse pointer outside the frame borders): -![On mouse over trail](https://i.imgur.com/ppoMHub.png) +![On mouse over trail](https://i.imgur.com/ZxnHn1N.png) For each threat there is a column `tag` that can be filled with arbitrary "tags" to closely describe all threats sharing the same trail. Also, it is a great way to describe threats individually, so all threats sharing the same tag (e.g. `yahoo`) could be grouped out later: @@ -297,9 +347,9 @@ Here is a reverse DNS and WHOIS lookup of the "attacker"'s address: ![Shodan 1](https://i.imgur.com/LQ6Vu00.png) -When hovering mouse pointer over the `trail` column's content (IP address), you'll be presented with the search results from [DuckDuckGo](https://duckduckgo.com/) where you'll be able to find more information about the "attacker" (i.e. Shodan): +When hovering mouse pointer over the `trail` column's content (IP address), you'll be presented with the search results from [searX](https://searx.nixnet.services/) where you'll be able to find more information about the "attacker": -![Shodan 2](https://i.imgur.com/sv7ONzk.png) +![Shodan 2](https://i.imgur.com/vIzB8bA.png) In the `dst_ip` column, if you have a large organization, you'll be presented with large list of scanned IP addresses: ![Shodan 3](https://i.imgur.com/EhAtXs7.png) @@ -392,7 +442,7 @@ By using filter `ipinfo` all potentially infected computers in our organization' #### Suspicious direct file downloads -Maltrail tracks all suspicious direct file download attempts (e.g. `.apk`, `.exe` and `.scr` file extensions). This can trigger lots of false positives, but eventually could help in reconstruction of the chain of infection (Note: legitimate service providers, like Google, usually use encrypted HTTPS to perform this kind of downloads): +Maltrail tracks all suspicious direct file download attempts (e.g. `.apk`, `.bin`, `.class`, `.chm`, `.dll`, `.egg`, `.exe`, `.hta`, `.hwp`, `.lnk`, `.ps1`, `.scr`, `.sct`, `.wbk` and `.xpi` file extensions). This can trigger lots of false positives, but eventually could help in reconstruction of the chain of infection (Note: legitimate service providers, like Google, usually use encrypted HTTPS to perform this kind of downloads): ![Direct .exe download](https://i.imgur.com/jr5BS1h.png) @@ -406,7 +456,7 @@ In following example, web application vulnerability scan has been marked as "sus ![Vulnerability scan](https://i.imgur.com/QzcaEsG.png) -If we click on the cloud icon (i.e. ![Cloud ellipsis](https://raw.githubusercontent.com/stamparm/maltrail/master/html/images/ellipsis.png)) for details and copy paste the whole content to a textual file, we'll be able to see all suspicious HTTP requests: +If we click on the bubble icon (i.e. ![Ellipsis](https://raw.githubusercontent.com/stamparm/maltrail/master/html/images/ellipsis.png)) for details and copy paste the whole content to a textual file, we'll be able to see all suspicious HTTP requests: ![Vulnerability scan requests](https://i.imgur.com/XY9K01o.png) @@ -438,9 +488,9 @@ Like in all other security solutions, Maltrail is prone to "[false positives](ht ![Google false positive 1](https://i.imgur.com/HFvCNNK.png) -By hovering mouse over the trail, frame with results from [DuckDuckGo](https://duckduckgo.com/) search show that this is a regular Google's server: +By hovering mouse over the trail, frame with results from [searX](https://searx.nixnet.services/) search show that this is (most probably) a regular Google's server: -![Google false positive 2](https://i.imgur.com/4cS9NJB.png) +![Google false positive 2](https://i.imgur.com/i3oydv6.png) As another example, access to regular `.work` domains (popular TLD for malicious purposes) resulted with the following threat: @@ -450,17 +500,100 @@ Nevertheless, administrator(s) should invest some extra time and check (with oth ![Suspicious .ws](https://i.imgur.com/bOLmXUE.png) -## Requirements +## Best practice(s) + +1. Install Maltrail: + +- On **Ubuntu/Debian** + + ```sh + sudo apt-get install git python3 python3-dev python3-pip python-is-python3 libpcap-dev build-essential procps schedtool + sudo pip3 install pcapy-ng + cd /tmp + git clone --depth 1 https://github.com/stamparm/maltrail.git + sudo mv /tmp/maltrail /opt + sudo chown -R $USER:$USER /opt/maltrail + ``` + +- On **SUSE/openSUSE** + + ```sh + sudo zypper install gcc gcc-c++ git libpcap-devel python3-devel python3-pip procps schedtool + sudo pip3 install pcapy-ng + cd /tmp + git clone --depth 1 https://github.com/stamparm/maltrail.git + sudo mv /tmp/maltrail /opt + sudo chown -R $USER:$USER /opt/maltrail + ``` + +2. Set working environment: + + ```sh + sudo mkdir -p /var/log/maltrail + sudo mkdir -p /etc/maltrail + sudo cp /opt/maltrail/maltrail.conf /etc/maltrail + sudo nano /etc/maltrail/maltrail.conf + ``` + +3. Set running environment: + + * `crontab -e # autostart server & periodic update` + + ``` + */5 * * * * if [ -n "$(ps -ef | grep -v grep | grep 'server.py')" ]; then : ; else python3 /opt/maltrail/server.py -c /etc/maltrail/maltrail.conf; fi + 0 1 * * * cd /opt/maltrail && git pull + ``` + + * `sudo crontab -e # autostart sensor & periodic restart` + + ``` + */1 * * * * if [ -n "$(ps -ef | grep -v grep | grep 'sensor.py')" ]; then : ; else python3 /opt/maltrail/sensor.py -c /etc/maltrail/maltrail.conf; fi + 2 1 * * * /usr/bin/pkill -f maltrail + ``` + +4. Enable as systemd services (Linux only): + + ```sh + sudo cp /opt/maltrail/maltrail-sensor.service /etc/systemd/system/maltrail-sensor.service + sudo cp /opt/maltrail/maltrail-server.service /etc/systemd/system/maltrail-server.service + sudo systemctl daemon-reload + sudo systemctl start maltrail-server.service + sudo systemctl start maltrail-sensor.service + sudo systemctl enable maltrail-server.service + sudo systemctl enable maltrail-sensor.service + systemctl status maltrail-server.service && systemctl status maltrail-sensor.service + + ``` + + **Note**: ```/maltrail-sensor.service``` can be started as dedicated service without pre-started ```/maltrail-server.service```. This is useful for case, when ```/maltrail-server.service``` is installed and works on another machine in you network environment. -To properly run the Maltrail, [Python](http://www.python.org/download/) **2.6.x** or **2.7.x** is required, together with [pcapy](http://corelabs.coresecurity.com/index.php?module=Wiki&action=view&type=tool&name=Pcapy) (e.g. `sudo apt-get install python-pcapy`). There are no other requirements, other than to run the **Sensor** component with the administrative/root privileges. ## License -This software is provided under under a MIT License. See the accompanying [LICENSE](https://github.com/stamparm/maltrail/blob/master/LICENSE) file for more information. +This software is provided under a MIT License. See the accompanying [LICENSE](https://github.com/stamparm/maltrail/blob/master/LICENSE) file for more information. + +## Sponsors + +* [Sansec](https://sansec.io/) (2024-) +* [Sansec](https://sansec.io/) (2020-2021) + +## Developers + +* Miroslav Stampar ([@stamparm](https://github.com/stamparm)) +* Mikhail Kasimov ([@MikhailKasimov](https://github.com/MikhailKasimov)) + +## Presentations + +* 47th TF-CSIRT Meeting, Prague (Czech Republic), 2016 ([slides](https://www.terena.org/activities/tf-csirt/meeting47/M.Stampar-Maltrail.pdf)) + +## Publications + +* Detect attacks on your network with Maltrail, Linux Magazine, 2022 ([Annotation](https://www.linux-magazine.com/Issues/2022/258/Maltrail)) +* Best Cyber Threat Intelligence Feeds ([SilentPush Review, 2022](https://www.silentpush.com/blog/best-cyber-threat-intelligence-feeds)) -## Donations +## Blacklist -This project is a result of numerous hours of software development. If you appreciate the work and you want to see it further developed, please consider making a donation via [PayPal](https://www.paypal.com/) to `miroslav.stampar@gmail.com` or via [Ƀitcoin](bitcoin:1JCtgmpC1eWvdHXrKfvMAunfvcaaMXLP5G) to `1JCtgmpC1eWvdHXrKfvMAunfvcaaMXLP5G`. +* Maltrail's daily updated blacklist of malware-related domains can be found [here](https://raw.githubusercontent.com/stamparm/aux/master/maltrail-malware-domains.txt). It is based on trails found at [trails/static/malware](trails/static/malware) and can be safely used for DNS traffic blocking purposes. ## Thank you @@ -469,3 +602,27 @@ This project is a result of numerous hours of software development. If you appre * James Lay * Ladislav Baco (@laciKE) * John Kristoff (@jtkdpu) +* Michael Münz (@mimugmail) +* David Brush +* @Godwottery +* Chris Wild (@briskets) + +## Third-party integrations + +* [FreeBSD Port](https://www.freshports.org/security/maltrail) +* [OPNSense Gateway Plugin](https://github.com/opnsense/plugins/pull/1257) +* [D4 Project](https://www.d4-project.org/2019/09/25/maltrail-integration.html) +* [BlackArch Linux](https://github.com/BlackArch/blackarch/blob/master/packages/maltrail/PKGBUILD) +* [Validin LLC](https://twitter.com/ValidinLLC/status/1719666086390517762) +* [GScan](https://github.com/grayddq/GScan) 1 +* [MalwareWorld](https://www.malwareworld.com/) 1 +* [oisd | domain blocklist](https://oisd.nl/?p=inc) 1 +* [NextDNS](https://github.com/nextdns/metadata/blob/e0c9c7e908f5d10823b517ad230df214a7251b13/security/threat-intelligence-feeds.json) 1 +* [NoTracking](https://github.com/notracking/hosts-blocklists/blob/master/SOURCES.md) 1 +* [mobileAudit](https://github.com/mpast/mobileAudit#environment-variables) 1 +* [Mobile-Security-Framework-MobSF](https://github.com/MobSF/Mobile-Security-Framework-MobSF/commit/12b07370674238fa4281fc7989b34decc2e08876) 1 +* [pfBlockerNG-devel](https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/www/pfblockerng/pfblockerng_feeds.json) 1 +* [Sansec eComscan](https://sansec.io/kb/about-ecomscan/ecomscan-license)1 +* [Palo Alto Networks Cortex XSOAR](https://xsoar.pan.dev/docs/reference/integrations/github-maltrail-feed)1 + +1 Using (only) trails diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000000..99e8dc14a27 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,23 @@ +--- +title: Maltrail Security Vulnerability Reports +category: contributing +layout: default +SPDX-License-Identifier: MIT +--- + +## Reporting Maltrail Security Vulnerability + +Maltrail team appreciates your efforts on discovering security vulnerabilities in [Maltrail](https://github.com/stamparm/maltrail): Malicious traffic detection system. + +If you discover a Maltrail security vulnerability, we'd appreciate a non-public disclosure. Maltrail team developers can be contacted privately on the **maltrail.vulns[@]gmail.com** email address. + +The disclosure of discovered security vulnerability will be coordinated with Maltrail team. + +Maltrail's [issues tracker](https://github.com/stamparm/maltrail/issues) and [pull requests tracker](https://github.com/stamparm/maltrail/pulls) are fully public. + +## Supported Versions + + +| Version | Supported | +| ------- | ------------------ | +| All versions | :white_check_mark: | diff --git a/core/__init__.py b/core/__init__.py index 39205b91796..0b19092dc0c 100644 --- a/core/__init__.py +++ b/core/__init__.py @@ -1,7 +1,7 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ diff --git a/core/addr.py b/core/addr.py index 84e7ae0da09..014520d5f21 100644 --- a/core/addr.py +++ b/core/addr.py @@ -1,15 +1,17 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ import re +from core.compat import xrange + def addr_to_int(value): _ = value.split('.') - return (long(_[0]) << 24) + (long(_[1]) << 16) + (long(_[2]) << 8) + long(_[3]) + return (int(_[0]) << 24) + (int(_[1]) << 16) + (int(_[2]) << 8) + int(_[3]) def int_to_addr(value): return '.'.join(str(value >> n & 0xff) for n in (24, 16, 8, 0)) @@ -21,12 +23,51 @@ def compress_ipv6(address): zeros = re.findall("(?:0000:)+", address) if zeros: address = address.replace(sorted(zeros, key=lambda _: len(_))[-1], ":", 1) - address = re.sub(r"(\A|:)0+(\w)", "\g<1>\g<2>", address) + address = re.sub(r"(\A|:)0+(\w)", r"\g<1>\g<2>", address) if address == ":1": address = "::1" return address # Note: socket.inet_ntop not available everywhere (Reference: https://docs.python.org/2/library/socket.html#socket.inet_ntop) def inet_ntoa6(packed_ip): - _ = packed_ip.encode("hex") + _ = packed_ip.hex() if hasattr(packed_ip, "hex") else packed_ip.encode("hex") return compress_ipv6(':'.join(_[i:i + 4] for i in xrange(0, len(_), 4))) + +def expand_range(value): + retval = [] + value = value.strip() + + match = re.match(r"(\d+\.\d+\.\d+\.\d+)/(\d+)", value) + if match: + prefix, mask = match.groups() + mask = int(mask) + assert(mask <= 32) + + start_int = addr_to_int(prefix) & make_mask(mask) + end_int = start_int | ((1 << 32 - mask) - 1) + if 0 <= end_int - start_int <= 65536: + address = start_int + while start_int <= address <= end_int: + retval.append(int_to_addr(address)) + address += 1 + + elif '-' in value: + start, end = value.split('-') + start_int, end_int = addr_to_int(start), addr_to_int(end) + current = start_int + while start_int <= current <= end_int: + retval.append(int_to_addr(current)) + current += 1 + + else: + retval.append(value) + + return retval + +def addr_port(addr, port): + if ':' in addr and '.' not in addr: + retval = "[%s]:%s" % (addr.strip("[]"), port) + else: + retval = "%s:%s" % (addr, port) + + return retval diff --git a/core/attribdict.py b/core/attribdict.py index 14cdb5844dc..e66c55f62ba 100644 --- a/core/attribdict.py +++ b/core/attribdict.py @@ -1,13 +1,13 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ class AttribDict(dict): def __getattr__(self, name): - return self[name] if name in self else None + return self.get(name) def __setattr__(self, name, value): - self[name] = value \ No newline at end of file + self[name] = value diff --git a/core/colorized.py b/core/colorized.py new file mode 100644 index 00000000000..dfc23c9e41c --- /dev/null +++ b/core/colorized.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python + +import os +import re +import sys + +from core.enums import BACKGROUND +from core.enums import COLOR +from core.enums import SEVERITY + +IS_TTY = hasattr(sys.stdout, "fileno") and os.isatty(sys.stdout.fileno()) + +class ColorizedStream: + def __init__(self, original): + self._original = original + self._log_colors = {'i': COLOR.LIGHT_BLUE, '!': COLOR.LIGHT_YELLOW, '*': COLOR.LIGHT_CYAN, 'x': COLOR.BOLD_LIGHT_RED, '?': COLOR.LIGHT_YELLOW, 'o': COLOR.BOLD_WHITE, '+': COLOR.BOLD_LIGHT_GREEN, '^': COLOR.BOLD_LIGHT_GREEN} + self._severity_colors = {SEVERITY.LOW: COLOR.BOLD_LIGHT_CYAN, SEVERITY.MEDIUM: COLOR.BOLD_LIGHT_YELLOW, SEVERITY.HIGH: COLOR.BOLD_LIGHT_RED} + self._type_colors = {"DNS": BACKGROUND.BLUE, "UA": BACKGROUND.MAGENTA, "IP": BACKGROUND.RED, "URL": BACKGROUND.YELLOW, "HTTP": BACKGROUND.GREEN, "IPORT": BACKGROUND.RED} + self._info_colors = {"malware": COLOR.LIGHT_RED, "suspicious": COLOR.LIGHT_YELLOW, "malicious": COLOR.YELLOW} + + def write(self, text): + match = re.search(r"\A(\s*)\[(.)\]", text) + if match and match.group(2) in self._log_colors: + text = text.replace(match.group(0), "%s[%s%s%s]" % (match.group(1), self._log_colors[match.group(2)], match.group(2), COLOR.RESET), 1) + + if "Maltrail (" in text: + text = re.sub(r"\((sensor|server)\)", lambda match: "(%s%s%s)" % ({"sensor": COLOR.BOLD_LIGHT_GREEN, "server": COLOR.BOLD_LIGHT_MAGENTA}[match.group(1)], match.group(1), COLOR.RESET), text) + text = re.sub(r"https?://[\w.:/?=]+", lambda match: "%s%s%s%s" % (COLOR.BLUE, COLOR.UNDERLINE, match.group(0), COLOR.RESET), text) + + if "Usage: " in text: + text = re.sub(r"(.*Usage: )(.+)", r"\g<1>%s\g<2>%s" % (COLOR.BOLD_WHITE, COLOR.RESET), text) + + if text.startswith('"2'): + text = re.sub(r"(TCP|UDP|ICMP) ([A-Z]+)", lambda match: "%s %s%s%s" % (match.group(1), self._type_colors.get(match.group(2), COLOR.WHITE), match.group(2), COLOR.RESET), text) + text = re.sub(r'"([^"]+)"', r'"%s\g<1>%s"' % (COLOR.LIGHT_GRAY, COLOR.RESET), text, count=1) + text = re.sub(r"\((malware|suspicious|malicious)\)", lambda match: "(%s%s%s)" % (self._info_colors.get(match.group(1), COLOR.WHITE), match.group(1), COLOR.RESET), text) + text = re.sub(r"\(([^)]+)\)", lambda match: "(%s%s%s)" % (COLOR.LIGHT_GRAY, match.group(1), COLOR.RESET) if match.group(1) not in self._info_colors else match.group(0), text) + + for match in re.finditer(r"[^\w]'([^']+)'", text): # single-quoted + text = text.replace("'%s'" % match.group(1), r"'%s%s%s'" % (COLOR.LIGHT_GRAY, match.group(1), COLOR.RESET)) + + self._original.write("%s" % text) + + def flush(self): + self._original.flush() + +def init_output(): + if IS_TTY: + sys.stderr = ColorizedStream(sys.stderr) + sys.stdout = ColorizedStream(sys.stdout) diff --git a/core/common.py b/core/common.py index d18390a8323..22ef4b99cf7 100644 --- a/core/common.py +++ b/core/common.py @@ -1,35 +1,43 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ +from __future__ import print_function + import csv import gzip +import io import os import re import sqlite3 -import StringIO -import subprocess -import urllib2 +import sys import zipfile import zlib from core.addr import addr_to_int from core.addr import int_to_addr +from core.compat import xrange +from core.settings import config +from core.settings import BOGON_IPS from core.settings import BOGON_RANGES from core.settings import CHECK_CONNECTION_URL from core.settings import CDN_RANGES -from core.settings import NAME from core.settings import IPCAT_SQLITE_FILE +from core.settings import IS_WIN +from core.settings import MAX_HELP_OPTION_LENGTH from core.settings import STATIC_IPCAT_LOOKUPS from core.settings import TIMEOUT -from core.settings import TRAILS_FILE +from core.settings import UNICODE_ENCODING +from core.settings import USER_AGENT from core.settings import WHITELIST from core.settings import WHITELIST_RANGES from core.settings import WORST_ASNS from core.trailsdict import TrailsDict +from thirdparty import six +from thirdparty.six.moves import urllib as _urllib _ipcat_cache = {} @@ -39,24 +47,29 @@ def retrieve_content(url, data=None, headers=None): """ try: - req = urllib2.Request("".join(url[i].replace(' ', "%20") if i > url.find('?') else url[i] for i in xrange(len(url))), data, headers or {"User-agent": NAME, "Accept-encoding": "gzip, deflate"}) - resp = urllib2.urlopen(req, timeout=TIMEOUT) + req = _urllib.request.Request("".join(url[i].replace(' ', "%20") if i > url.find('?') else url[i] for i in xrange(len(url))), data, headers or {"User-agent": USER_AGENT, "Accept-encoding": "gzip, deflate"}) + resp = _urllib.request.urlopen(req, timeout=TIMEOUT) retval = resp.read() encoding = resp.headers.get("Content-Encoding") if encoding: if encoding.lower() == "deflate": - data = StringIO.StringIO(zlib.decompress(retval, -15)) + data = io.BytesIO(zlib.decompress(retval, -15)) elif encoding.lower() == "gzip": - data = gzip.GzipFile("", "rb", 9, StringIO.StringIO(retval)) + data = gzip.GzipFile("", "rb", 9, io.BytesIO(retval)) retval = data.read() - except Exception, ex: - retval = ex.read() if hasattr(ex, "read") else getattr(ex, "msg", str()) + except Exception as ex: + retval = ex.read() if hasattr(ex, "read") else (get_ex_message(ex) or "") - if url.startswith("https://") and "handshake failure" in retval: + if url.startswith("https://") and isinstance(retval, str) and "handshake failure" in retval: return retrieve_content(url.replace("https://", "http://"), data, headers) - return retval or "" + retval = retval or b"" + + if six.PY3 and isinstance(retval, bytes): + retval = retval.decode(UNICODE_ENCODING, errors="replace") + + return retval def ipcat_lookup(address): if not address: @@ -135,16 +148,19 @@ def bogon_ip(address): except (IndexError, ValueError): pass + if address in BOGON_IPS: + return True + return False def check_sudo(): """ - Checks for sudo/Administrator privileges + Checks for root privileges """ check = None - if not subprocess.mswindows: + if not IS_WIN: if getattr(os, "geteuid"): check = os.geteuid() == 0 else: @@ -177,7 +193,7 @@ def process(current): items = [] previous = None start = None - for _ in sorted(current) + [unichr(65535)]: + for _ in sorted(current) + [six.unichr(65535)]: if previous is not None: if ord(_) == ord(previous) + 1: pass @@ -196,7 +212,7 @@ def process(current): return ("[%s]" % "".join(items)) if len(items) > 1 or '-' in items[0] else "".join(items) else: - return re.escape(current.keys()[0]) + return re.escape(list(current.keys())[0]) else: return ("(?:%s)" if len(current) > 1 else "%s") % ('|'.join("%s%s" % (re.escape(_), process(current[_])) for _ in sorted(current))).replace('|'.join(str(_) for _ in xrange(10)), r"\d") @@ -224,13 +240,13 @@ def check_whitelisted(trail): def load_trails(quiet=False): if not quiet: - print "[i] loading trails..." + print("[i] loading trails...") retval = TrailsDict() - if os.path.isfile(TRAILS_FILE): + if os.path.isfile(config.TRAILS_FILE): try: - with open(TRAILS_FILE, "rb") as f: + with open(config.TRAILS_FILE, "r") as f: reader = csv.reader(f, delimiter=',', quotechar='\"') for row in reader: if row and len(row) == 3: @@ -238,8 +254,8 @@ def load_trails(quiet=False): if not check_whitelisted(trail): retval[trail] = (info, reference) - except Exception, ex: - exit("[!] something went wrong during trails file read '%s' ('%s')" % (TRAILS_FILE, ex)) + except Exception as ex: + sys.exit("[!] something went wrong during trails file read '%s' ('%s')" % (config.TRAILS_FILE, ex)) if not quiet: _ = len(retval) @@ -247,6 +263,62 @@ def load_trails(quiet=False): _ = '{0:,}'.format(_) except: pass - print "[i] %s trails loaded" % _ + print("[i] %s trails loaded" % _) return retval + +def get_text(value): + retval = value + + if six.PY2: + try: + retval = str(retval) + except: + pass + else: + if isinstance(value, six.binary_type): + retval = value.decode(UNICODE_ENCODING, errors="replace") + + return retval + +def get_ex_message(ex): + retval = None + + if getattr(ex, "message", None): + retval = ex.message + elif getattr(ex, "msg", None): + retval = ex.msg + elif getattr(ex, "args", None): + for candidate in ex.args[::-1]: + if isinstance(candidate, six.string_types): + retval = candidate + break + + if retval is None: + retval = str(ex) + + return retval + +def is_local(address): + return re.search(r"\A(127|10|172\.[13][0-9]|192\.168)\.", address or "") is not None + +def patch_parser(parser): + # Dirty hack to display longer options without breaking into two lines + if hasattr(parser, "formatter"): + def _(self, *args): + retval = parser.formatter._format_option_strings(*args) + if len(retval) > MAX_HELP_OPTION_LENGTH: + retval = ("%%.%ds.." % (MAX_HELP_OPTION_LENGTH - parser.formatter.indent_increment)) % retval + return retval.capitalize() + + parser.formatter._format_option_strings = parser.formatter.format_option_strings + parser.formatter.format_option_strings = type(parser.formatter.format_option_strings)(_, parser) + else: + def _format_action_invocation(self, action): + retval = self.__format_action_invocation(action) + if len(retval) > MAX_HELP_OPTION_LENGTH: + retval = ("%%.%ds.." % (MAX_HELP_OPTION_LENGTH - self._indent_increment)) % retval + return retval.capitalize() + + parser.formatter_class.__format_action_invocation = parser.formatter_class._format_action_invocation + parser.formatter_class._format_action_invocation = _format_action_invocation diff --git a/core/compat.py b/core/compat.py new file mode 100644 index 00000000000..b72dc15b596 --- /dev/null +++ b/core/compat.py @@ -0,0 +1,13 @@ +#!/usr/bin/env python + +""" +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) +See the file 'LICENSE' for copying permission +""" + +import sys + +if sys.version_info >= (3, 0): + xrange = range +else: + xrange = xrange diff --git a/core/datatype.py b/core/datatype.py new file mode 100644 index 00000000000..76145c20e3b --- /dev/null +++ b/core/datatype.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python + +from thirdparty.odict import OrderedDict + +# Reference: https://www.kunxi.org/2014/05/lru-cache-in-python +class LRUDict(object): + """ + This class defines the LRU dictionary + + >>> foo = LRUDict(capacity=2) + >>> foo["first"] = 1 + >>> foo["second"] = 2 + >>> foo["third"] = 3 + >>> "first" in foo + False + >>> "third" in foo + True + """ + + def __init__(self, capacity): + self.capacity = capacity + self.cache = OrderedDict() + + def __len__(self): + return len(self.cache) + + def __contains__(self, key): + return key in self.cache + + def __getitem__(self, key): + try: + value = self.cache.pop(key) + self.cache[key] = value + except: + value = None + + return value + + def get(self, key): + return self.__getitem__(key) + + def __setitem__(self, key, value): + try: + self.cache.pop(key) + except KeyError: + if len(self.cache) >= self.capacity: + self.cache.popitem(last=False) + self.cache[key] = value + + def set(self, key, value): + self.__setitem__(key, value) + + def keys(self): + return self.cache.keys() diff --git a/core/enums.py b/core/enums.py index 4ce577d7ea9..13acdec50bd 100644 --- a/core/enums.py +++ b/core/enums.py @@ -1,22 +1,34 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ +import sys + +from thirdparty import six + class _(type): - def __getattr__(self, attr): - return attr + def __getattr__(self, attr): + return attr +@six.add_metaclass(_) class TRAIL(object): - __metaclass__ = _ + pass -class BLOCK_MARKER: - NOP = chr(0x00) - READ = chr(0x01) - WRITE = chr(0x02) - END = chr(0xFF) +if sys.version_info >= (3, 0): + class BLOCK_MARKER: + NOP = 0x00 + READ = 0x01 + WRITE = 0x02 + END = 0xff +else: + class BLOCK_MARKER: + NOP = b'\x00' + READ = b'\x01' + WRITE = b'\x02' + END = b'\xff' class PROTO: TCP = "TCP" @@ -35,6 +47,7 @@ class HTTP_HEADER: CONTENT_LENGTH = "Content-Length" CONTENT_RANGE = "Content-Range" CONTENT_TYPE = "Content-Type" + CONTENT_SECURITY_POLICY = "Content-Security-Policy" COOKIE = "Cookie" EXPIRES = "Expires" HOST = "Host" @@ -53,3 +66,73 @@ class HTTP_HEADER: USER_AGENT = "User-Agent" VIA = "Via" X_POWERED_BY = "X-Powered-By" + +class CACHE_TYPE: + DOMAIN = 0 + USER_AGENT = 1 + PATH = 2 + POST_DATA = 3 + DOMAIN_WHITELISTED = 4 + LOCAL_PREFIX = 4 + +class COLOR: + BLUE = "\033[34m" + BOLD_MAGENTA = "\033[35;1m" + BOLD_GREEN = "\033[32;1m" + BOLD_LIGHT_MAGENTA = "\033[95;1m" + LIGHT_GRAY = "\033[37m" + BOLD_RED = "\033[31;1m" + BOLD_LIGHT_GRAY = "\033[37;1m" + YELLOW = "\033[33m" + DARK_GRAY = "\033[90m" + BOLD_CYAN = "\033[36;1m" + LIGHT_RED = "\033[91m" + CYAN = "\033[36m" + MAGENTA = "\033[35m" + LIGHT_MAGENTA = "\033[95m" + LIGHT_GREEN = "\033[92m" + RESET = "\033[0m" + BOLD_DARK_GRAY = "\033[90;1m" + BOLD_LIGHT_YELLOW = "\033[93;1m" + BOLD_LIGHT_RED = "\033[91;1m" + BOLD_LIGHT_GREEN = "\033[92;1m" + LIGHT_YELLOW = "\033[93m" + BOLD_LIGHT_BLUE = "\033[94;1m" + BOLD_LIGHT_CYAN = "\033[96;1m" + LIGHT_BLUE = "\033[94m" + BOLD_WHITE = "\033[97;1m" + LIGHT_CYAN = "\033[96m" + BLACK = "\033[30m" + BOLD_YELLOW = "\033[33;1m" + BOLD_BLUE = "\033[34;1m" + GREEN = "\033[32m" + WHITE = "\033[97m" + BOLD_BLACK = "\033[30;1m" + RED = "\033[31m" + UNDERLINE = "\033[4m" + +class BACKGROUND: + BLUE = "\033[44m" + LIGHT_GRAY = "\033[47m" + YELLOW = "\033[43m" + DARK_GRAY = "\033[100m" + LIGHT_RED = "\033[101m" + CYAN = "\033[46m" + MAGENTA = "\033[45m" + LIGHT_MAGENTA = "\033[105m" + LIGHT_GREEN = "\033[102m" + RESET = "\033[0m" + LIGHT_YELLOW = "\033[103m" + LIGHT_BLUE = "\033[104m" + LIGHT_CYAN = "\033[106m" + BLACK = "\033[40m" + GREEN = "\033[42m" + WHITE = "\033[107m" + RED = "\033[41m" + +class SEVERITY: + NONE = "none" + LOW = "low" + MEDIUM = "medium" + HIGH = "high" + CRITICAL = "critical" diff --git a/core/httpd.py b/core/httpd.py index 082696809bc..14e28a15549 100644 --- a/core/httpd.py +++ b/core/httpd.py @@ -1,14 +1,12 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ +from __future__ import print_function -import BaseHTTPServer -import cStringIO import datetime -import httplib import glob import gzip import hashlib @@ -18,13 +16,11 @@ import os import re import socket -import SocketServer import subprocess +import sys import threading import time import traceback -import urllib -import urlparse from core.addr import addr_to_int from core.addr import int_to_addr @@ -33,6 +29,7 @@ from core.common import get_regex from core.common import ipcat_lookup from core.common import worst_asns +from core.compat import xrange from core.enums import HTTP_HEADER from core.settings import config from core.settings import CONTENT_EXTENSIONS_EXCLUSIONS @@ -41,17 +38,23 @@ from core.settings import DISPOSED_NONCES from core.settings import HTML_DIR from core.settings import HTTP_TIME_FORMAT +from core.settings import IS_WIN from core.settings import MAX_NOFILE from core.settings import NAME from core.settings import PING_RESPONSE -from core.settings import SERVER_HEADER from core.settings import SESSION_COOKIE_NAME +from core.settings import SESSION_COOKIE_FLAG_SAMESITE from core.settings import SESSION_EXPIRATION_HOURS from core.settings import SESSION_ID_LENGTH from core.settings import SESSIONS -from core.settings import TRAILS_FILE from core.settings import UNAUTHORIZED_SLEEP_TIME +from core.settings import UNICODE_ENCODING from core.settings import VERSION +from thirdparty import six +from thirdparty.six.moves import BaseHTTPServer as _BaseHTTPServer +from thirdparty.six.moves import http_client as _http_client +from thirdparty.six.moves import socketserver as _socketserver +from thirdparty.six.moves import urllib as _urllib try: # Reference: https://bugs.python.org/issue7980 @@ -66,43 +69,61 @@ except: pass +_fail2ban_cache = None +_fail2ban_key = None +_blacklist_cache = None +_blacklist_key = None + + def start_httpd(address=None, port=None, join=False, pem=None): """ Starts HTTP server """ - class ThreadingServer(SocketServer.ThreadingMixIn, BaseHTTPServer.HTTPServer): + class ThreadingServer(_socketserver.ThreadingMixIn, _BaseHTTPServer.HTTPServer): def server_bind(self): self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) - BaseHTTPServer.HTTPServer.server_bind(self) + _BaseHTTPServer.HTTPServer.server_bind(self) def finish_request(self, *args, **kwargs): try: - BaseHTTPServer.HTTPServer.finish_request(self, *args, **kwargs) + _BaseHTTPServer.HTTPServer.finish_request(self, *args, **kwargs) except: if config.SHOW_DEBUG: traceback.print_exc() class SSLThreadingServer(ThreadingServer): def __init__(self, server_address, pem, HandlerClass): - import OpenSSL # python-openssl + if six.PY2: + import OpenSSL # pyopenssl + + ThreadingServer.__init__(self, server_address, HandlerClass) + for method in ("TLSv1_2_METHOD", "TLSv1_1_METHOD", "TLSv1_METHOD", "TLS_METHOD", "SSLv23_METHOD", "SSLv2_METHOD"): + if hasattr(OpenSSL.SSL, method): + ctx = OpenSSL.SSL.Context(getattr(OpenSSL.SSL, method)) + break + ctx.use_privatekey_file(pem) + ctx.use_certificate_file(pem) + self.socket = OpenSSL.SSL.Connection(ctx, socket.socket(self.address_family, self.socket_type)) + self.server_bind() + self.server_activate() + else: + import ssl - ThreadingServer.__init__(self, server_address, HandlerClass) - ctx = OpenSSL.SSL.Context(OpenSSL.SSL.TLSv1_METHOD) - ctx.use_privatekey_file(pem) - ctx.use_certificate_file(pem) - self.socket = OpenSSL.SSL.Connection(ctx, socket.socket(self.address_family, self.socket_type)) - self.server_bind() - self.server_activate() + ThreadingServer.__init__(self, server_address, ReqHandler) + ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + ctx.load_cert_chain(pem, pem) + self.socket = ctx.wrap_socket(socket.socket(self.address_family, self.socket_type), server_side=True) + self.server_bind() + self.server_activate() def shutdown_request(self, request): try: request.shutdown() except: - if config.SHOW_DEBUG: - traceback.print_exc() + pass - class ReqHandler(BaseHTTPServer.BaseHTTPRequestHandler): + class ReqHandler(_BaseHTTPServer.BaseHTTPRequestHandler): def do_GET(self): path, query = self.path.split('?', 1) if '?' in self.path else (self.path, "") params = {} @@ -110,10 +131,10 @@ def do_GET(self): skip = False if hasattr(self, "data"): - params.update(urlparse.parse_qs(self.data)) + params.update(_urllib.parse.parse_qs(self.data)) if query: - params.update(urlparse.parse_qs(query)) + params.update(_urllib.parse.parse_qs(query)) for key in params: if params[key]: @@ -125,8 +146,11 @@ def do_GET(self): path = path.strip('/') extension = os.path.splitext(path)[-1].lower() - if hasattr(self, "_%s" % path): - content = getattr(self, "_%s" % path)(params) + splitpath = path.split('/', 1) + if hasattr(self, "_%s" % splitpath[0]): + if len(splitpath) > 1: + params["subpath"] = splitpath[1] + content = getattr(self, "_%s" % splitpath[0])(params) else: path = path.replace('/', os.path.sep) @@ -135,6 +159,10 @@ def do_GET(self): if not os.path.isfile(path) and os.path.isfile("%s.html" % path): path = "%s.html" % path + if any((config.IP_ALIASES,)) and self.path.split('?')[0] == "/js/main.js": + content = open(path, 'r').read() + content = re.sub(r"\bvar IP_ALIASES =.+", "var IP_ALIASES = {%s};" % ", ".join('"%s": "%s"' % (_.split(':', 1)[0].strip(), _.split(':', 1)[-1].strip()) for _ in config.IP_ALIASES), content) + if ".." not in os.path.relpath(path, HTML_DIR) and os.path.isfile(path) and (extension not in DISABLED_CONTENT_EXTENSIONS or os.path.split(path)[-1] in CONTENT_EXTENSIONS_EXCLUSIONS): mtime = time.gmtime(os.path.getmtime(path)) if_modified_since = self.headers.get(HTTP_HEADER.IF_MODIFIED_SINCE) @@ -142,17 +170,24 @@ def do_GET(self): if if_modified_since and extension not in (".htm", ".html"): if_modified_since = [_ for _ in if_modified_since.split(';') if _.upper().endswith("GMT")][0] if time.mktime(mtime) <= time.mktime(time.strptime(if_modified_since, HTTP_TIME_FORMAT)): - self.send_response(httplib.NOT_MODIFIED) + self.send_response(_http_client.NOT_MODIFIED) self.send_header(HTTP_HEADER.CONNECTION, "close") skip = True if not skip: - content = open(path, "rb").read() + content = content or open(path, "rb").read() last_modified = time.strftime(HTTP_TIME_FORMAT, mtime) - self.send_response(httplib.OK) + self.send_response(_http_client.OK) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, mimetypes.guess_type(path)[0] or "application/octet-stream") self.send_header(HTTP_HEADER.LAST_MODIFIED, last_modified) + + # For CSP policy directives see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/ + self.send_header(HTTP_HEADER.CONTENT_SECURITY_POLICY, "default-src 'self'; style-src 'self' 'unsafe-inline'; img-src * blob:; script-src 'self' 'unsafe-eval' https://stat.ripe.net; frame-src *; object-src 'none'; block-all-mixed-content;") + + if os.path.basename(path) == "index.html": + content = re.sub(b'\s*]+src="js/demo.js">', b'', content) + if extension not in (".htm", ".html"): self.send_header(HTTP_HEADER.EXPIRES, "Sun, 17-Jan-2038 19:14:07 GMT") # Reference: http://blog.httpwatch.com/2007/12/10/two-simple-rules-for-http-caching/ self.send_header(HTTP_HEADER.CACHE_CONTROL, "max-age=3600, must-revalidate") # Reference: http://stackoverflow.com/a/5084555 @@ -160,20 +195,23 @@ def do_GET(self): self.send_header(HTTP_HEADER.CACHE_CONTROL, "no-cache") else: - self.send_response(httplib.NOT_FOUND) + self.send_response(_http_client.NOT_FOUND) self.send_header(HTTP_HEADER.CONNECTION, "close") content = '404 Not Found

Not Found

The requested URL %s was not found on this server.

' % self.path.split('?')[0] if content is not None: - for match in re.finditer(r"<\!(\w+)\!>", content): - name = match.group(1) + if isinstance(content, six.text_type): + content = content.encode(UNICODE_ENCODING) + + for match in re.finditer(b"<\\!(\\w+)\\!>", content): + name = match.group(1).decode(UNICODE_ENCODING) _ = getattr(self, "_%s" % name.lower(), None) if _: - content = self._format(content, **{ name: _() }) + content = self._format(content, **{name: _()}) - if "gzip" in self.headers.getheader(HTTP_HEADER.ACCEPT_ENCODING, ""): + if "gzip" in self.headers.get(HTTP_HEADER.ACCEPT_ENCODING, ""): self.send_header(HTTP_HEADER.CONTENT_ENCODING, "gzip") - _ = cStringIO.StringIO() + _ = six.BytesIO() compress = gzip.GzipFile("", "w+b", 9, _) compress._stream = _ compress.write(content) @@ -185,16 +223,18 @@ def do_GET(self): self.end_headers() - if content: - self.wfile.write(content) + try: + if content: + self.wfile.write(content) - self.wfile.flush() - self.wfile.close() + self.wfile.flush() + except: + pass def do_POST(self): - length = self.headers.getheader(HTTP_HEADER.CONTENT_LENGTH) - data = self.rfile.read(int(length)) - data = urllib.unquote_plus(data) + length = self.headers.get(HTTP_HEADER.CONTENT_LENGTH) + data = self.rfile.read(int(length)).decode(UNICODE_ENCODING) + data = _urllib.parse.unquote_plus(data) self.data = data self.do_GET() @@ -214,6 +254,9 @@ def get_session(self): else: del SESSIONS[session] + if retval is None and not config.USERS: + retval = AttribDict({"username": "?"}) + return retval def delete_session(self): @@ -227,11 +270,11 @@ def delete_session(self): del SESSIONS[session] def version_string(self): - return SERVER_HEADER + return "%s/%s" % (NAME, self._version()) def end_headers(self): if not hasattr(self, "_headers_ended"): - BaseHTTPServer.BaseHTTPRequestHandler.end_headers(self) + _BaseHTTPServer.BaseHTTPRequestHandler.end_headers(self) self._headers_ended = True def log_message(self, format, *args): @@ -239,18 +282,41 @@ def log_message(self, format, *args): def finish(self): try: - BaseHTTPServer.BaseHTTPRequestHandler.finish(self) + _BaseHTTPServer.BaseHTTPRequestHandler.finish(self) except: if config.SHOW_DEBUG: traceback.print_exc() def _version(self): - return VERSION + version = VERSION + + try: + for line in open(os.path.join(os.path.dirname(__file__), "settings.py"), 'r'): + match = re.search(r'VERSION = "([^"]*)', line) + if match: + version = match.group(1) + break + except: + pass + + return version + + def _statics(self): + latest = max(glob.glob(os.path.join(os.path.dirname(__file__), "..", "trails", "static", "malware", "*.txt")), key=os.path.getmtime) + return "/%s" % datetime.datetime.fromtimestamp(os.path.getmtime(latest)).strftime(DATE_FORMAT) + + def _logo(self): + if config.HEADER_LOGO: + retval = config.HEADER_LOGO + else: + retval = 'altrail' + + return retval def _format(self, content, **params): if content: for key, value in params.items(): - content = content.replace("" % key, value) + content = content.replace(b"" % key.encode(UNICODE_ENCODING), value.encode(UNICODE_ENCODING)) return content @@ -263,9 +329,15 @@ def _login(self, params): for entry in (config.USERS or []): entry = re.sub(r"\s", "", entry) username, stored_hash, uid, netfilter = entry.split(':') + + try: + uid = int(uid) + except ValueError: + uid = None + if username == params.get("username"): try: - if params.get("hash") == hashlib.sha256(stored_hash.strip() + params.get("nonce")).hexdigest(): + if params.get("hash") == hashlib.sha256((stored_hash.strip() + params.get("nonce")).encode(UNICODE_ENCODING)).hexdigest(): valid = True break except: @@ -273,14 +345,21 @@ def _login(self, params): traceback.print_exc() if valid: - session_id = os.urandom(SESSION_ID_LENGTH).encode("hex") + _ = os.urandom(SESSION_ID_LENGTH) + session_id = _.hex() if hasattr(_, "hex") else _.encode("hex") expiration = time.time() + 3600 * SESSION_EXPIRATION_HOURS - self.send_response(httplib.OK) + self.send_response(_http_client.OK) self.send_header(HTTP_HEADER.CONNECTION, "close") - self.send_header(HTTP_HEADER.SET_COOKIE, "%s=%s; expires=%s; path=/; HttpOnly" % (SESSION_COOKIE_NAME, session_id, time.strftime(HTTP_TIME_FORMAT, time.gmtime(expiration)))) - if netfilter in ("", "0.0.0.0/0"): + cookie = "%s=%s; expires=%s; path=/; HttpOnly" % (SESSION_COOKIE_NAME, session_id, time.strftime(HTTP_TIME_FORMAT, time.gmtime(expiration))) + if config.USE_SSL: + cookie += "; Secure" + if SESSION_COOKIE_FLAG_SAMESITE: + cookie += "; SameSite=strict" + self.send_header(HTTP_HEADER.SET_COOKIE, cookie) + + if netfilter in ("", '*', "::", "0.0.0.0/0"): netfilters = None else: addresses = set() @@ -312,18 +391,18 @@ def _login(self, params): if addresses: netfilters.add(get_regex(addresses)) - SESSIONS[session_id] = AttribDict({"username": username, "uid": uid, "netfilters": netfilters, "expiration": expiration, "client_ip": self.client_address[0]}) + SESSIONS[session_id] = AttribDict({"username": username, "uid": uid, "netfilters": netfilters, "mask_custom": config.ENABLE_MASK_CUSTOM and uid >= 1000, "expiration": expiration, "client_ip": self.client_address[0]}) else: time.sleep(UNAUTHORIZED_SLEEP_TIME) - self.send_response(httplib.UNAUTHORIZED) + self.send_response(_http_client.UNAUTHORIZED) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") content = "Login %s" % ("success" if valid else "failed") - if not subprocess.mswindows: + if not IS_WIN: try: - subprocess.check_output("logger -p auth.info -t \"%s[%d]\" \"%s password for %s from %s port %s\"" % (NAME.lower(), os.getpid(), "Accepted" if valid else "Failed", params.get("username"), self.client_address[0], self.client_address[1]), stderr=subprocess.STDOUT, shell=True) + subprocess.check_output(["logger", "-p", "auth.info", "-t", "%s[%d]" % (NAME.lower(), os.getpid()), "%s password for %s from %s port %s" % ("Accepted" if valid else "Failed", params.get("username"), self.client_address[0], self.client_address[1])], stderr=subprocess.STDOUT, shell=False) except Exception: if config.SHOW_DEBUG: traceback.print_exc() @@ -332,7 +411,7 @@ def _login(self, params): def _logout(self, params): self.delete_session() - self.send_response(httplib.FOUND) + self.send_response(_http_client.FOUND) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.LOCATION, "/") @@ -340,7 +419,7 @@ def _whoami(self, params): session = self.get_session() username = session.username if session else "" - self.send_response(httplib.OK) + self.send_response(_http_client.OK) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") @@ -350,11 +429,11 @@ def _check_ip(self, params): session = self.get_session() if session is None: - self.send_response(httplib.UNAUTHORIZED) + self.send_response(_http_client.UNAUTHORIZED) self.send_header(HTTP_HEADER.CONNECTION, "close") return None - self.send_response(httplib.OK) + self.send_response(_http_client.OK) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") @@ -371,24 +450,132 @@ def _check_ip(self, params): traceback.print_exc() def _trails(self, params): - self.send_response(httplib.OK) + self.send_response(_http_client.OK) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") - return open(TRAILS_FILE, "rb").read() + return open(config.TRAILS_FILE, "rb").read() def _ping(self, params): - self.send_response(httplib.OK) + self.send_response(_http_client.OK) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") return PING_RESPONSE + def _fail2ban(self, params): + global _fail2ban_cache + global _fail2ban_key + + self.send_response(_http_client.OK) + self.send_header(HTTP_HEADER.CONNECTION, "close") + self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") + + content = "" + key = int(time.time()) >> 3 + + if config.FAIL2BAN_REGEX: + try: + re.compile(config.FAIL2BAN_REGEX) + except re.error: + content = "invalid regular expression used in option FAIL2BAN_REGEX" + else: + if key == _fail2ban_key: + content = _fail2ban_cache + else: + result = set() + _ = os.path.join(config.LOG_DIR, "%s.log" % datetime.datetime.now().strftime("%Y-%m-%d")) + if os.path.isfile(_): + for line in open(_, "r"): + if re.search(config.FAIL2BAN_REGEX, line, re.I): + result.add(line.split()[3]) + + content = "\n".join(result) + + _fail2ban_cache = content + _fail2ban_key = key + else: + content = "configuration option FAIL2BAN_REGEX not set" + + return content + + def _blacklist(self, params): + global _blacklist_cache + global _blacklist_key + + self.send_response(_http_client.OK) + self.send_header(HTTP_HEADER.CONNECTION, "close") + self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") + + bl_name = "" + if 'subpath' in params: + bl_name = "_%s" % params['subpath'].split('/')[0].upper() + + content = "" + key = int(time.time()) >> 3 + + if "BLACKLIST%s" % bl_name in config: + try: + blacklist = [] + for bl in config["BLACKLIST%s" % bl_name]: + rules = [] + for e in bl.split(' and '): + f, n, p = e.strip().split(' ', 2) + regexp = [ + [ + '', + '', + '', + 'src_ip', + 'src_port', + 'dst_ip', + 'dst_port', + 'protocol', + 'type', + 'trail', + 'filter' + ].index(f), + (n[0] == '!'), + re.compile(p, re.I) + ] + rules.append(regexp) + blacklist.append(rules) + except e: + content = "invalid rule in option BLACKLIST%s" % bl_name + else: + if key == _blacklist_key: + content = _blacklist_cache + else: + result = set() + _ = os.path.join(config.LOG_DIR, "%s.log" % datetime.datetime.now().strftime("%Y-%m-%d")) + if os.path.isfile(_): + for line in open(_, "r"): + line = line.split(' ', 10) + for bl in blacklist: + failed = False + for f, n, r in bl: + if not ( + (r.search(line[f]) is not None) ^ n + ): + failed = True + break + if not failed: + result.add(line[3]) + break + + content = "\n".join(result) + + _blacklist_cache = content + _blacklist_key = key + else: + content = "configuration option BLACKLIST%s not set" % bl_name + return content + def _events(self, params): session = self.get_session() if session is None: - self.send_response(httplib.UNAUTHORIZED) + self.send_response(_http_client.UNAUTHORIZED) self.send_header(HTTP_HEADER.CONNECTION, "close") return None @@ -407,7 +594,7 @@ def _events(self, params): range_handle = open(event_log_path, "rb") log_exists = True except ValueError: - print "[!] invalid date format in request" + print("[!] invalid date format in request") log_exists = False else: logs_data = "" @@ -426,7 +613,7 @@ def _events(self, params): range_handle = io.BytesIO(logs_data) log_exists = True except ValueError: - print "[!] invalid date format in request" + print("[!] invalid date format in request") log_exists = False if log_exists: @@ -445,35 +632,36 @@ def _events(self, params): if start == 0 or not session.range_handle: session.range_handle = range_handle - if session.netfilters is None: + if session.netfilters is None and not session.mask_custom: session.range_handle.seek(start) - self.send_response(httplib.PARTIAL_CONTENT) + self.send_response(_http_client.PARTIAL_CONTENT) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") self.send_header(HTTP_HEADER.CONTENT_RANGE, "bytes %d-%d/%d" % (start, end, total)) content = session.range_handle.read(size) else: - self.send_response(httplib.OK) + self.send_response(_http_client.OK) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") - buffer, addresses, netmasks, regex = cStringIO.StringIO(), set(), [], "" - for netfilter in session.netfilters: + buffer, addresses, netmasks, regex = io.StringIO(), set(), [], "" + for netfilter in session.netfilters or []: if not netfilter: continue if '/' in netfilter: netmasks.append(netfilter) elif re.search(r"\A[\d.]+\Z", netfilter): addresses.add(netfilter) - elif '\.' in netfilter: + elif "\\." in netfilter: regex = r"\b(%s)\b" % netfilter else: - print "[!] invalid network filter '%s'" % netfilter + print("[!] invalid network filter '%s'" % netfilter) return for line in session.range_handle: - display = False + display = session.netfilters is None ip = None + line = line.decode(UNICODE_ENCODING, "ignore") if regex: match = re.search(regex, line) @@ -499,6 +687,9 @@ def _events(self, params): display = True break + if session.mask_custom and "(custom)" in line: + line = re.sub(r'("[^"]+"|[^ ]+) \(custom\)', "- (custom)", line) + if display: if ",%s" % ip in line or "%s," % ip in line: line = re.sub(r" ([\d.,]+,)?%s(,[\d.,]+)? " % re.escape(ip), " %s " % ip, line) @@ -515,7 +706,7 @@ def _events(self, params): session.range_handle = None if size == -1: - self.send_response(httplib.OK) + self.send_response(_http_client.OK) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, "text/plain") self.end_headers() @@ -529,7 +720,7 @@ def _events(self, params): self.wfile.write(data) else: - self.send_response(httplib.OK) # instead of httplib.NO_CONTENT (compatibility reasons) + self.send_response(_http_client.OK) # instead of _http_client.NO_CONTENT (compatibility reasons) self.send_header(HTTP_HEADER.CONNECTION, "close") if self.headers.get(HTTP_HEADER.RANGE): self.send_header(HTTP_HEADER.CONTENT_RANGE, "bytes 0-0/0") @@ -542,11 +733,11 @@ def _counts(self, params): session = self.get_session() if session is None: - self.send_response(httplib.UNAUTHORIZED) + self.send_response(_http_client.UNAUTHORIZED) self.send_header(HTTP_HEADER.CONNECTION, "close") return None - self.send_response(httplib.OK) + self.send_response(_http_client.OK) self.send_header(HTTP_HEADER.CONNECTION, "close") self.send_header(HTTP_HEADER.CONTENT_TYPE, "application/json") @@ -581,10 +772,10 @@ def _counts(self, params): with open(filepath, "rb") as f: content = f.read(io.DEFAULT_BUFFER_SIZE) if size >= io.DEFAULT_BUFFER_SIZE: - total = 1.0 * content.count('\n') * size / io.DEFAULT_BUFFER_SIZE - counts[timestamp] = int(round(total / 100) * 100) + total = 1.0 * (1 + content.count(b'\n')) * size / io.DEFAULT_BUFFER_SIZE + counts[timestamp] = int(round(total / 100.0) * 100) else: - counts[timestamp] = content.count('\n') + counts[timestamp] = content.count(b'\n') return json.dumps(counts) @@ -594,24 +785,38 @@ def setup(self): self.rfile = socket._fileobject(self.request, "rb", self.rbufsize) self.wfile = socket._fileobject(self.request, "wb", self.wbufsize) + # IPv6 support + if ':' in (address or ""): + address = address.strip("[]") + + _BaseHTTPServer.HTTPServer.address_family = socket.AF_INET6 + + # Reference: https://github.com/squeaky-pl/zenchmarks/blob/master/vendor/twisted/internet/tcp.py + _AI_NUMERICSERV = getattr(socket, "AI_NUMERICSERV", 0) + _NUMERIC_ONLY = socket.AI_NUMERICHOST | _AI_NUMERICSERV + + _address = socket.getaddrinfo(address, int(port) if str(port or "").isdigit() else 0, 0, 0, 0, _NUMERIC_ONLY)[0][4] + else: + _address = (address or '', int(port) if str(port or "").isdigit() else 0) + try: if pem: - server = SSLThreadingServer((address or '', int(port) if str(port or "").isdigit() else 0), pem, SSLReqHandler) + server = SSLThreadingServer(_address, pem, SSLReqHandler) else: - server = ThreadingServer((address or '', int(port) if str(port or "").isdigit() else 0), ReqHandler) + server = ThreadingServer(_address, ReqHandler) except Exception as ex: if "Address already in use" in str(ex): - exit("[!] another instance already running") + sys.exit("[!] another instance already running") elif "Name or service not known" in str(ex): - exit("[!] invalid configuration value for 'HTTP_ADDRESS' ('%s')" % config.HTTP_ADDRESS) + sys.exit("[!] invalid configuration value for 'HTTP_ADDRESS' ('%s')" % config.HTTP_ADDRESS) elif "Cannot assign requested address" in str(ex): - exit("[!] can't use configuration value for 'HTTP_ADDRESS' ('%s')" % config.HTTP_ADDRESS) + sys.exit("[!] can't use configuration value for 'HTTP_ADDRESS' ('%s')" % config.HTTP_ADDRESS) else: raise - print "[i] starting HTTP%s server at 'http%s://%s:%d/'" % ('S' if pem else "", 's' if pem else "", server.server_address[0], server.server_address[1]) + print("[i] starting HTTP%s server at http%s://%s:%d/" % ('S' if pem else "", 's' if pem else "", server.server_address[0], server.server_address[1])) - print "[o] running..." + print("[^] running...") if join: server.serve_forever() diff --git a/core/ignore.py b/core/ignore.py new file mode 100755 index 00000000000..72c345c3e04 --- /dev/null +++ b/core/ignore.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python + +""" +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) +See the file 'LICENSE' for copying permission +""" +from __future__ import print_function + +# simple ignore rule mechanism configured by file 'misc/ignore_event.txt' and/or user defined `USER_IGNORELIST` + +import re + +from core.settings import config +from core.settings import IGNORE_EVENTS + +def ignore_event(event_tuple): + retval = False + _, _, src_ip, src_port, dst_ip, dst_port, _, _, _, _, _ = event_tuple + + if config.IGNORE_EVENTS_REGEX and re.search(config.IGNORE_EVENTS_REGEX, repr(event_tuple), re.I): + retval = True + + for ignore_src_ip, ignore_src_port, ignore_dst_ip, ignore_dst_port in IGNORE_EVENTS: + if ignore_src_ip != '*' and ignore_src_ip != src_ip: + continue + if ignore_src_port != '*' and ignore_src_port != str(src_port): + continue + if ignore_dst_ip != '*' and ignore_dst_ip != dst_ip: + continue + if ignore_dst_port != '*' and ignore_dst_port != str(dst_port): + continue + retval = True + break + + if retval and config.SHOW_DEBUG: + print("[i] ignore_event src_ip=%s, src_port=%s, dst_ip=%s, dst_port=%s" % (src_ip, src_port, dst_ip, dst_port)) + + return retval diff --git a/core/log.py b/core/log.py index ca41e75d49d..78d0d4167fb 100644 --- a/core/log.py +++ b/core/log.py @@ -1,14 +1,17 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ +from __future__ import print_function +import datetime +import json import os +import re import signal import socket -import SocketServer import sys import threading import time @@ -16,6 +19,7 @@ from core.common import check_whitelisted from core.common import check_sudo +from core.compat import xrange from core.enums import TRAIL from core.settings import CEF_FORMAT from core.settings import config @@ -26,16 +30,23 @@ from core.settings import HOSTNAME from core.settings import NAME from core.settings import TIME_FORMAT -from core.settings import TRAILS_FILE +from core.settings import UNICODE_ENCODING from core.settings import VERSION - +from core.ignore import ignore_event +from thirdparty.odict import OrderedDict +from thirdparty.six.moves import socketserver as _socketserver + +_condensed_events = {} +_condensing_thread = None +_condensing_lock = threading.Lock() +_single_messages = set() _thread_data = threading.local() def create_log_directory(): if not os.path.isdir(config.LOG_DIR): - if check_sudo() is False: - exit("[!] please rerun with sudo/Administrator privileges") - os.makedirs(config.LOG_DIR, 0755) + if not config.DISABLE_CHECK_SUDO and check_sudo() is False: + sys.exit("[!] please rerun with sudo/Administrator privileges") + os.makedirs(config.LOG_DIR, 0o755) print("[i] using '%s' for log storage" % config.LOG_DIR) def get_event_log_handle(sec, flags=os.O_APPEND | os.O_CREAT | os.O_WRONLY, reuse=True): @@ -71,7 +82,7 @@ def get_event_log_handle(sec, flags=os.O_APPEND | os.O_CREAT | os.O_WRONLY, reus def get_error_log_handle(flags=os.O_APPEND | os.O_CREAT | os.O_WRONLY): if not hasattr(_thread_data, "error_log_handle"): - _ = os.path.join(config.LOG_DIR, "error.log") + _ = os.path.join(config.get("LOG_DIR") or os.curdir, "error.log") if not os.path.exists(_): open(_, "w+").close() os.chmod(_, DEFAULT_ERROR_LOG_PERMISSIONS) @@ -83,54 +94,71 @@ def safe_value(value): retval = str(value or '-') if any(_ in retval for _ in (' ', '"')): retval = "\"%s\"" % retval.replace('"', '""') + retval = re.sub(r"[\x0a\x0d]", " ", retval) return retval -def log_event(event_tuple, packet=None, skip_write=False, skip_condensing=False): - try: - sec, usec, src_ip, src_port, dst_ip, dst_port, proto, trail_type, trail, info, reference = event_tuple - if not (any(check_whitelisted(_) for _ in (src_ip, dst_ip)) and trail_type != TRAIL.DNS): # DNS requests/responses can't be whitelisted based on src_ip/dst_ip - if not skip_write: - localtime = "%s.%06d" % (time.strftime(TIME_FORMAT, time.localtime(int(sec))), usec) +def flush_condensed_events(single=False): + while True: + if not single: + time.sleep(CONDENSED_EVENTS_FLUSH_PERIOD) - if not skip_condensing: - if (sec - getattr(_thread_data, "condensed_events_flush_sec", 0)) > CONDENSED_EVENTS_FLUSH_PERIOD: - _thread_data.condensed_events_flush_sec = sec + with _condensing_lock: + for key in _condensed_events: + condensed = False + events = _condensed_events[key] - for key in getattr(_thread_data, "condensed_events", []): - condensed = False - events = _thread_data.condensed_events[key] + first_event = events[0] + condensed_event = [_ for _ in first_event] - first_event = events[0] - condensed_event = [_ for _ in first_event] + for i in xrange(1, len(events)): + current_event = events[i] + for j in xrange(3, 7): # src_port, dst_ip, dst_port, proto + if current_event[j] != condensed_event[j]: + condensed = True + if not isinstance(condensed_event[j], set): + condensed_event[j] = set((condensed_event[j],)) + condensed_event[j].add(current_event[j]) - for i in xrange(1, len(events)): - current_event = events[i] - for j in xrange(3, 7): # src_port, dst_ip, dst_port, proto - if current_event[j] != condensed_event[j]: - condensed = True - if not isinstance(condensed_event[j], set): - condensed_event[j] = set((condensed_event[j],)) - condensed_event[j].add(current_event[j]) + if condensed: + for i in xrange(len(condensed_event)): + if isinstance(condensed_event[i], set): + condensed_event[i] = ','.join(str(_) for _ in sorted(condensed_event[i])) - if condensed: - for i in xrange(len(condensed_event)): - if isinstance(condensed_event[i], set): - condensed_event[i] = ','.join(str(_) for _ in sorted(condensed_event[i])) + log_event(condensed_event, skip_condensing=True) - log_event(condensed_event, skip_condensing=True) + _condensed_events.clear() - _thread_data.condensed_events = {} + if single: + break + +def log_event(event_tuple, packet=None, skip_write=False, skip_condensing=False): + global _condensing_thread + + if _condensing_thread is None: + _condensing_thread = threading.Thread(target=flush_condensed_events) + _condensing_thread.daemon = True + _condensing_thread.start() + + try: + sec, usec, src_ip, src_port, dst_ip, dst_port, proto, trail_type, trail, info, reference = event_tuple + if ignore_event(event_tuple): + return + + if not (any(check_whitelisted(_) for _ in (src_ip, dst_ip)) and trail_type != TRAIL.DNS): # DNS requests/responses can't be whitelisted based on src_ip/dst_ip + if not skip_write: + localtime = "%s.%06d" % (time.strftime(TIME_FORMAT, time.localtime(int(sec))), usec) + if not skip_condensing: if any(_ in info for _ in CONDENSE_ON_INFO_KEYWORDS): - if not hasattr(_thread_data, "condensed_events"): - _thread_data.condensed_events = {} - key = (src_ip, trail) - if key not in _thread_data.condensed_events: - _thread_data.condensed_events[key] = [] - _thread_data.condensed_events[key].append(event_tuple) + with _condensing_lock: + key = (src_ip, trail) + if key not in _condensed_events: + _condensed_events[key] = [] + _condensed_events[key].append(event_tuple) + return - current_bucket = sec / config.PROCESS_COUNT + current_bucket = sec // config.PROCESS_COUNT if getattr(_thread_data, "log_bucket", None) != current_bucket: # log throttling _thread_data.log_bucket = current_bucket _thread_data.log_trails = set() @@ -144,21 +172,49 @@ def log_event(event_tuple, packet=None, skip_write=False, skip_condensing=False) event = "%s %s %s\n" % (safe_value(localtime), safe_value(config.SENSOR_NAME), " ".join(safe_value(_) for _ in event_tuple[2:])) if not config.DISABLE_LOCAL_LOG_STORAGE: handle = get_event_log_handle(sec) - os.write(handle, event) + os.write(handle, event.encode(UNICODE_ENCODING)) if config.LOG_SERVER: - remote_host, remote_port = config.LOG_SERVER.split(':') - s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) - s.sendto("%s %s" % (sec, event), (remote_host, int(remote_port))) - - if config.SYSLOG_SERVER: - extension = "src=%s spt=%s dst=%s dpt=%s trail=%s ref=%s" % (src_ip, src_port, dst_ip, dst_port, trail, reference) - _ = CEF_FORMAT.format(syslog_time=time.strftime("%b %d %H:%M:%S", time.localtime(int(sec))), host=HOSTNAME, device_vendor=NAME, device_product="sensor", device_version=VERSION, signature_id=time.strftime("%Y-%m-%d", time.localtime(os.path.getctime(TRAILS_FILE))), name=info, severity=0, extension=extension) - remote_host, remote_port = config.SYSLOG_SERVER.split(':') - s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) - s.sendto(_, (remote_host, int(remote_port))) - - if config.DISABLE_LOCAL_LOG_STORAGE and not any(config.LOG_SERVER, config.SYSLOG_SERVER) or config.console: + if config.LOG_SERVER.count(':') > 1: + remote_host, remote_port = config.LOG_SERVER.replace('[', '').replace(']', '').rsplit(':', 1) + + # Reference: https://github.com/squeaky-pl/zenchmarks/blob/master/vendor/twisted/internet/tcp.py + _AI_NUMERICSERV = getattr(socket, "AI_NUMERICSERV", 0) + _NUMERIC_ONLY = socket.AI_NUMERICHOST | _AI_NUMERICSERV + + _address = socket.getaddrinfo(remote_host, int(remote_port) if str(remote_port or "").isdigit() else 0, 0, 0, 0, _NUMERIC_ONLY)[0][4] + else: + remote_host, remote_port = config.LOG_SERVER.split(':') + _address = (remote_host, int(remote_port)) + + s = socket.socket(socket.AF_INET if len(_address) == 2 else socket.AF_INET6, socket.SOCK_DGRAM) + s.sendto(("%s %s" % (sec, event)).encode(UNICODE_ENCODING), _address) + + if config.SYSLOG_SERVER or config.LOGSTASH_SERVER: + severity = "medium" + + if config.REMOTE_SEVERITY_REGEX: + match = re.search(config.REMOTE_SEVERITY_REGEX, info) + if match: + for _ in ("low", "medium", "high"): + if match.group(_): + severity = _ + break + + if config.SYSLOG_SERVER: + extension = "src=%s spt=%s dst=%s dpt=%s trail=%s ref=%s" % (src_ip, src_port, dst_ip, dst_port, trail, reference) + _ = CEF_FORMAT.format(syslog_time=time.strftime("%b %d %H:%M:%S", time.localtime(int(sec))), host=HOSTNAME, device_vendor=NAME, device_product="sensor", device_version=VERSION, signature_id=time.strftime("%Y-%m-%d", time.localtime(os.path.getctime(config.TRAILS_FILE))), name=info, severity={"low": 0, "medium": 1, "high": 2}.get(severity), extension=extension) + remote_host, remote_port = config.SYSLOG_SERVER.split(':') + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + s.sendto(_.encode(UNICODE_ENCODING), (remote_host, int(remote_port))) + + if config.LOGSTASH_SERVER: + _ = OrderedDict((("timestamp", sec), ("sensor", HOSTNAME), ("severity", severity), ("src_ip", src_ip), ("src_port", src_port), ("dst_ip", dst_ip), ("dst_port", dst_port), ("proto", proto), ("type", trail_type), ("trail", trail), ("info", info), ("reference", reference))) + s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + remote_host, remote_port = config.LOGSTASH_SERVER.split(':') + s.sendto(json.dumps(_).encode(UNICODE_ENCODING), (remote_host, int(remote_port))) + + if (config.DISABLE_LOCAL_LOG_STORAGE and not any((config.LOG_SERVER, config.SYSLOG_SERVER))) or config.console: sys.stderr.write(event) sys.stderr.flush() @@ -169,23 +225,39 @@ def log_event(event_tuple, packet=None, skip_write=False, skip_condensing=False) if config.SHOW_DEBUG: traceback.print_exc() -def log_error(msg): +def log_error(msg, single=False): + if single: + if msg in _single_messages: + return + else: + _single_messages.add(msg) + try: handle = get_error_log_handle() - os.write(handle, "%s %s\n" % (time.strftime(TIME_FORMAT, time.localtime()), msg)) + os.write(handle, ("%s %s\n" % (time.strftime(TIME_FORMAT, time.localtime()), msg)).encode(UNICODE_ENCODING)) except (OSError, IOError): if config.SHOW_DEBUG: traceback.print_exc() def start_logd(address=None, port=None, join=False): - class ThreadingUDPServer(SocketServer.ThreadingMixIn, SocketServer.UDPServer): + class ThreadingUDPServer(_socketserver.ThreadingMixIn, _socketserver.UDPServer): pass - class UDPHandler(SocketServer.BaseRequestHandler): + class UDPHandler(_socketserver.BaseRequestHandler): def handle(self): try: data, _ = self.request - sec, event = data.split(" ", 1) + + if data[0:1].isdigit(): # Note: regular format with timestamp in front + sec, event = data.split(b' ', 1) + else: # Note: naive format without timestamp in front + event_date = datetime.datetime.strptime(data[1:data.find(b'.')].decode(UNICODE_ENCODING), TIME_FORMAT) + sec = int(time.mktime(event_date.timetuple())) + event = data + + if not event.endswith(b'\n'): + event = b"%s\n" % event + handle = get_event_log_handle(int(sec), reuse=False) os.write(handle, event) os.close(handle) @@ -193,9 +265,23 @@ def handle(self): if config.SHOW_DEBUG: traceback.print_exc() - server = ThreadingUDPServer((address, port), UDPHandler) + # IPv6 support + if ':' in (address or ""): + address = address.strip("[]") + + _socketserver.UDPServer.address_family = socket.AF_INET6 + + # Reference: https://github.com/squeaky-pl/zenchmarks/blob/master/vendor/twisted/internet/tcp.py + _AI_NUMERICSERV = getattr(socket, "AI_NUMERICSERV", 0) + _NUMERIC_ONLY = socket.AI_NUMERICHOST | _AI_NUMERICSERV + + _address = socket.getaddrinfo(address, int(port) if str(port or "").isdigit() else 0, 0, 0, 0, _NUMERIC_ONLY)[0][4] + else: + _address = (address or '', int(port) if str(port or "").isdigit() else 0) + + server = ThreadingUDPServer(_address, UDPHandler) - print "[i] running UDP server at '%s:%d'" % (server.server_address[0], server.server_address[1]) + print("[i] running UDP server at '%s:%d'" % (server.server_address[0], server.server_address[1])) if join: server.serve_forever() diff --git a/core/parallel.py b/core/parallel.py index 075f18e5e86..46825357dd5 100644 --- a/core/parallel.py +++ b/core/parallel.py @@ -1,7 +1,7 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ @@ -18,7 +18,8 @@ from core.settings import REGULAR_SENSOR_SLEEP_TIME from core.settings import SHORT_SENSOR_SLEEP_TIME from core.settings import trails -from core.settings import TRAILS_FILE + +_timer = None def read_block(buffer, i): offset = i * BLOCK_LENGTH % config.CAPTURE_BUFFER @@ -59,7 +60,9 @@ def worker(buffer, n, offset, mod, process_packet): """ def update_timer(): - if (time.time() - os.stat(TRAILS_FILE).st_mtime) >= config.UPDATE_PERIOD: + global _timer + + if (time.time() - os.stat(config.TRAILS_FILE).st_mtime) >= config.UPDATE_PERIOD: _ = None while True: _ = load_trails(True) @@ -69,11 +72,13 @@ def update_timer(): break else: time.sleep(LOAD_TRAILS_RETRY_SLEEP_TIME) - threading.Timer(config.UPDATE_PERIOD, update_timer).start() + + _timer = threading.Timer(config.UPDATE_PERIOD, update_timer) + _timer.start() update_timer() - count = 0L + count = 0 while True: try: if (count % mod) == offset: @@ -86,7 +91,7 @@ def update_timer(): if content is None: break - if len(content) < 12: + elif len(content) < 12: continue sec, usec, ip_offset = struct.unpack("=III", content[:12]) @@ -97,3 +102,6 @@ def update_timer(): except KeyboardInterrupt: break + + if _timer: + _timer.cancel() diff --git a/core/settings.py b/core/settings.py index 9e61d34e9c7..6d1457b5213 100644 --- a/core/settings.py +++ b/core/settings.py @@ -1,37 +1,41 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ +from __future__ import print_function import os import re import socket import stat -import string +import struct import subprocess import sys -import urllib -import urllib2 from core.addr import addr_to_int +from core.addr import expand_range from core.addr import make_mask from core.attribdict import AttribDict +from core.colorized import init_output from core.trailsdict import TrailsDict - -config = AttribDict() -trails = TrailsDict() +from thirdparty.six.moves import urllib as _urllib NAME = "Maltrail" -VERSION = "0.10.261" -SERVER_HEADER = "%s/%s" % (NAME, VERSION) +VERSION = "0.73" +HOMEPAGE = "https://maltrail.github.io" +PLATFORM = os.name +IS_WIN = PLATFORM == "nt" +IS_SENSOR = "sensor" in sys.argv[0] +USER_AGENT = "%s/%s (%s/py%s/x%d)" % (NAME, VERSION, re.sub(r"\d$", "", sys.platform), sys.version.split(' ')[0], struct.calcsize('P') * 8) DATE_FORMAT = "%Y-%m-%d" ROTATING_CHARS = ('\\', '|', '|', '/', '-') TIMEOUT = 30 +UNICODE_ENCODING = "utf8" FRESH_IPCAT_DELTA_DAYS = 10 USERS_DIR = os.path.join(os.path.expanduser("~"), ".%s" % NAME.lower()) -TRAILS_FILE = os.path.join(USERS_DIR, "trails.csv") +DEFAULT_TRAILS_FILE = os.path.join(USERS_DIR, "trails.csv") IPCAT_CSV_FILE = os.path.join(USERS_DIR, "ipcat.csv") IPCAT_SQLITE_FILE = os.path.join(USERS_DIR, "ipcat.sqlite") IPCAT_URL = "https://raw.githubusercontent.com/client9/ipcat/master/datacenters.csv" @@ -42,6 +46,7 @@ HTTP_TIME_FORMAT = "%a, %d %b %Y %H:%M:%S GMT" # Reference: http://stackoverflow.com/a/225106 CEF_FORMAT = "{syslog_time} {host} CEF:0|{device_vendor}|{device_product}|{device_version}|{signature_id}|{name}|{severity}|{extension}" SESSION_COOKIE_NAME = "%s_sessid" % NAME.lower() +SESSION_COOKIE_FLAG_SAMESITE = True SNAP_LEN = 2000 BLOCK_LENGTH = 1 + 2 + 4 + 4 + 4 + SNAP_LEN # primitive mutex + short for packet size + int for sec + int for usec + int for IP offset + max packet size SHORT_SENSOR_SLEEP_TIME = 0.00001 @@ -49,7 +54,6 @@ LOAD_TRAILS_RETRY_SLEEP_TIME = 60 UNAUTHORIZED_SLEEP_TIME = 5 NO_SUCH_NAME_PER_HOUR_THRESHOLD = 20 -CHECK_MEMORY_SIZE = 384 * 1024 * 1024 NO_BLOCK = -1 END_BLOCK = -2 ROOT_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) @@ -58,55 +62,63 @@ PING_RESPONSE = "pong" MAX_NOFILE = 65000 CAPTURE_TIMEOUT = 100 # ms +MAX_HELP_OPTION_LENGTH = 18 CONFIG_FILE = os.path.join(ROOT_DIR, "maltrail.conf") -SYSTEM_LOG_DIR = "/var/log" if not subprocess.mswindows else "C:\\Windows\\Logs" +SYSTEM_LOG_DIR = "/var/log" if not IS_WIN else "C:\\Windows\\Logs" DEFAULT_EVENT_LOG_PERMISSIONS = stat.S_IREAD | stat.S_IWRITE | stat.S_IRGRP | stat.S_IROTH DEFAULT_ERROR_LOG_PERMISSIONS = stat.S_IREAD | stat.S_IWRITE | stat.S_IRGRP | stat.S_IWGRP | stat.S_IROTH | stat.S_IWOTH HOSTNAME = socket.gethostname() PROXIES = {} DISABLED_CONTENT_EXTENSIONS = (".py", ".pyc", ".md", ".txt", ".bak", ".conf", ".zip", "~") CONTENT_EXTENSIONS_EXCLUSIONS = ("robots.txt",) -CONDENSE_ON_INFO_KEYWORDS = ("attacker", "reputation", "scanner", "user agent", "tor exit", "port scanning") +CONDENSE_ON_INFO_KEYWORDS = ("attacker", "reputation", "scanner", "user agent", "tor exit", "port scanning", "potential infection") CONDENSED_EVENTS_FLUSH_PERIOD = 10 LOW_PRIORITY_INFO_KEYWORDS = ("reputation", "attacker", "spammer", "abuser", "malicious", "dnspod", "nicru", "crawler", "compromised", "bad history") HIGH_PRIORITY_INFO_KEYWORDS = ("mass scanner", "ipinfo") -HIGH_PRIORITY_REFERENCES = ("bambenekconsulting.com", "(static)", "(custom)") +HIGH_PRIORITY_REFERENCES = ("bambenekconsulting.com", "github.com/stamparm/blackbook", "(static)", "(custom)") CONSONANTS = "bcdfghjklmnpqrstvwxyz" BAD_TRAIL_PREFIXES = ("127.", "192.168.", "localhost") -LOCALHOST_IP = { 4: "127.0.0.1", 6: "::1" } -IGNORE_DNS_QUERY_SUFFIXES = (".arpa", ".local", ".guest") -VALID_DNS_CHARS = string.letters + string.digits + '-' + '.' # Reference: http://stackoverflow.com/a/3523068 -SUSPICIOUS_CONTENT_TYPES = ("application/x-sh", "application/x-shellscript", "text/x-sh", "text/x-shellscript") -SUSPICIOUS_DIRECT_DOWNLOAD_EXTENSIONS = set((".apk", ".exe", ".scr")) +LOCALHOST_IP = {4: "127.0.0.1", 6: "::1"} +POTENTIAL_INFECTION_PORTS = (135, 139, 445, 1433, 3389, 6379, 6892, 6893, 6901) +IGNORE_DNS_QUERY_SUFFIXES = set(("arpa", "local", "guest", "intranet", "int", "corp", "home", "lan", "intra", "intran", "workgroup", "localdomain", "url", "alienvault")) +VALID_DNS_NAME_REGEX = r"\A[a-zA-Z0-9.-]*\.[a-zA-Z0-9-]+\Z" # Reference: http://stackoverflow.com/a/3523068 +SUSPICIOUS_CONTENT_TYPES = ("application/vnd.ms-htmlhelp", "application/x-bsh", "application/x-chm", "application/x-ms-shortcut", "application/x-sh", "application/x-shellscript", "application/hta", "text/x-scriptlet", "text/x-sh", "text/x-shellscript") +SUSPICIOUS_DIRECT_DOWNLOAD_EXTENSIONS = set((".apk", ".bin", ".class", ".chm", ".dll", ".egg", ".exe", ".hta", ".hwp", ".lnk", ".ps1", ".scr", ".sct", ".wbk", ".xpi")) WHITELIST_DIRECT_DOWNLOAD_KEYWORDS = ("cgi", "/scripts/", "/_vti_bin/", "/bin/", "/pub/softpaq/", "/bios/", "/pc-axis/") SUSPICIOUS_HTTP_REQUEST_REGEXES = ( ("potential sql injection", r"information_schema|sysdatabases|sysusers|floor\(rand\(|ORDER BY \d+|\bUNION\s+(ALL\s+)?SELECT\b|\b(UPDATEXML|EXTRACTVALUE)\(|\bCASE[^\w]+WHEN.*THEN\b|\bWAITFOR[^\w]+DELAY\b|\bCONVERT\(|VARCHAR\(|\bCOUNT\(\*\)|\b(pg_)?sleep\(|\bSELECT\b.*\bFROM\b.*\b(WHERE|GROUP|ORDER)\b|\bSELECT \w+ FROM \w+|\b(AND|OR|SELECT)\b.*/\*.*\*/|/\*.*\*/.*\b(AND|OR|SELECT)\b|\b(AND|OR)[^\w]+\d+['\") ]?[=><]['\"( ]?\d+|ODBC;DRIVER|\bINTO\s+(OUT|DUMP)FILE"), ("potential xml injection", r"/text\(\)='"), - ("potential php injection", r"<\?php"), + ("potential php injection", r"<\?php|php://input"), ("potential ldap injection", r"\(\|\(\w+=\*"), - ("potential xss injection", r"|\balert\(|(alert|confirm|prompt)\((\d+|document\.|response\.write\(|[^\w]*XSS)|on(mouseover|error|focus)=[^&;\n]+\("), + ("potential xss injection", r"|\balert\(|(alert|confirm|prompt)\((\d+|document\.|response\.write\(|[^\w]*XSS)|on(mouseover|error|focus|transitionend)=[^&;\n]+\("), ("potential xxe injection", r"\[&1|\b(cat|ls) /|nc -l -p \d+|>\s*/dev/null|-d (allow_url_include|safe_mode|auto_prepend_file)"), - ("potential directory traversal", r"(\.{2,}[/\\]+){3,}|/etc/(passwd|shadow|issue|hostname)|[/\\](boot|system|win)\.ini|[/\\]system32\b|%SYSTEMROOT%"), - ("potential web scan", r"(acunetix|injected_by)_wvs_|SomeCustomInjectedHeader|some_inexistent_file_with_long_name|testasp\.vulnweb\.com/t/fit\.txt|www\.acunetix\.tst|\.bxss\.me|thishouldnotexistandhopefullyitwillnot|OWASP%\d+ZAP|chr\(122\)\.chr\(97\)\.chr\(112\)|Vega-Inject|VEGA123|vega\.invalid|PUT-putfile|w00tw00t|muieblackcat") + ("potential remote code execution", r"\$_(REQUEST|GET|POST)\[|xp_cmdshell|shell_exec|exec_code|shell:::\{|oscmd\(|\bping(\.exe)? -[nc] \d+|timeout(\.exe)? /T|tftp -|wget http|curl -O|sh /tmp/|touch /tmp/|cmd\.exe|/bin/(ba)?sh\b|/sbin/launchd\b|2>&1|\b(cat|ls) /|chmod [0-7]{3,4}\b|chmod +x\b|base64 -d|nc -l -p \d+|>\s*/dev/null|-d (allow_url_include|safe_mode|auto_prepend_file)|ms-msdt:|mhtml:ftp:|jndi:(corba|dns|http|iiop|n(d|i)s|ldap[s]?|rmi):?|base64:JHtqbmRp|ipconfig|net (config|view)|nltest|netsh (firewall|wlan)|\$\{IFS\}|getRuntime\(\)\.exec\("), + ("potential directory traversal", r"(\.{2,}[/\\]+){3,}|/etc/(group|passwd|shadow|issue|hostname|hosts|sudoers)|[/\\](boot|system|win)\.ini|[/\\]system32\b|%SYSTEMROOT%"), + ("potential web scan", r"(acunetix|injected_by)_wvs_|SomeCustomInjectedHeader|some_inexistent_file_with_long_name|testasp\.vulnweb\.com/t/fit\.txt|www\.acunetix\.tst|\.bxss\.me|thishouldnotexistandhopefullyitwillnot|OWASP%\d+ZAP|chr\(122\)\.chr\(97\)\.chr\(112\)|Vega-Inject|VEGA123|vega\.invalid|PUT-putfile|w00tw00t|muieblackcat"), + ("potential dns changer", r"\b(dhcpPriDns|dhcpSecDns|staticPriDns|staticSecDns|staticThiDns|PriDnsv6|SecDnsv6|ThiDnsv6|staticPriDnsv6|staticSecDnsv6|staticThiDnsv6|dnsipv4|dns2ipv4|dnsipv6|dns2ipv6|pppoePriDns|pppoeSecDns|wan_dns1|wan_dns2|dnsPrimary|dnsSecondary|dnsDynamic|dnsRefresh|DNS_FST|DNS_SND|dhcpPriDns|dhcpSecDns|dnsserver|dnsserver1|dnsserver2|dns_server_ip_1|dns_server_ip_2|dns_server_ip_3|dns_server_ip_4|dns1|dns2|dns3|dns4|dns1_1|dns1_2|dns1_3|dns1_4|dns2_1|dns2_2|dns2_3|dns2_4|wan_dns_x|wan_dns1_x|wan_dns2_x|wan_dns3_x|wan_dns4_x|wan_dnsenable_x|dns_status|p_DNS|a_DNS|uiViewDns1Mark|uiViewDns2Mark|uiViewDNSRelay|is_router_as_dns|Enable_DNSFollowing|domainserverip|DSEN|DNSEN|dnsmode|dns%5Bserver1%5D|dns%5Bserver2%5D)=") ) SUSPICIOUS_HTTP_PATH_REGEXES = ( ("non-existent page", r"defaultwebpage\.cgi"), ("potential web scan", r"inexistent_file_name\.inexistent|test-for-some-inexistent-file|long_inexistent_path|some-inexistent-website\.acu") ) SUSPICIOUS_HTTP_REQUEST_PRE_CONDITION = ("?", "..", ".ht", "=", " ", "'") -SUSPICIOUS_HTTP_REQUEST_FORCE_ENCODE_CHARS = dict((_, urllib.quote(_)) for _ in "( )\r\n") +SUSPICIOUS_DIRECT_IP_URL_REGEX = r"\A[\w./-]*/[\w.]*\b(aarch|amd64\b|arm(\b|v?\d)|arcle-(750d|hs38)|exploit|m68k?\b|m[i1]ps\w{0,4}\b|mpsl\w?\b|pcc|powerp{1,2}c|pp-?c|riscv\w{0,3}\b|root|s390\w?\b|x86|x32|x64|i\d{1,2}\b|i386|i486|i586|i686|sparc|sh\b|wtf|yarn|zte)\Z" +SUSPICIOUS_PROXY_PROBE_PRE_CONDITION = ("probe", "proxy", "echo", "check") +SUSPICIOUS_HTTP_REQUEST_FORCE_ENCODE_CHARS = dict((_, _urllib.parse.quote(_)) for _ in "( )\r\n") SUSPICIOUS_UA_REGEX = "" OBSOLETE_UA_REGEX = r"(?i)windows NT [3-5]\.\d+|windows (3\.\d+|95|98|xp)|MSIE [1-6]\.\d+|Navigator/|Safari/[1-4]|Opera/[1-3]|Firefox/1?[0-9]\." -WEB_SHELLS = set() +GENERIC_SINKHOLE_REGEX = r"(?im)^(X-Sinkhole|Server): (malware-?)?sinkhole|\bSinkholed? by |^(X-Sinkholed?(-Domain)?|X-Zinkhole|X-Sinkhole):| a malware sinkhole|\bSinkhole( Project)?|This is a sinkhole|bots party hard|computers connecting to this sinkhole| Sinkhole by |^Set-Cookie: snkz=|^Server: Apache [0-9.]+/SinkSoft|^Location:[^\n]+\.sinkdns\.org:80" WORST_ASNS = {} +BOGON_IPS = {"::1"} BOGON_RANGES = {} CDN_RANGES = {} WHITELIST_HTTP_REQUEST_PATHS = ("fql", "yql", "ads", "../images/", "../themes/", "../design/", "../scripts/", "../assets/", "../core/", "../js/", "/gwx/") -WHITELIST_UA_KEYWORDS = ("AntiVir-NGUpd", "TMSPS", "AVGSETUP", "SDDS", "Sophos", "Symantec", "internal dummy connection") +WHITELIST_UA_REGEX = r"AntiVir\-NGUpd|TMSPS|AVGSETUP|SDDS|Sophos|Symantec|internal dummy connection|Microsoft\-CryptoAPI" WHITELIST_LONG_DOMAIN_NAME_KEYWORDS = ("blogspot",) +LOCAL_SUBDOMAIN_LOOKUPS = ("wpad", "autodiscover", "_ldap._tcp") SESSIONS = {} NO_SUCH_NAME_COUNTERS = {} # this won't be (expensive) shared in multiprocessing run (hence, the threshold will effectively be n-times higher) SESSION_ID_LENGTH = 16 @@ -114,19 +126,23 @@ IPPROTO_LUT = dict(((getattr(socket, _), _.replace("IPPROTO_", "")) for _ in dir(socket) if _.startswith("IPPROTO_"))) DEFLATE_COMPRESS_LEVEL = 9 PORT_SCANNING_THRESHOLD = 10 -MAX_RESULT_CACHE_ENTRIES = 10000 +WEB_SCANNING_THRESHOLD = 10 +INFECTION_SCANNING_THRESHOLD = 32 +MAX_CACHE_ENTRIES = 1000 MMAP_ZFILL_CHUNK_LENGTH = 1024 * 1024 +HOURLY_SECS = 1 * 60 * 60 DAILY_SECS = 24 * 60 * 60 DNS_EXHAUSTION_THRESHOLD = 1000 SUSPICIOUS_DOMAIN_LENGTH_THRESHOLD = 24 -SUSPICIOUS_DOMAIN_CONSONANT_THRESHOLD = 7 +SUSPICIOUS_DOMAIN_CONSONANT_THRESHOLD = 9 SUSPICIOUS_DOMAIN_ENTROPY_THRESHOLD = 3.5 WHITELIST = set() WHITELIST_RANGES = set() +IGNORE_EVENTS = set() STATIC_IPCAT_LOOKUPS = {"shadowserver.org": ("184.105.139.66-184.105.139.126", "184.105.247.194-184.105.247.254", "74.82.47.1-74.82.47.63", "216.218.206.66-216.218.206.126"), "labs.rapid7.com": ("71.6.216.32-71.6.216.63",), "shodan.io": ("66.240.192.138", "66.240.236.119", "71.6.135.131", "71.6.165.200", "71.6.167.142", "82.221.105.6", "82.221.105.7", "85.25.43.94", "85.25.103.50", "93.120.27.62", "104.131.0.69", "104.236.198.48", "162.159.244.38", "188.138.9.50", "198.20.69.74", "198.20.69.98", "198.20.70.114", "198.20.87.98", "198.20.99.130", "208.180.20.97", "209.126.110.38"), "eecs.umich.edu": ("141.212.121.0-141.212.121.255", "141.212.122.0-141.212.122.255"), "netsec.colostate.edu": ("129.82.138.12", "129.82.138.31", "129.82.138.32", "129.82.138.33", "129.82.138.34", "129.82.138.44"), "ant.isi.edu": ("128.9.168.98", "203.178.148.18", "203.178.148.19"), "eecs.berkeley.edu": ("169.229.3.89", "169.229.3.90", "169.229.3.91", "169.229.3.92", "169.229.3.93", "169.229.3.94"), "openresolverproject.org": ("204.42.253.2", "204.42.254.5"), "opensnmpproject.org": ("204.42.253.130",), "openntpproject.org": ("204.42.253.131",), "openssdpproject.org": ("204.42.253.132",), "projectblindferret.com": ("107.150.52.82-107.150.52.86",), "kudelskisecurity.com": ("185.35.62.0-185.35.62.255",), "riskiq.com": ("64.125.239.0-64.125.239.255",), "comsys.rwth-aachen.de": ("137.226.113.0-137.226.113.63",), "sba-research.org": ("98.189.26.18",)} # Reference: https://gist.github.com/ryanwitt/588678 -DLT_OFFSETS = { 0: 4, 1: 14, 6: 22, 7: 6, 8: 16, 9: 4, 10: 21, 117: 48, 18: 4, 12 if sys.platform.find('openbsd') != -1 else 108: 4, 14 if sys.platform.find('openbsd') != -1 else 12: 0, 113: 16 } +DLT_OFFSETS = {0: 4, 1: 14, 6: 22, 7: 6, 8: 16, 9: 4, 10: 21, 117: 48, 18: 4, 12 if sys.platform.find('openbsd') != -1 else 108: 4, 14 if sys.platform.find('openbsd') != -1 else 12: 0, 113: 16} try: import multiprocessing @@ -134,15 +150,19 @@ except ImportError: CPU_CORES = 1 +config = AttribDict({"TRAILS_FILE": DEFAULT_TRAILS_FILE}) +trails = TrailsDict() + def _get_total_physmem(): retval = None try: - if subprocess.mswindows: + if IS_WIN: import ctypes kernel32 = ctypes.windll.kernel32 c_ulong = ctypes.c_ulong + class MEMORYSTATUS(ctypes.Structure): _fields_ = [ ('dwLength', c_ulong), @@ -204,26 +224,19 @@ class MEMORYSTATUS(ctypes.Structure): return retval -def check_memory(): - print "[?] at least %dMB of free memory required" % (CHECK_MEMORY_SIZE / 1024 / 1024) - try: - _ = '0' * CHECK_MEMORY_SIZE - except MemoryError: - exit("[!] not enough memory") - def read_config(config_file): global config if not os.path.isfile(config_file): - exit("[!] missing configuration file '%s'" % config_file) + sys.exit("[!] missing configuration file '%s'" % config_file) else: - print "[i] using configuration file '%s'" % config_file + print("[i] using configuration file '%s'" % config_file) config.clear() try: array = None - content = open(config_file, "rb").read() + content = open(config_file, "r").read() for line in content.split("\n"): line = line.strip('\r') @@ -234,15 +247,21 @@ def read_config(config_file): if line.count(' ') == 0: if re.search(r"[^\w]", line): if array == "USERS": - exit("[!] invalid USERS entry '%s'\n[?] (hint: add whitespace at start of line)" % line) + sys.exit("[!] invalid USERS entry '%s'\n[?] (hint: add whitespace at start of line)" % line) else: - exit("[!] invalid configuration (line: '%s')" % line) + sys.exit("[!] invalid configuration (line: '%s')" % line) array = line.upper() config[array] = [] continue if array and line.startswith(' '): - config[array].append(line.strip()) + line = line.strip() + if array == "IP_ALIASES" and any(_ in line.split(':')[0] for _ in ('/', '-')): + for addr in expand_range(line.split(':')[0]): + config[array].append("%s:%s" % (addr, line.split(':', 1)[-1])) + else: + config[array].append(line) + continue else: array = None @@ -278,14 +297,14 @@ def read_config(config_file): pass for option in ("MONITOR_INTERFACE", "CAPTURE_BUFFER", "LOG_DIR"): - if not option in config: - exit("[!] missing mandatory option '%s' in configuration file '%s'" % (option, config_file)) + if option not in config: + sys.exit("[!] missing mandatory option '%s' in configuration file '%s'" % (option, config_file)) for entry in (config.USERS or []): if len(entry.split(':')) != 4: - exit("[!] invalid USERS entry '%s'" % entry) + sys.exit("[!] invalid USERS entry '%s'" % entry) if re.search(r"\$\d+\$", entry): - exit("[!] invalid USERS entry '%s'\n[?] (hint: please update PBKDF2 hashes to SHA256 in your configuration file)" % entry) + sys.exit("[!] invalid USERS entry '%s'\n[?] (hint: please update PBKDF2 hashes to SHA256 in your configuration file)" % entry) if config.SSL_PEM: config.SSL_PEM = config.SSL_PEM.replace('/', os.sep) @@ -294,10 +313,16 @@ def read_config(config_file): if ',' in config.USER_WHITELIST: print("[x] configuration value 'USER_WHITELIST' has been changed. Please use it to set location of whitelist file") elif not os.path.isfile(config.USER_WHITELIST): - exit("[!] missing 'USER_WHITELIST' file '%s'" % config.USER_WHITELIST) + sys.exit("[!] missing 'USER_WHITELIST' file '%s'" % config.USER_WHITELIST) else: read_whitelist() + if config.USER_IGNORELIST: + if not os.path.isfile(config.USER_IGNORELIST): + sys.exit("[!] missing 'USER_IGNORELIST' file '%s'" % config.USER_IGNORELIST) + else: + read_ignorelist() + config.PROCESS_COUNT = int(config.PROCESS_COUNT or CPU_CORES) if config.USE_MULTIPROCESSING: @@ -307,16 +332,19 @@ def read_config(config_file): print("[x] configuration switch 'DISABLE_LOCAL_LOG_STORAGE' turned on and neither option 'LOG_SERVER' nor 'SYSLOG_SERVER' are set. Falling back to console output of event data") if config.UDP_ADDRESS is not None and config.UDP_PORT is None: - exit("[!] usage of configuration value 'UDP_ADDRESS' requires also usage of 'UDP_PORT'") + sys.exit("[!] usage of configuration value 'UDP_ADDRESS' requires also usage of 'UDP_PORT'") if config.UDP_ADDRESS is None and config.UDP_PORT is not None: - exit("[!] usage of configuration value 'UDP_PORT' requires also usage of 'UDP_ADDRESS'") + sys.exit("[!] usage of configuration value 'UDP_PORT' requires also usage of 'UDP_ADDRESS'") + + if not str(config.HTTP_PORT or "").isdigit() and not IS_SENSOR: + sys.exit("[!] invalid configuration value for 'HTTP_PORT' ('%s')" % ("" if config.HTTP_PORT is None else config.HTTP_PORT)) - if not str(config.HTTP_PORT or "").isdigit(): - exit("[!] invalid configuration value for 'HTTP_PORT' ('%s')" % config.HTTP_PORT) + if not str(config.UPDATE_PERIOD or "").isdigit(): + sys.exit("[!] invalid configuration value for 'UPDATE_PERIOD' ('%s')" % ("" if config.UPDATE_PERIOD is None else config.UPDATE_PERIOD)) - if config.PROCESS_COUNT and subprocess.mswindows: - print "[x] multiprocessing is currently not supported on Windows OS" + if config.PROCESS_COUNT and IS_WIN: + print("[x] multiprocessing is currently not supported on Windows OS") config.PROCESS_COUNT = 1 if config.CAPTURE_BUFFER: @@ -329,18 +357,26 @@ def read_config(config_file): physmem = _get_total_physmem() if physmem: - config.CAPTURE_BUFFER = physmem * int(re.search(r"(\d+)%", config.CAPTURE_BUFFER).group(1)) / 100 + config.CAPTURE_BUFFER = physmem * int(re.search(r"(\d+)%", config.CAPTURE_BUFFER).group(1)) // 100 else: - exit("[!] unable to determine total physical memory. Please use absolute value for 'CAPTURE_BUFFER'") + sys.exit("[!] unable to determine total physical memory. Please use absolute value for 'CAPTURE_BUFFER'") else: - exit("[!] invalid configuration value for 'CAPTURE_BUFFER' ('%s')" % config.CAPTURE_BUFFER) + sys.exit("[!] invalid configuration value for 'CAPTURE_BUFFER' ('%s')" % config.CAPTURE_BUFFER) - config.CAPTURE_BUFFER = config.CAPTURE_BUFFER / BLOCK_LENGTH * BLOCK_LENGTH + config.CAPTURE_BUFFER = config.CAPTURE_BUFFER // BLOCK_LENGTH * BLOCK_LENGTH if config.PROXY_ADDRESS: PROXIES.update({"http": config.PROXY_ADDRESS, "https": config.PROXY_ADDRESS}) - opener = urllib2.build_opener(urllib2.ProxyHandler(PROXIES)) - urllib2.install_opener(opener) + opener = _urllib.request.build_opener(_urllib.request.ProxyHandler(PROXIES)) + _urllib.request.install_opener(opener) + + if not config.TRAILS_FILE: + config.TRAILS_FILE = DEFAULT_TRAILS_FILE + else: + config.TRAILS_FILE = os.path.abspath(os.path.expanduser(config.TRAILS_FILE)) + + if int(os.environ.get("MALTRAIL_DREI", 0)) > 0: + config.SHOW_DEBUG = True def read_whitelist(): WHITELIST.clear() @@ -377,6 +413,28 @@ def read_whitelist(): else: WHITELIST.add(line) +# add rules to ignore event list from passed file +def add_ignorelist(filepath): + if filepath and os.path.isfile(filepath): + with open(filepath, "r") as f: + for line in f: + line = re.sub(r"\s+", "", line) + + if not line or line.startswith('#'): + continue + elif line.count(';') == 3: + src_ip, src_port, dst_ip, dst_port = line.split(';') + IGNORE_EVENTS.add((src_ip, src_port, dst_ip, dst_port)) + +def read_ignorelist(): + IGNORE_EVENTS.clear() + + _ = os.path.abspath(os.path.join(ROOT_DIR, "misc", "ignore_events.txt")) + add_ignorelist(_) + + if config.USER_IGNORELIST and os.path.isfile(config.USER_IGNORELIST): + add_ignorelist(config.USER_IGNORELIST) + def read_ua(): global SUSPICIOUS_UA_REGEX @@ -390,25 +448,19 @@ def read_ua(): line = line.strip() if not line or line.startswith('#'): continue + elif " (compatible" in line: + line = re.escape(line) else: - items.append(line) + try: + re.compile(line) + except: + line = re.escape(line) + + items.append(line) if items: SUSPICIOUS_UA_REGEX = "(?i)%s" % '|'.join(items) -def read_web_shells(): - WEB_SHELLS.clear() - - _ = os.path.abspath(os.path.join(ROOT_DIR, "misc", "web_shells.txt")) - if os.path.isfile(_): - with open(_, "r") as f: - for line in f: - line = line.strip() - if not line or line.startswith('#'): - continue - else: - WEB_SHELLS.add(line) - def read_worst_asn(): _ = os.path.abspath(os.path.join(ROOT_DIR, "misc", "worst_asns.txt")) if os.path.isfile(_): @@ -454,10 +506,21 @@ def read_bogon_ranges(): prefix, mask = line.split('/') BOGON_RANGES[key].append((addr_to_int(prefix), make_mask(int(mask)))) +def check_deprecated(): + if "--no-updates" in sys.argv: + print("[!] switch '--no-updates' was renamed to '--offline'") + sys.argv = [(_ if _ != "--no-updates" else "--offline") for _ in sys.argv] + + if "-i" in sys.argv: + print("[x] option '-i' was renamed to '-r'") + sys.argv = [(_ if _ != "-i" else "-r") for _ in sys.argv] + if __name__ != "__main__": + init_output() read_whitelist() + read_ignorelist() read_ua() - read_web_shells() read_worst_asn() read_cdn_ranges() read_bogon_ranges() + check_deprecated() diff --git a/core/trailsdict.py b/core/trailsdict.py index f61bab0236c..efb1a311a9f 100644 --- a/core/trailsdict.py +++ b/core/trailsdict.py @@ -1,7 +1,7 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ @@ -10,6 +10,7 @@ class TrailsDict(dict): def __init__(self): self._trails = {} + self._regex = "" self._infos = [] self._reverse_infos = {} self._references = [] @@ -25,9 +26,7 @@ def __contains__(self, key): return key in self._trails def clear(self): - self._trails.clear() - self._infos = [] - self._references = [] + self.__init__() def keys(self): return self._trails.keys() @@ -43,9 +42,10 @@ def __iter__(self): def get(self, key, default=None): if key in self._trails: _ = self._trails[key].split(',') - return (self._infos[int(_[0])], self._references[int(_[1])]) - else: - return default + if len(_) == 2: + return (self._infos[int(_[0])], self._references[int(_[1])]) + + return default def update(self, value): if isinstance(value, TrailsDict): @@ -75,9 +75,10 @@ def __len__(self): def __getitem__(self, key): if key in self._trails: _ = self._trails[key].split(',') - return (self._infos[int(_[0])], self._references[int(_[1])]) - else: - raise KeyError(key) + if len(_) == 2: + return (self._infos[int(_[0])], self._references[int(_[1])]) + + raise KeyError(key) def __setitem__(self, key, value): if isinstance(value, (tuple, list)): diff --git a/core/update.py b/core/update.py index 7aa96c9e8ff..42c2ca403fd 100755 --- a/core/update.py +++ b/core/update.py @@ -1,32 +1,35 @@ #!/usr/bin/env python """ -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) +Copyright (c) 2014-2024 Maltrail developers (https://github.com/stamparm/maltrail/) See the file 'LICENSE' for copying permission """ +from __future__ import print_function +import codecs import csv import glob import inspect import os import re import sqlite3 -import subprocess import sys import time -import urllib2 -import urlparse sys.dont_write_bytecode = True sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) # to enable calling from current directory too from core.addr import addr_to_int +from core.addr import int_to_addr +from core.addr import make_mask from core.common import bogon_ip from core.common import cdn_ip from core.common import check_whitelisted from core.common import load_trails from core.common import retrieve_content +from core.compat import xrange from core.settings import config +from core.settings import read_config from core.settings import read_whitelist from core.settings import BAD_TRAIL_PREFIXES from core.settings import FRESH_IPCAT_DELTA_DAYS @@ -36,244 +39,321 @@ from core.settings import IPCAT_CSV_FILE from core.settings import IPCAT_SQLITE_FILE from core.settings import IPCAT_URL -from core.settings import PROXIES +from core.settings import IS_WIN from core.settings import ROOT_DIR -from core.settings import TRAILS_FILE +from core.settings import UNICODE_ENCODING from core.settings import USERS_DIR +from core.trailsdict import TrailsDict +from thirdparty import six +from thirdparty.six.moves import urllib as _urllib + +# patch for self-signed certificates (e.g. CUSTOM_TRAILS_URL) +try: + import ssl + ssl._create_default_https_context = ssl._create_unverified_context +except (ImportError, AttributeError): + pass def _chown(filepath): - if not subprocess.mswindows and os.path.exists(filepath): + if not IS_WIN and os.path.exists(filepath): try: os.chown(filepath, int(os.environ.get("SUDO_UID", -1)), int(os.environ.get("SUDO_GID", -1))) - except Exception, ex: - print "[!] chown problem with '%s' ('%s')" % (filepath, ex) + except Exception as ex: + print("[!] chown problem with '%s' ('%s')" % (filepath, ex)) -def _fopen(filepath, mode="rb"): - retval = open(filepath, mode) +def _fopen(filepath, mode="rb", opener=open): + retval = opener(filepath, mode) if "w+" in mode: _chown(filepath) return retval -def update_trails(server=None, force=False, offline=False): +def update_trails(force=False, offline=False): """ Update trails from feeds """ success = False - trails = {} + trails = TrailsDict() duplicates = {} try: if not os.path.isdir(USERS_DIR): - os.makedirs(USERS_DIR, 0755) - except Exception, ex: - exit("[!] something went wrong during creation of directory '%s' ('%s')" % (USERS_DIR, ex)) + os.makedirs(USERS_DIR, 0o755) + except Exception as ex: + sys.exit("[!] something went wrong during creation of directory '%s' ('%s')" % (USERS_DIR, ex)) _chown(USERS_DIR) - if server: - print "[i] retrieving trails from provided 'UPDATE_SERVER' server..." - content = retrieve_content(server) - if not content: - exit("[!] unable to retrieve data from '%s'" % server) + if config.UPDATE_SERVER: + print("[i] retrieving trails from provided 'UPDATE_SERVER' server...") + content = retrieve_content(config.UPDATE_SERVER) + if not content or content.count(',') < 2: + print("[x] unable to retrieve data from '%s'" % config.UPDATE_SERVER) else: - with _fopen(TRAILS_FILE, "w+b") as f: + with _fopen(config.TRAILS_FILE, "w+b" if six.PY2 else "w+", open if six.PY2 else codecs.open) as f: f.write(content) trails = load_trails() - trail_files = set() - for dirpath, dirnames, filenames in os.walk(os.path.abspath(os.path.join(ROOT_DIR, "trails"))) : - for filename in filenames: - trail_files.add(os.path.abspath(os.path.join(dirpath, filename))) - - if config.CUSTOM_TRAILS_DIR: - for dirpath, dirnames, filenames in os.walk(os.path.abspath(os.path.join(ROOT_DIR, os.path.expanduser(config.CUSTOM_TRAILS_DIR)))) : + else: + trail_files = set() + for dirpath, dirnames, filenames in os.walk(os.path.abspath(os.path.join(ROOT_DIR, "trails"))): for filename in filenames: trail_files.add(os.path.abspath(os.path.join(dirpath, filename))) - if not trails and (force or not os.path.isfile(TRAILS_FILE) or (time.time() - os.stat(TRAILS_FILE).st_mtime) >= config.UPDATE_PERIOD or os.stat(TRAILS_FILE).st_size == 0 or any(os.stat(_).st_mtime > os.stat(TRAILS_FILE).st_mtime for _ in trail_files)): - print "[i] updating trails (this might take a while)..." + if config.CUSTOM_TRAILS_DIR: + for dirpath, dirnames, filenames in os.walk(os.path.abspath(os.path.join(ROOT_DIR, os.path.expanduser(config.CUSTOM_TRAILS_DIR)))): + for filename in filenames: + trail_files.add(os.path.abspath(os.path.join(dirpath, filename))) + + if not trails and (force or not os.path.isfile(config.TRAILS_FILE) or (time.time() - os.stat(config.TRAILS_FILE).st_mtime) >= config.UPDATE_PERIOD or os.stat(config.TRAILS_FILE).st_size == 0 or any(os.stat(_).st_mtime > os.stat(config.TRAILS_FILE).st_mtime for _ in trail_files)): + if not config.offline: + print("[i] updating trails (this might take a while)...") + else: + print("[i] checking trails...") + + if not offline and (force or config.USE_FEED_UPDATES): + _ = os.path.abspath(os.path.join(ROOT_DIR, "trails", "feeds")) + if _ not in sys.path: + sys.path.append(_) - if not offline and (force or config.USE_FEED_UPDATES): - _ = os.path.abspath(os.path.join(ROOT_DIR, "trails", "feeds")) + filenames = sorted(glob.glob(os.path.join(_, "*.py"))) + else: + filenames = [] + + _ = os.path.abspath(os.path.join(ROOT_DIR, "trails")) if _ not in sys.path: sys.path.append(_) - filenames = sorted(glob.glob(os.path.join(_, "*.py"))) - else: - filenames = [] + filenames += [os.path.join(_, "custom")] + filenames += [os.path.join(_, "static")] # Note: higher priority than previous one because of dummy user trails (FE) - _ = os.path.abspath(os.path.join(ROOT_DIR, "trails")) - if _ not in sys.path: - sys.path.append(_) + filenames = [_ for _ in filenames if "__init__.py" not in _] - filenames += [os.path.join(_, "static")] - filenames += [os.path.join(_, "custom")] + if config.DISABLED_FEEDS: + filenames = [filename for filename in filenames if os.path.splitext(os.path.split(filename)[-1])[0] not in re.split(r"[^\w]+", config.DISABLED_FEEDS)] - filenames = [_ for _ in filenames if "__init__.py" not in _] + for i in xrange(len(filenames)): + filename = filenames[i] - if config.DISABLED_FEEDS: - filenames = [filename for filename in filenames if os.path.splitext(os.path.split(filename)[-1])[0] not in re.split(r"[^\w]+", config.DISABLED_FEEDS)] + try: + module = __import__(os.path.basename(filename).split(".py")[0]) + except (ImportError, SyntaxError) as ex: + print("[x] something went wrong during import of feed file '%s' ('%s')" % (filename, ex)) + continue + + for name, function in inspect.getmembers(module, inspect.isfunction): + if name == "fetch": + url = module.__url__ # Note: to prevent "SyntaxError: can not delete variable 'module' referenced in nested scope" + + print(" [o] '%s'%s" % (url, " " * 20 if len(url) < 20 else "")) + sys.stdout.write("[?] progress: %d/%d (%d%%)\r" % (i, len(filenames), i * 100 // len(filenames))) + sys.stdout.flush() + + if config.DISABLED_TRAILS_INFO_REGEX and re.search(config.DISABLED_TRAILS_INFO_REGEX, getattr(module, "__info__", "")): + continue + + try: + results = function() + for item in results.items(): + if item[0].startswith("www.") and '/' not in item[0]: + item = [item[0][len("www."):], item[1]] + if item[0] in trails: + if item[0] not in duplicates: + duplicates[item[0]] = set((trails[item[0]][1],)) + duplicates[item[0]].add(item[1][1]) + if not (item[0] in trails and (any(_ in item[1][0] for _ in LOW_PRIORITY_INFO_KEYWORDS) or trails[item[0]][1] in HIGH_PRIORITY_REFERENCES)) or (item[1][1] in HIGH_PRIORITY_REFERENCES and "history" not in item[1][0]) or any(_ in item[1][0] for _ in HIGH_PRIORITY_INFO_KEYWORDS): + trails[item[0]] = item[1] + if not results and not any(_ in url for _ in ("abuse.ch", "cobaltstrike")): + print("[x] something went wrong during remote data retrieval ('%s')" % url) + except Exception as ex: + print("[x] something went wrong during processing of feed file '%s' ('%s')" % (filename, ex)) - for i in xrange(len(filenames)): - filename = filenames[i] + try: + sys.modules.pop(module.__name__) + del module + except Exception: + pass + + # custom trails from remote location + if config.CUSTOM_TRAILS_URL: + print(" [o] '(remote custom)'%s" % (" " * 20)) + for url in re.split(r"[;,]", config.CUSTOM_TRAILS_URL): + url = url.strip() + if not url: + continue - try: - module = __import__(os.path.basename(filename).split(".py")[0]) - except (ImportError, SyntaxError), ex: - print "[x] something went wrong during import of feed file '%s' ('%s')" % (filename, ex) - continue - - for name, function in inspect.getmembers(module, inspect.isfunction): - if name == "fetch": - print(" [o] '%s'%s" % (module.__url__, " " * 20 if len(module.__url__) < 20 else "")) - sys.stdout.write("[?] progress: %d/%d (%d%%)\r" % (i, len(filenames), i * 100 / len(filenames))) - sys.stdout.flush() - try: - results = function() - for item in results.items(): - if item[0].startswith("www.") and '/' not in item[0]: - item = [item[0][len("www."):], item[1]] - if item[0] in trails: - if item[0] not in duplicates: - duplicates[item[0]] = set((trails[item[0]][1],)) - duplicates[item[0]].add(item[1][1]) - if not (item[0] in trails and (any(_ in item[1][0] for _ in LOW_PRIORITY_INFO_KEYWORDS) or trails[item[0]][1] in HIGH_PRIORITY_REFERENCES)) or (item[1][1] in HIGH_PRIORITY_REFERENCES and "history" not in item[1][0]) or any(_ in item[1][0] for _ in HIGH_PRIORITY_INFO_KEYWORDS): - trails[item[0]] = item[1] - if not results and "abuse.ch" not in module.__url__: - print "[x] something went wrong during remote data retrieval ('%s')" % module.__url__ - except Exception, ex: - print "[x] something went wrong during processing of feed file '%s' ('%s')" % (filename, ex) + url = ("http://%s" % url) if "//" not in url else url + content = retrieve_content(url) - try: - sys.modules.pop(module.__name__) - del module - except Exception: - pass - - # custom trails from remote location - if config.CUSTOM_TRAILS_URL: - print(" [o] '(remote custom)'%s" % (" " * 20)) - content = retrieve_content(config.CUSTOM_TRAILS_URL) - if not content: - exit("[!] unable to retrieve data (or empty response) from '%s'" % config.CUSTOM_TRAILS_URL) - else: - url = config.CUSTOM_TRAILS_URL - url = ("http://%s" % url) if not "//" in url else url - __info__ = "blacklisted" - __reference__ = "(remote custom)" # urlparse.urlsplit(url).netloc - for line in content.split('\n'): - line = line.strip() - if not line or line.startswith('#'): + if not content: + print("[x] unable to retrieve data (or empty response) from '%s'" % url) + else: + __info__ = "blacklisted" + __reference__ = "(remote custom)" # urlparse.urlsplit(url).netloc + for line in content.split('\n'): + line = line.strip() + if not line or line.startswith('#'): + continue + line = re.sub(r"\s*#.*", "", line) + if '://' in line: + line = re.search(r"://(.*)", line).group(1) + line = line.rstrip('/') + + if line in trails and any(_ in trails[line][1] for _ in ("custom", "static")): + continue + + if '/' in line: + trails[line] = (__info__, __reference__) + line = line.split('/')[0] + elif re.search(r"\A\d+\.\d+\.\d+\.\d+\Z", line): + trails[line] = (__info__, __reference__) + else: + trails[line.strip('.')] = (__info__, __reference__) + + for match in re.finditer(r"(\d+\.\d+\.\d+\.\d+)/(\d+)", content): + prefix, mask = match.groups() + mask = int(mask) + if mask > 32: + continue + start_int = addr_to_int(prefix) & make_mask(mask) + end_int = start_int | ((1 << 32 - mask) - 1) + if 0 <= end_int - start_int <= 1024: + address = start_int + while start_int <= address <= end_int: + trails[int_to_addr(address)] = (__info__, __reference__) + address += 1 + + print("[i] post-processing trails (this might take a while)...") + + # basic cleanup + for key in list(trails.keys()): + if key not in trails: + continue + + if config.DISABLED_TRAILS_INFO_REGEX: + if re.search(config.DISABLED_TRAILS_INFO_REGEX, trails[key][0]): + del trails[key] continue - line = re.sub(r"\s*#.*", "", line) - if '://' in line: - line = re.search(r"://(.*)", line).group(1) - line = line.rstrip('/') - if line in trails and any(_ in trails[line][1] for _ in ("custom", "static")): + try: + _key = key.decode(UNICODE_ENCODING) if isinstance(key, bytes) else key + _key = _key.encode("idna") + if six.PY3: + _key = _key.decode(UNICODE_ENCODING) + if _key != key: # for domains with non-ASCII letters (e.g. phishing) + trails[_key] = trails[key] + del trails[key] + key = _key + except: + pass + + if not key or re.search(r"(?i)\A\.?[a-z]+\Z", key) and not any(_ in trails[key][1] for _ in ("custom", "static")): + del trails[key] + continue + + if re.search(r"\A\d+\.\d+\.\d+\.\d+\Z", key): + if any(_ in trails[key][0] for _ in ("parking site", "sinkhole")) and key in duplicates: # Note: delete (e.g.) junk custom trails if static trail is a sinkhole + del duplicates[key] + + if trails[key][0] == "malware": + trails[key] = ("potential malware site", trails[key][1]) + + if config.get("IP_MINIMUM_FEEDS", 3) > 1: + if (key not in duplicates or len(duplicates[key]) < config.get("IP_MINIMUM_FEEDS", 3)) and re.search(r"\b(custom|static)\b", trails[key][1]) is None: + del trails[key] + continue + + if any(int(_) > 255 for _ in key.split('.')): + del trails[key] continue - if '/' in line: - trails[line] = (__info__, __reference__) - line = line.split('/')[0] - elif re.search(r"\A\d+\.\d+\.\d+\.\d+\Z", line): - trails[line] = (__info__, __reference__) - else: - trails[line.strip('.')] = (__info__, __reference__) - - # basic cleanup - for key in trails.keys(): - if key not in trails: - continue - if not key or re.search(r"\A(?i)\.?[a-z]+\Z", key) and not any(_ in trails[key][1] for _ in ("custom", "static")): - del trails[key] - continue - if re.search(r"\A\d+\.\d+\.\d+\.\d+\Z", key): - if any(_ in trails[key][0] for _ in ("parking site", "sinkhole")) and key in duplicates: - del duplicates[key] - if trails[key][0] == "malware": - trails[key] = ("potential malware site", trails[key][1]) - if trails[key][0] == "ransomware": - trails[key] = ("ransomware (malware)", trails[key][1]) - if key.startswith("www.") and '/' not in key: - _ = trails[key] - del trails[key] - key = key[len("www."):] - if key: + if trails[key][0] == "ransomware": + trails[key] = ("ransomware (malware)", trails[key][1]) + + if key.startswith("www.") and '/' not in key: + _ = trails[key] + del trails[key] + key = key[len("www."):] + if key: + trails[key] = _ + + if '?' in key and not key.startswith('/'): + _ = trails[key] + del trails[key] + key = key.split('?')[0] + if key: + trails[key] = _ + + if '//' in key: + _ = trails[key] + del trails[key] + key = key.replace('//', '/') trails[key] = _ - if '?' in key: - _ = trails[key] - del trails[key] - key = key.split('?')[0] - if key: + + if key != key.lower(): + _ = trails[key] + del trails[key] + key = key.lower() trails[key] = _ - if '//' in key: - _ = trails[key] - del trails[key] - key = key.replace('//', '/') - trails[key] = _ - if key != key.lower(): - _ = trails[key] - del trails[key] - key = key.lower() - trails[key] = _ - if key in duplicates: - _ = trails[key] - others = sorted(duplicates[key] - set((_[1],))) - if others and " (+" not in _[1]: - trails[key] = (_[0], "%s (+%s)" % (_[1], ','.join(others))) - - read_whitelist() - - for key in trails.keys(): - if check_whitelisted(key) or any(key.startswith(_) for _ in BAD_TRAIL_PREFIXES): - del trails[key] - elif re.search(r"\A\d+\.\d+\.\d+\.\d+\Z", key) and (bogon_ip(key) or cdn_ip(key)): - del trails[key] - else: - try: - key.decode("utf8") - trails[key][0].decode("utf8") - trails[key][1].decode("utf8") - except UnicodeDecodeError: + + if key in duplicates: + _ = trails[key] + others = sorted(duplicates[key] - set((_[1],))) + if others and " (+" not in _[1]: + trails[key] = (_[0], "%s (+%s)" % (_[1], ','.join(others))) + + read_whitelist() + + for key in list(trails.keys()): + match = re.search(r"\A(\d+\.\d+\.\d+\.\d+)\b", key) + if check_whitelisted(key) or any(key.startswith(_) for _ in BAD_TRAIL_PREFIXES): + del trails[key] + elif match and (bogon_ip(match.group(1)) or cdn_ip(match.group(1))) and not any(_ in trails[key][0] for _ in ("parking", "sinkhole")): del trails[key] + else: + try: + key.decode("utf8") if hasattr(key, "decode") else key.encode("utf8") + trails[key][0].decode("utf8") if hasattr(trails[key][0], "decode") else trails[key][0].encode("utf8") + trails[key][1].decode("utf8") if hasattr(trails[key][1], "decode") else trails[key][1].encode("utf8") + except UnicodeError: + del trails[key] - try: - if trails: - with _fopen(TRAILS_FILE, "w+b") as f: - writer = csv.writer(f, delimiter=',', quotechar='\"', quoting=csv.QUOTE_MINIMAL) - for trail in trails: - writer.writerow((trail, trails[trail][0], trails[trail][1])) + try: + if trails: + with _fopen(config.TRAILS_FILE, "w+b" if six.PY2 else "w+", open if six.PY2 else codecs.open) as f: + writer = csv.writer(f, delimiter=',', quotechar='\"', quoting=csv.QUOTE_MINIMAL) + for trail in trails: + row = (trail, trails[trail][0], trails[trail][1]) + writer.writerow(row) - success = True - except Exception, ex: - print "[x] something went wrong during trails file write '%s' ('%s')" % (TRAILS_FILE, ex) + success = True + except Exception as ex: + print("[x] something went wrong during trails file write '%s' ('%s')" % (config.TRAILS_FILE, ex)) - print "[i] update finished%s" % (40 * " ") + print("[i] update finished%s" % (40 * " ")) - if success: - print "[i] trails stored to '%s'" % TRAILS_FILE + if success: + print("[i] trails stored to '%s'" % config.TRAILS_FILE) return trails def update_ipcat(force=False): try: if not os.path.isdir(USERS_DIR): - os.makedirs(USERS_DIR, 0755) - except Exception, ex: - exit("[!] something went wrong during creation of directory '%s' ('%s')" % (USERS_DIR, ex)) + os.makedirs(USERS_DIR, 0o755) + except Exception as ex: + sys.exit("[!] something went wrong during creation of directory '%s' ('%s')" % (USERS_DIR, ex)) _chown(USERS_DIR) if force or not os.path.isfile(IPCAT_CSV_FILE) or not os.path.isfile(IPCAT_SQLITE_FILE) or (time.time() - os.stat(IPCAT_CSV_FILE).st_mtime) >= FRESH_IPCAT_DELTA_DAYS * 24 * 3600 or os.stat(IPCAT_SQLITE_FILE).st_size == 0: - print "[i] updating ipcat database..." + print("[i] updating ipcat database...") try: - with file(IPCAT_CSV_FILE, "w+b") as f: - f.write(urllib2.urlopen(IPCAT_URL).read()) - except Exception, ex: - print "[x] something went wrong during retrieval of '%s' ('%s')" % (IPCAT_URL, ex) + with open(IPCAT_CSV_FILE, "w+b") as f: + f.write(_urllib.request.urlopen(IPCAT_URL).read()) + except Exception as ex: + print("[x] something went wrong during retrieval of '%s' ('%s')" % (IPCAT_URL, ex)) else: try: @@ -294,22 +374,27 @@ def update_ipcat(force=False): cur.execute("COMMIT") cur.close() con.commit() - except Exception, ex: - print "[x] something went wrong during ipcat database update ('%s')" % ex + except Exception as ex: + print("[x] something went wrong during ipcat database update ('%s')" % ex) _chown(IPCAT_CSV_FILE) _chown(IPCAT_SQLITE_FILE) def main(): + if "-c" in sys.argv: + read_config(sys.argv[sys.argv.index("-c") + 1]) + try: - update_trails(force=True) - update_ipcat() + offline = "--offline" in sys.argv + update_trails(force=True, offline=offline) + if not offline: + update_ipcat() except KeyboardInterrupt: - print "\r[x] Ctrl-C pressed" + print("\r[x] Ctrl-C pressed") else: if "-r" in sys.argv: results = [] - with _fopen(TRAILS_FILE) as f: + with _fopen(config.TRAILS_FILE, "rb" if six.PY2 else 'r', open if six.PY2 else codecs.open) as f: for line in f: if line and line[0].isdigit(): items = line.split(',', 2) @@ -332,5 +417,10 @@ def main(): sys.stderr.write("%s\t%s\n" % (result[0], result[1])) sys.stderr.flush() + if "--console" in sys.argv: + with _fopen(config.TRAILS_FILE, "rb" if six.PY2 else 'r', open if six.PY2 else codecs.open) as f: + for line in f: + sys.stdout.write(line) + if __name__ == "__main__": main() diff --git a/core/versioncheck.py b/core/versioncheck.py deleted file mode 100755 index a0de9a7d069..00000000000 --- a/core/versioncheck.py +++ /dev/null @@ -1,13 +0,0 @@ -#!/usr/bin/env python - -""" -Copyright (c) 2014-2017 Miroslav Stampar (@stamparm) -See the file 'LICENSE' for copying permission -""" - -import sys - -PYVERSION = sys.version.split()[0] - -if PYVERSION >= "3" or PYVERSION < "2.6": - exit("[CRITICAL] incompatible Python version detected ('%s'). For successfully running Maltrail you'll have to use version 2.6 or 2.7 (visit 'http://www.python.org/download/')" % PYVERSION) diff --git a/docker/Dockerfile b/docker/Dockerfile index 7d516b42540..911812d3f39 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -1,14 +1,20 @@ -FROM debian:jessie +FROM ubuntu:focal RUN apt-get update \ && apt-get upgrade -y \ - && apt-get install -y python-pcapy git curl schedtool \ - && git clone https://github.com/stamparm/maltrail.git /root/maltrail \ - && python /root/maltrail/core/update.py + && apt-get install -y git python3 python3-dev python3-pip python-is-python3 libpcap-dev build-essential procps schedtool cron \ + && pip3 install pcapy-ng \ + && git clone --depth=1 https://github.com/stamparm/maltrail.git /opt/maltrail \ + && python /opt/maltrail/core/update.py -WORKDIR /root/maltrail +RUN touch /var/log/cron.log -COPY run.sh /root/run.sh +RUN (echo '*/1 * * * * if [ -n "$(ps -ef | grep -v grep | grep -v bash | grep server.py)" ]; then : ; else python /opt/maltrail/server.py -c /opt/maltrail/maltrail.conf; fi >> /var/log/cron.log') | crontab +RUN (crontab -l ; echo '*/1 * * * * if [ -n "$(ps -ef | grep -v grep | grep -v bash | grep sensor.py)" ]; then : ; else python /opt/maltrail/sensor.py -c /opt/maltrail/maltrail.conf; fi >> /var/log/cron.log') | crontab +RUN (crontab -l ; echo '0 1 * * * cd /opt/maltrail && git pull') | crontab +RUN (crontab -l ; echo '2 1 * * * /usr/bin/pkill -f maltrail') | crontab -ENTRYPOINT ["/bin/bash", "/root/run.sh"] +EXPOSE 8337/udp +EXPOSE 8338/tcp +CMD bash -c "python /opt/maltrail/server.py &" && bash -c "python /opt/maltrail/sensor.py &" && cron && tail -f /var/log/cron.log diff --git a/docker/README.md b/docker/README.md index be4cb417368..077f944ca27 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,11 +1,16 @@ -# Docker run +## Ubuntu/Debian -``` -sudo su -apt-get -qq install realpath -export MALTRAIL_CONF=$(realpath ../maltrail.conf) -for dev in $(ifconfig | grep eth | cut -d " " -f 1); do ifconfig $dev promisc; done -docker build -t maltrail . && \ -docker run -d --net=host --privileged -v /var/log/maltrail/:/var/log/maltrail/ -v $MALTRAIL_CONF:/root/maltrail/maltrail.conf maltrail && \ -docker ps # reporting interface: http://localhost:8338 +```sh + #!/bin/bash + export MALTRAIL_LOCAL=$(realpath ~/.local/share/maltrail) + mkdir -p $MALTRAIL_LOCAL + cd $MALTRAIL_LOCAL + wget https://raw.githubusercontent.com/stamparm/maltrail/master/docker/Dockerfile + wget https://raw.githubusercontent.com/stamparm/maltrail/master/maltrail.conf + sudo su + apt -qq -y install coreutils net-tools docker.io + for dev in $(ifconfig | grep mtu | grep -Eo '^\w+'); do ifconfig $dev promisc; done + mkdir -p /var/log/maltrail/ + docker build -t maltrail . && \ + docker run -d --name maltrail-docker --privileged -p 8337:8337/udp -p 8338:8338 -v /var/log/maltrail/:/var/log/maltrail/ -v $(pwd)/maltrail.conf:/opt/maltrail/maltrail.conf:ro maltrail ``` diff --git a/docker/run.sh b/docker/run.sh deleted file mode 100755 index bf60d39cf5a..00000000000 --- a/docker/run.sh +++ /dev/null @@ -1,3 +0,0 @@ -#!/bin/bash -python /root/maltrail/sensor.py & -python /root/maltrail/server.py diff --git a/fail2ban/maltrail.conf.example b/fail2ban/maltrail.conf.example new file mode 100644 index 00000000000..2e5b5bb2435 --- /dev/null +++ b/fail2ban/maltrail.conf.example @@ -0,0 +1,9 @@ +[Definition] +failregex = ^.*?\s+\s+\d+\s+\d+\.\d+\.\d+\.\d+\s+\d+\s+\w+\s+\w+\s+\d+\.\d+\.\d+\.\d+\s+"known attacker".*$ + ^.*?\s+\s+\d+\s+\d+\.\d+\.\d+\.\d+\s+\d+\s+\w+\s+\w+\s+\d+\.\d+\.\d+\.\d+\s+"mass scanner".*$ + ^.*?\s+\s+\d+\s+\d+\.\d+\.\d+\.\d+\s+\d+\s+\w+\s+\w+\s+\d+\.\d+\.\d+\.\d+\s+"malware".*$ + ^.*?\s+\s+\d+\s+\d+\.\d+\.\d+\.\d+\s+\d+\s+\w+\s+\w+\s+\d+\.\d+\.\d+\.\d+\s+"heuristic".*$ + ^.*?\s+\s+\d+\s+\d+\.\d+\.\d+\.\d+\s+\d+\s+\w+\s+\w+\s+\d+\.\d+\.\d+\.\d+\s+"attacker".*$ + ^.*?\s+\s+\d+\s+\d+\.\d+\.\d+\.\d+\s+\d+\s+\w+\s+\w+\s+\d+\.\d+\.\d+\.\d+\s+"reputation".*$ + ^.*?\s+\s+\d+\s+\d+\.\d+\.\d+\.\d+\s+\d+\s+\w+\s+\w+\s+\d+\.\d+\.\d+\.\d+\s+"potential[^"]*(web scan|directory traversal|injection|remote code|iot-malware download)".*$ + ^.*?\s+\s+\d+\s+\d+\.\d+\.\d+\.\d+\s+\d+\s+\w+\s+\w+\s+\d+\.\d+\.\d+\.\d+\s+"spammer".*$ diff --git a/html/css/main.css b/html/css/main.css index 993842cfa7b..721831838f5 100644 --- a/html/css/main.css +++ b/html/css/main.css @@ -50,7 +50,7 @@ table.dataTable .highlighted { background-color: #abb9d3 !important } text-shadow: -1px -1px 0 rgba(0, 0, 0, 0.25), -1px 1px 0 rgba(0, 0, 0, 0.25), 1px 1px 0 rgba(0, 0, 0, 0.25), 1px -1px 0 rgba(0, 0, 0, 0.25); color: white; } -.black-label-text { +.black-label-text, .ui-dialog-title { text-shadow: -1px -1px 0 rgba(255, 255, 255, 0.25), -1px 1px 0 rgba(255, 255, 255, 0.25), 1px 1px 0 rgba(255, 255, 255, 0.25), 1px -1px 0 rgba(255, 255, 255, 0.25); color: black; } @@ -67,9 +67,12 @@ table.dataTable .highlighted { background-color: #abb9d3 !important } width: initial; display: inline-block; } -.hidden { display: none } +.hidden, .searchable { display: none } /*.section { font-size: 120%; font-weight: bold; margin-top: 10px; margin-bottom: 10px; visibility: hidden }*/ div.DTTT_container { margin-left: 5px; margin-top: 2px } +.DTTT_button { + background: #fff !important; +} .shadow-border { border: 1px solid #d3d3d3; border-radius: 4px; @@ -100,9 +103,11 @@ div.DTTT_container { margin-left: 5px; margin-top: 2px } color: #fff; display: block; text-align: center; - border-radius: 11px; - margin: 5px; - width: 150px; + border-radius: 30px; + margin-left: 0px; + margin-right: 0px; + margin-top: 5px; + width: 125px; display: inline-block; line-height: 20px; cursor: pointer; @@ -144,7 +149,7 @@ div.DTTT_container { margin-left: 5px; margin-top: 2px } padding-right: 5px; cursor: pointer; } -.header-a:hover, #logo:hover, .status-button:hover * { +.header-a:hover, #logo:hover, .label-type:hover, .severity:hover, .status-button:hover * { color: #ffffff; text-decoration: none; -webkit-transition: 20ms linear 0s; @@ -152,11 +157,6 @@ div.DTTT_container { margin-left: 5px; margin-top: 2px } -o-transition: 20ms linear 0s; transition: 20ms linear 0s; outline: 0 none; -} -.header-a:hover, #logo:hover { - text-shadow: -1px -1px 0 rgba(255, 255, 255, 0.25), -1px 1px 0 rgba(255, 255, 255, 0.25), 1px 1px 0 rgba(255, 255, 255, 0.25), 1px -1px 0 rgba(255, 255, 255, 0.25); -} -.status-button:hover * { text-shadow: -1px -1px 0 rgba(0, 0, 0, 0.10), -1px 1px 0 rgba(0, 0, 0, 0.10), 1px 1px 0 rgba(0, 0, 0, 0.10), 1px -1px 0 rgba(0, 0, 0, 0.10); } .header-a:visited { @@ -298,12 +298,12 @@ rect.highlight { } .ui-widget-content { border: 1px solid #aaaaaa; - background: #ffffff url("../images/thirdparty/ui-bg_flat_75_ffffff_40x100.png") 50% 50% repeat-x; + background: #ffffff 50% 50% repeat-x; color: #222222; } .ui-widget-header { border: 1px solid #aaaaaa; - background: #cccccc url("../images/thirdparty/ui-bg_highlight-soft_75_cccccc_1x100.png") 50% 50% repeat-x; + background: #ccc 50% 50% repeat-x; color: #222222; font-weight: bold; } @@ -311,7 +311,7 @@ rect.highlight { .ui-widget-content .ui-state-default, .ui-widget-header .ui-state-default { border: 1px solid #d3d3d3; - background: #e6e6e6 url("../images/thirdparty/ui-bg_glass_75_e6e6e6_1x400.png") 50% 50% repeat-x; + background: #e6e6e6 50% 50% repeat-x; font-weight: normal; color: #555555; } @@ -322,7 +322,7 @@ rect.highlight { .ui-widget-content .ui-state-focus, .ui-widget-header .ui-state-focus { border: 1px solid #999999; - background: #dadada url("../images/thirdparty/ui-bg_glass_75_dadada_1x400.png") 50% 50% repeat-x; + background: #dadada 50% 50% repeat-x; font-weight: normal; color: #212121; } @@ -330,7 +330,7 @@ rect.highlight { .ui-widget-content .ui-state-active, .ui-widget-header .ui-state-active { border: 1px solid #aaaaaa; - background: #ffffff url("../images/thirdparty/ui-bg_glass_65_ffffff_1x400.png") 50% 50% repeat-x; + background: #ffffff 50% 50% repeat-x; font-weight: normal; color: #212121; } @@ -338,14 +338,14 @@ rect.highlight { .ui-widget-content .ui-state-highlight, .ui-widget-header .ui-state-highlight { border: 1px solid #fcefa1; - background: #fbf9ee url("../images/thirdparty/ui-bg_glass_55_fbf9ee_1x400.png") 50% 50% repeat-x; + background: #fbf9ee 50% 50% repeat-x; color: #363636; } .ui-state-error, .ui-widget-content .ui-state-error, .ui-widget-header .ui-state-error { border: 1px solid #cd0a0a; - background: #fef1ec url("../images/thirdparty/ui-bg_glass_95_fef1ec_1x400.png") 50% 50% repeat-x; + background: #fef1ec 50% 50% repeat-x; color: #cd0a0a; } .ui-icon, @@ -373,14 +373,14 @@ rect.highlight { background-image: url("../images/thirdparty/ui-icons_cd0a0a_256x240.png"); } .ui-widget-overlay { - background: #aaaaaa url("../images/thirdparty/ui-bg_flat_0_aaaaaa_40x100.png") 50% 50% repeat-x; + background: #aaaaaa 50% 50% repeat-x; opacity: .3; filter: Alpha(Opacity=30); /* support: IE8 */ } .ui-widget-shadow { margin: -8px 0 0 -8px; padding: 8px; - background: #aaaaaa url("../images/thirdparty/ui-bg_flat_0_aaaaaa_40x100.png") 50% 50% repeat-x; + background: #aaaaaa 50% 50% repeat-x; opacity: .3; filter: Alpha(Opacity=30); /* support: IE8 */ border-radius: 8px; @@ -483,15 +483,19 @@ rect.highlight { padding: 10px; z-index: -2; } -#table_hosts, #table_hosts th, #table_hosts td { +#table_aliases, #table_aliases th, #table_aliases td { border: 1px solid #a8a8a8; } -#table_hosts th { +#table_aliases th { color: white; } +#table_aliases td { + color: black; + text-shadow: none; +} #chart_area { background-color: #fcfcfc; - margin-bottom: 30px; + margin-bottom: 0; } .noselect { -webkit-touch-callout: none; @@ -553,8 +557,47 @@ rect.highlight { padding: 0 10px; width: 40%; } +#details_wrapper { + margin-top: 10px; +} input[type=search] { background: url("../images/search.png") no-repeat scroll 98% 50%; background-color: white; padding-right: 16px; } + +/* Reference: https://github.com/twbs/bootstrap/issues/16201#issuecomment-498358474 */ +@supports (-moz-appearance:none) { + #details_length select + { + -moz-appearance:none !important; + background: transparent url("../images/down.gif") right center no-repeat !important; + background-position: calc(100% - 5px) center !important; + line-height: inherit; + padding-right: 15px; + } +} + +/* Reference: https://stackoverflow.com/a/20471268 */ +.custom-menu { + display: none; + z-index: 1000; + position: absolute; + overflow: hidden; + border: 1px solid #CCC; + white-space: nowrap; + background: #FFF; + color: #333; + border-radius: 5px; + padding: 0; +} +.custom-menu li { + padding: 8px 12px; + cursor: pointer; + list-style-type: none; + transition: all .3s ease; + user-select: none; +} +.custom-menu li:hover { + background-color: #DEF; +} diff --git a/html/css/media.css b/html/css/media.css index 4e57ec1b999..17ab811ac3a 100644 --- a/html/css/media.css +++ b/html/css/media.css @@ -1,15 +1,22 @@ -@media screen and (max-width: 900px) { - #status_container { +@media screen and (max-width: 680px) { + #status_container, #documentation_link, #collaboration_link, #issues_link, .link-splitter, .DTTT_container { display: none !important; } } -@media screen and (max-width: 650px) { - .DTTT_container, #details_info { + +@media screen and (max-width: 600px) { + #details_info { display: none !important; } -} -@media screen and (max-width: 460px) { - #details_length, #documentation_link, #issues_link, .link-splitter { + #details_paginate, #details_filter, #details_filter label { + text-align: center; + float: unset; + } + #details_length { display: none !important; } -} \ No newline at end of file + + td:nth-child(1), td:nth-child(2), td:nth-child(3), td:nth-child(5), td:nth-child(6), td:nth-child(9), td:nth-child(11), td:nth-child(12), td:last-child, th:nth-child(1), th:nth-child(2), th:nth-child(3), th:nth-child(5), th:nth-child(6), th:nth-child(9), th:nth-child(11), th:nth-child(12), th:last-child { + display: none; + } +} diff --git a/html/images/down.gif b/html/images/down.gif new file mode 100644 index 00000000000..87767b8955c Binary files /dev/null and b/html/images/down.gif differ diff --git a/html/index.html b/html/index.html index c9208eed41e..1d25d198ef3 100644 --- a/html/index.html +++ b/html/index.html @@ -9,34 +9,25 @@ - + -
- +
-
+