WatchAD, open source code. beta

README.md

@@ -1,6 +1,6 @@
 # WatchAD
 
-[![PyPI version](https://img.shields.io/badge/Python-3.6+-blue.svg)](http://git.websec.cc/zhusiyu/dc_log_analyze) [![ElasticSearch version](https://img.shields.io/badge/ElasticSearch-5.X-success.svg)](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/index.html) [![Logstash version](https://img.shields.io/badge/Logstash-6.X-yellowgreen.svg)](https://www.elastic.co/guide/en/logstash/6.2/index.html) [![RabbitMQ version](https://img.shields.io/badge/RabbitMQ-3.7-orange.svg)](https://www.rabbitmq.com/) [![DEF CON 27 Blue Team Village](https://img.shields.io/badge/DEF%20CON%2027-Blue%20Team%20Village-blue.svg)](https://www.blueteamvillage.org/home/dc27/talks#h.p_5uroKErLDdmP)
+![PyPI version](https://img.shields.io/badge/Python-3.6+-blue.svg) [![ElasticSearch version](https://img.shields.io/badge/ElasticSearch-5.X-success.svg)](https://www.elastic.co/guide/en/elasticsearch/reference/5.2/index.html) [![Logstash version](https://img.shields.io/badge/Logstash-6.X-yellowgreen.svg)](https://www.elastic.co/guide/en/logstash/6.2/index.html) [![RabbitMQ version](https://img.shields.io/badge/RabbitMQ-3.7-orange.svg)](https://www.rabbitmq.com/) [![DEF CON 27 Blue Team Village](https://img.shields.io/badge/DEF%20CON%2027-Blue%20Team%20Village-blue.svg)](https://www.blueteamvillage.org/home/dc27/talks#h.p_5uroKErLDdmP)
 
 > 域安全入侵感知系统
 
@@ -21,18 +21,38 @@ WatchAD收集所有域控上的事件日志和kerberos流量，通过特征匹
 
 ## 安装部署
 
-WatchAD是一个完整的检测系统，涉及的内容较多，请参考 [INSTALL.md](https://github.com/0Kee-Team/WatchAD/wiki/Install) 进行安装。
+WatchAD是一个完整的检测系统，涉及的内容较多，请参考 [安装教程](https://github.com/0Kee-Team/WatchAD/wiki/Install) 进行安装。如果你需要设置蜜罐账户，也可以参考我们关于蜜罐账户的[说明](https://github.com/0Kee-Team/WatchAD/wiki/Honeypot-Account)。
 
 项目架构简图：
 
 ![Architecture](./images/Architecture.png)
 
-本项目 WatchAD 只包含了检测引擎相关的代码，Web平台的前后端代码在项目 [WatchAD-Web](https://github.com/0Kee-Team/WatchAD-Web) 中。
+本项目 WatchAD 只包含了检测引擎相关的代码，你可以选择直接将告警数据导入运营中心统一管理，或者使用我们开发的Web平台 [WatchAD-Web](https://github.com/0Kee-Team/WatchAD-Web) ，它是一个为WatchAD定制的简易平台，可进行简单的运营工作，如果你对界面设计或者操作体验有更高的要求，请根据WatchAD的告警数据格式自行定制化开发。
 
 ## 自定义检测模块
 
 WatchAD支持开发自定义的检测模块，详情请参考我们的[教程](https://github.com/0Kee-Team/WatchAD/wiki/Development)。
 
+如果你不需要某个**检测模块**，直接将**该py文件删除**并重启检测引擎即可，无需其它配置。`record`目录中的文件不参与告警检测，只负责记录域内实体的关键活动，请勿删除。
+
+## // TODO
+
+- English Document
+- ElasticSearch兼容6.X
+- 各个检测模块的误报持续优化
+- Kerberoasting：基于事件日志检测的代码被流量替代，后续可添加
+- Pass-the-Hash(PtH)：内部存在误报，暂缓开源
+- Pass-the-Ticket(PtT)：内部存在误报，暂缓开源
+- Silver-Ticket：内部存在误报，暂缓开源
+- 伪造用户信息：内部存在误报，暂缓开源
+- 基于用户行为的失陷账户检测
+- 基于Kerberos流量的分析开源
+- NTLM流量分析
+
+如果你有认为需要加入到WatchAD检测的攻击方法，请提issue告诉我们相关复现方式，或者提交PR成为本项目的贡献者。
+
+如果你发现某个检测模块有较多的误报（日均超过**10**条），请提issue告诉我们或由你优化之后提交PR。
+
 ## Follow me
 
 微博： [@9ian1i](https://weibo.com/u/5242748339)

WatchAD.py

@@ -0,0 +1,149 @@
+#!/usr/bin/python3
+# coding: utf-8
+# author: 9ian1i   https://github.com/Qianlitp
+
+"""
+    install     安装ES索引模板,初始化LDAP配置
+    check       检查各个数据库连接状态、消息队列状态
+    start       加载动态配置信息、创建计划任务、启动检测引擎
+    restart     重新加载动态配置信息、删除计划任务、重启检测引擎
+    stop        停止引擎 （删除现有消息队列，防止数据量过大造成积压）
+    status      查看当前引擎状态
+"""
+
+
+from io import StringIO
+import optparse
+import sys
+from _project_dir import project_dir
+import subprocess
+from tools.common.Logger import logger
+from scripts.init_settings import init_es_template, check_es_template, check_mongo_connection, check_mq_connection, \
+    init_ldap_settings, init_default_settings, get_all_dc_names, set_learning_end_time_setting, init_sensitive_groups, \
+    set_crontab_tasks
+
+
+def install(domain, server, user, password):
+    logger.info("Install the WatchAD ...")
+    # 初始化ES索引模板
+    init_es_template()
+    # 初始化LDAP配置信息
+    init_ldap_settings(domain, server, user, password)
+    # 获取域控计算机名保存入库
+    get_all_dc_names(domain)
+    # 初始化其余配置信息
+    init_default_settings(domain)
+    # 初始化填入敏感用户组
+    init_sensitive_groups(domain)
+    # 根据当前安装时间，设置数据统计结束时间
+    set_learning_end_time_setting()
+    # 设置计划任务
+    set_crontab_tasks()
+
+
+def check() -> bool:
+    logger.info("Checking the WatchAD environment ...")
+    # 检查ES模板安装状态
+    if not check_es_template():
+        return False
+    # 检查数据库连接
+    if not check_mongo_connection():
+        return False
+    # 检查消息队列连接
+    if not check_mq_connection():
+        return False
+    logger.info("OK!")
+    logger.info("Check the WatchAD environment successfully!")
+    return True
+
+
+def start():
+    if not check():
+        sys.exit(-1)
+    logger.info("Starting the WatchAD detect engine ...")
+
+    rsp = subprocess.call("supervisord -c {root_dir}/supervisor.conf".format(root_dir=project_dir),
+                          shell=True,
+                          env={"WATCHAD_ENGINE_DIR": project_dir, "ENV_WATCHAD_ENGINE_NUM": 5})
+    if rsp == 0:
+        logger.info("Started!")
+    else:
+        logger.error("Start failed.")
+
+
+def stop():
+    logger.info("Stopping the WatchAD detect engine ...")
+
+    rsp = subprocess.call("supervisorctl -c {root_dir}/supervisor.conf shutdown".format(root_dir=project_dir),
+                          shell=True,
+                          env={"WATCHAD_ENGINE_DIR": project_dir, "ENV_WATCHAD_ENGINE_NUM": 5})
+
+    if rsp == 0:
+        logger.info("Stopped!")
+    else:
+        logger.error("Stop failed.")
+
+
+def restart():
+    stop()
+    start()
+
+
+def status():
+    subprocess.call("supervisorctl -c {root_dir}/supervisor.conf status".format(root_dir=project_dir),
+                    shell=True,
+                    env={"WATCHAD_ENGINE_DIR": project_dir})
+
+
+def usage():
+    s = StringIO()
+    s.write("Usage:  WatchAD.py <options> [settings]")
+    s.seek(0)
+    return s.read()
+
+
+def parse_option():
+    parser = optparse.OptionParser(usage=usage())
+    parser.add_option("--install", action="store_true", dest="install", help="执行WatchAD初始化安装，在次之前请确保已完整环境安装和配置。")
+    parser.add_option("-d", "--domain", action="store", dest="domain", help="A FQDN domain name of detection.")
+    parser.add_option("-s", "--ldap-server", action="store", dest="server",
+                      help="Server address for LDAP search. e.g: dc01.corp.com")
+    parser.add_option("-u", "--domain-user", action="store", dest="username",
+                      help="Username for LDAP search. e.g: CORP\\peter")
+    parser.add_option("-p", "--domain-passwd", action="store", dest="password",
+                      help="Password for LDAP search.")
+    parser.add_option("--check", action="store_true", dest="check", help="检查各个数据库连接状态、消息队列状态")
+    parser.add_option("--start", action="store_true", dest="start", help="启动检测引擎")
+    parser.add_option("--restart", action="store_true", dest="restart", help="重新加载动态配置信息、删除计划任务、重启检测引擎")
+    parser.add_option("--stop", action="store_true", dest="stop", help="停止引擎 （删除现有消息队列，防止数据量过大造成积压）")
+    parser.add_option("--status", action="store_true", dest="status", help="查看当前引擎状态")
+    return parser
+
+
+def main():
+    parser = parse_option()
+    if len(sys.argv) < 2:
+        logger.error("WatchAD must run with an action.")
+        parser.print_help()
+        sys.exit(1)
+    options, args = parser.parse_args()
+
+    if options.install:
+        if not options.domain or not options.server or not options.username or not options.password:
+            logger.error("WatchAD install action must provide domain, server, user and password params.")
+            sys.exit(1)
+        install(domain=options.domain, server=options.server, user=options.username, password=options.password)
+    elif options.check:
+        check()
+    elif options.start:
+        start()
+    elif options.restart:
+        restart()
+    elif options.stop:
+        stop()
+    elif options.status:
+        status()
+
+
+if __name__ == '__main__':
+    main()

_project_dir.py

@@ -0,0 +1,7 @@
+#!/usr/bin/python3
+# coding: utf-8
+# author: 9ian1i   https://github.com/Qianlitp
+
+import os
+
+project_dir = os.path.dirname(os.path.abspath(__file__))

docker-compose.yaml

@@ -0,0 +1,61 @@
+version: '3'
+services:
+  watchad_rabbitmq:
+    image: rabbitmq:management
+    ports:
+    - "15672:15672"
+    - "5672:5672"
+    environment:
+      RABBITMQ_IO_THREAD_POOL_SIZE: 100
+      RABBITMQ_HIPE_COMPILE: "true"
+      RABBITMQ_DEFAULT_USER: WatchAD
+      RABBITMQ_DEFAULT_PASS: WatchAD-by-0KEE
+    restart: always
+
+  watchad_logstash:
+    image: logstash:6.4.1
+    ports:
+      - "5044:5044"
+    environment:
+      XPACK_MONITORING_ENABLED: "false"
+      pipeline.batch.size: 10
+    volumes:
+      - ./settings/logstash/:/usr/share/logstash/pipeline/
+    links:
+      - watchad_elasticsearch
+    restart: always
+
+  watchad_elasticsearch:
+    image: elasticsearch:5.2.1
+    environment:
+      http.host: 0.0.0.0
+      transport.host: 127.0.0.1
+      cluster.name: docker-cluster
+      bootstrap.memory_lock: "true"
+      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
+      thread_pool.bulk.size: 9
+      thread_pool.bulk.queue_size: 1000
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
+    ports:
+      - "9200:9200"
+    restart: always
+
+  watchad_redis:
+    image: redis:latest
+    ports:
+      - "6379:6379"
+    restart: always
+
+  watchad_mongo:
+    image: mongo:latest
+    ports:
+      - "27017:27017"
+    environment:
+      MONGO_INITDB_ROOT_USERNAME: WatchAD
+      MONGO_INITDB_ROOT_PASSWORD: WatchAD-by-0KEE

libs/kek/_crypto/ARC4.py

@@ -0,0 +1,24 @@
+class ARC4Cipher(object):
+     def __init__(self, key):
+         self.key = key
+
+     def encrypt(self, data):
+         S = range(256)
+         j = 0
+         out = []
+         for i in range(256):
+             j = (j + S[i] + ord( self.key[i % len(self.key)] )) % 256
+             S[i] , S[j] = S[j] , S[i]
+         i = j = 0
+         for char in data:
+             i = ( i + 1 ) % 256
+             j = ( j + S[i] ) % 256
+             S[i] , S[j] = S[j] , S[i]
+             out.append(chr(ord(char) ^ S[(S[i] + S[j]) % 256]))
+         return ''.join(out)
+
+     def decrypt(self, data):
+         return self.encrypt(data)
+
+def new(key):
+    return ARC4Cipher(key)

libs/kek/_crypto/MD4.py

@@ -0,0 +1,4 @@
+import hashlib
+
+def new(*args):
+    return hashlib.new('md4', *args)

libs/kek/_crypto/MD5.py

@@ -0,0 +1,4 @@
+import hashlib
+
+def new(*args):
+    return hashlib.md5(*args)