Update old Data Needed documents to the new structure

README.md

@@ -1,101 +1,4 @@
-# atc-data
-
-![](docs/images/logo_v2.png)
-
-The atc-data is a community-driven project designed to accumulate and describe specific data that is required by Security Operations, such as Threat Detection/Hunting and Incident Response.  
-
-It includes a description of event logs, network telemetry, data lists, and so on. And with that — a detailed description of what has to be configured and how the data has to be processed to be used in the Security Operations.  
-
-The main advantage of the project is a clear, exact definition of where specific data is required, whether it's a Detection Rule, Response Action, or Visualisation.  
-
-The main use cases:
-
-- Data collection prioritization. And with that — Threat Detection/Hunting and Incident Response capabilities development
-- Gap analysis — determine "coverage" of existing Threat Detection/Hunting and Incident Response capabilities, depending on data collected
-
-The main resources:
-
-- Automatically generated atc-data [website](https://atc-project.github.io/atc-react/) is the best place for getting details about existing analytics  
-- Automatically generated [Atlassian Confluence knowledge base](https://atomicthreatcoverage.atlassian.net/wiki/spaces/REACT/pages/755469668/Response+Stages) - exporting functionality demonstration  
-
-## Actionable Analytics
-
-The ATC RE&CT project inherits the "Actionable Analytics" paradigm from the [ATC](https://github.com/atc-project/atomic-threat-coverage) project, which means that the analytics are:
-
-- **human-readable** (`.md`) for sharing/using in operations
-- **machine-readable** (`.yml`) for automatic processing/integrations
-- **executable** by Incident Response Platform ([TheHive Case Templates](docs/thehive_templates/) only, at the moment)
-
-Simply saying, the analytics are stored in `.yml` files, that are automatically converted to `.md` documents (with [jinja](https://palletsprojects.com/p/jinja/)) and `.json` TheHive Case Templates.  
-
-### Data Needed
-
-to be collected to produce detection of specific Threat
-
-This entity expected to simplify communication with SIEM/LM/Data Engineering teams. It includes the next data:
-
-- Sample of the raw log to describe what data they could expect to receive/collect
-- Description of data to collect (Platform/Type/Channel/etc) — needed for calculation of mappings to Detection Rules and general description
-- List of fields also needed for calculation of mappings to Detection Rules and Response Playbooks, as well as for `pivoting.csv` generation
-
-Response Action is a description of a specific atomic procedure/task that has to be executed during the Incident Response. It is an initial entity that is used to construct Response Playbooks.  
-
-Here is an example of Response Action:
-
-<details>
-  <summary>Initial YAML file (click to expand)</summary>
-  <img src="docs/images/ra_yaml_v5.png" />
-</details>
-
-- Automatically created [Markdown file](docs/Response_Actions/RA_2202_collect_email_message.md)
-- Automatically created [mkdocs web page](https://atc-project.github.io/atc-react/Response_Actions/RA_2202_collect_email_message/)
-- Automatically created [Confluence page](https://atomicthreatcoverage.atlassian.net/wiki/spaces/REACT/pages/755435640/RA2202+Collect+email+message)
-
-The categorization aims to improve Incident Response process maturity assessment and roadmap development.  
-
-### Logging Policies
-
-need to be configured on data source to be able to collect Data Needed
-
-This entity expected to explain SIEM/LM/Data Engineering teams and IT departments which logging policies have to be configured to have proper Data Needed for Detection and Response to specific Threat. It also explains how exactly this policy can be configured.
-
-### Enrichments
-
-for specific Data Needed which required for some Detection Rules
-
-This entity expected to simplify communication with SIEM/LM/Data Engineering teams. It includes the next data:
-
-- List of Data Needed which could be enriched
-- Description of the goal of the specific Enrichment (new fields, translation, renaming etc)
-- Example of implementation (for example, Logstash config)
-
-This way you will be able to simply explain why you need specific enrichments (mapping to Detection Rules) and specific systems for data enrichment (for example, Logstash).
-
-#### pivoting.csv
-
-The atc-data generates [pivoting.csv](docs/pivoting.csv) with a list of all fields (from Data Needed) mapped to description of Data Needed for very specific purpose — it provides information about data sources where some specific data type could be found, for example domain name, username, hash etc:
-
-<details>
-  <summary>Example of lookup for "hash" field (click to expand)</summary>
-  <img src="images/pivoting_hash_v1.png" />
-</details>
-
-<br>
-
-At the same time it highlights which fields could be found only with specific enrichments:
-
-<details>
-  <summary>Example of lookup for "ParentImage" field (click to expand)</summary>
-  <img src="images/pivoting_parent_v1.png" />
-</details>
-
-### Requirements
-
-- Python 3.7
-- [PyYAML](https://pypi.org/project/PyYAML/), [mkdocs](https://pypi.org/project/mkdocs/) and [jinja2](https://pypi.org/project/Jinja2/) Python libraries. They could be installed with the following command:
-    ```
-    python3 -m pip install -r requirements.txt
-    ```
+# atc-data (WORK IN PROGRESS)
 
 ## Contacts
 

atc-data-new-structure/data/av/DN_0084_av_alert.yml

@@ -0,0 +1,67 @@
+title: DN_0084_av_alert
+author: '@atc_project'
+description: >
+  Anti-virus alert
+loggingpolicy:
+  - None          # some of the AVs require additional configuraiotn to log filehash
+references:
+  - None
+category: AV Alerts
+platform: antivirus
+type: None
+channel: None
+provider: None
+fields:
+  - Hostname
+  - Signature
+  - AlertTitle
+  - Category
+  - Severity
+  - Sha1
+  - FileName
+  - FilePath
+  - IpAddress
+  - UserName
+  - UserDomain
+  - FileHash
+  - Hashes
+  - Imphash
+  - Sha256hash
+  - Sha1hash
+  - Md5hash
+sample: |
+  {
+    "AlertTime":"2017-01-23T07:32:54.1861171Z",
+    "ComputerDnsName":"desktop-bvccckk",
+    "AlertTitle":"Suspicious PowerShell commandline",
+    "Category":"SuspiciousActivity",
+    "Severity":"Medium",
+    "AlertId":"636207535742330111_-1114309685",
+    "Actor":null,
+    "LinkToWDATP":"https://securitycenter.windows.com/alert/636207535742330111_-1114309685",
+    "IocName":null,
+    "IocValue":null,
+    "CreatorIocName":null,
+    "CreatorIocValue":null,
+    "Sha1":"69484ca722b4285a234896a2e31707cbedc59ef9",
+    "FileName":"powershell.exe",
+    "FilePath":"C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0",
+    "IpAddress":null,
+    "Url":null,
+    "IoaDefinitiondId":"7f1c3609-a3ff-40e2-995b-c01770161d68",
+    "UserName":null,
+    "AlertPart":0,
+    "FullId":"636207535742330111_-1114309685:9DE735BA9FF87725E392C6DFBEB2AF279035CDE229FCC00D28C0F3242C5A50AF",
+    "LastProcessedTimeUtc":"2017-01-23T11:33:45.0760449Z",
+    "ThreatCategory":null,
+    "ThreatFamily":null,
+    "ThreatName":null,
+    "RemediationAction":null,
+    "RemediationIsSuccess":null,
+    "Source":"Windows Defender ATP",
+    "Md5":null,
+    "Sha256":null,
+    "WasExecutingWhileDetected":null,
+    "FileHash":"69484ca722b4285a234896a2e31707cbedc59ef9",
+    "IocUniqueId":"9DE735BA9FF87725E392C6DFBEB2AF279035CDE229FCC00D28C0F3242C5A50AF"
+  }

atc-data-new-structure/data/dataneeded.yml.template

@@ -0,0 +1,62 @@
+title: 'Human readable title, ideally — from the official documentation, without EventID if present' # ATC will automatically add EventID (if present) to the beginning of the title if present
+description: Human readable description, ideally — from the official documentation.
+event_id: 4688                                # [optional] ATC will automatically add it to the beginning of the title if present
+attack_data:                                  # [optional] list of sources (key) and components (value); 
+  - process: process creation                 # here is the list of options: https://github.com/mitre-attack/attack-datasources/blob/main/attack_data_sources.yaml
+platform: windows                             # linux | unix | macos | network | etc
+provider: Microsoft-Windows-Security-Auditing # Microsoft-Windows-Eventlog | BIND | <exact service/deamon name> | None
+channel: Security                             # [optional] System | Microsoft-Windows-Sysmon/Operational | queries_log | None
+atc_id: DN0001                                # Counting number (for now)
+loggingpolicy:                                # [optional] ATC Logging Policy ID
+  - LP0001: Success                           # [optional] ATC Logging Policy ID with audit Success/Failure as a value
+  - LP0002: [ "Success", "Failure" ]          # [optional] ATC Logging Policy ID with both audit Success and Failure
+  - LP0003                                    # [optional] Could be just ATC Logging Policy ID
+contributors:
+  - 'your name/nickname/twitter'
+references:
+  - text: 'MicrosoftDocs: 4688(S): A new process has been created'
+    link: 'https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-4688.md'
+fields:
+
+  - original_name: EventID                # Original value from the data source
+    description: The event identifier.    # A good source of the descripotion is elastic *beats "fields.yml" file. download required: https://www.elastic.co/downloads/beats
+    sample_value: '4688'
+    elastic_ecs_name: winlog.event_id     # Elastic Common Schema name. Source: https://github.com/elastic/ecs/blob/master/generated/csv/fields.csv ; if not present, check elastic *beats "fields.yml" file
+    splunk_cim_name: EventCode            # Splunk CIM name. Source: https://docs.splunk.com/Documentation/CIM/
+    otr_ossem_name: event_id              # OTR OSSEM name. Source: https://github.com/OTRF/OSSEM
+
+sample: | #raw log sample here
+  - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
+    - <System>
+      <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
+      <EventID>4688</EventID>
+      <Version>2</Version>
+      <Level>0</Level>
+      <Task>13312</Task>
+      <Opcode>0</Opcode>
+      <Keywords>0x8020000000000000</Keywords>
+      <TimeCreated SystemTime="2015-11-12T02:24:52.377352500Z" />
+      <EventRecordID>2814</EventRecordID>
+      <Correlation />
+      <Execution ProcessID="4" ThreadID="400" />
+      <Channel>Security</Channel>
+      <Computer>WIN-GG82ULGC9GO.contoso.local</Computer>
+      <Security />
+    </System>
+    - <EventData>
+      <Data Name="SubjectUserSid">S-1-5-18</Data>
+      <Data Name="SubjectUserName">WIN-GG82ULGC9GO$</Data>
+      <Data Name="SubjectDomainName">CONTOSO</Data>
+      <Data Name="SubjectLogonId">0x3e7</Data>
+      <Data Name="NewProcessId">0x2bc</Data>
+      <Data Name="NewProcessName">C:\\Windows\\System32\\rundll32.exe</Data>
+      <Data Name="TokenElevationType">%%1938</Data>
+      <Data Name="ProcessId">0xe74</Data>
+      <Data Name="TargetUserSid">S-1-5-21-1377283216-344919071-3415362939-1104</Data>
+      <Data Name="TargetUserName">dadmin</Data>
+      <Data Name="TargetDomainName">CONTOSO</Data>
+      <Data Name="TargetLogonId">0x4a5af0</Data>
+      <Data Name="ParentProcessName">C:\\Windows\\explorer.exe</Data>
+      <Data Name="MandatoryLabel">S-1-16-8192</Data>
+    </EventData>
+  </Event>

atc-data-new-structure/data/linux/file/DN_0055_linux_auditd_read_access_to_file.yml

@@ -0,0 +1,34 @@
+title: DN_0055_linux_auditd_read_access_to_file
+author: '@atc_project'
+description: >
+  Linux auditd log of read access to file
+loggingpolicy:
+  - LP_0034_linux_auditd_read_access_to_file
+references:
+  - https://github.com/linux-audit/audit-documentation
+  - https://github.com/linux-audit/audit-documentation/blob/master/specs/fields/field-dictionary.csv
+  - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/app-audit_reference
+category: OS Logs
+platform: Linux
+type: PATH
+channel: auditd
+provider: auditd
+fields:
+  - type     # the audit record's type
+  - msg      # the payload of the audit record
+  - item     # which item is being recorded
+  - name     # file name in avcs
+  - inode    # inode number
+  - dev      # device name as found in /dev
+  - mode     # mode flags on a file
+  - ouid     # file owner user ID
+  - ogid     # file owner group ID
+  - rdev     # the device identifier (special files only)
+  - obj      # lspp object context string
+  - objtype  # object in the context of a syscall
+  - cap_fp   # file permitted capability map
+  - cap_fi   # file inherited capability map
+  - cap_fe   # file assigned effective capability map
+  - cap_fver # file system capabilities version number
+sample: |
+  type=PATH msg=audit(1564423065.282:742): item=0 name="/etc/passwd" inode=24673227 dev=fd:00 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:passwd_file_t:s0 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0

atc-data-new-structure/data/linux/multiple/DN_0056_linux_auditd_syscall.yml

@@ -0,0 +1,48 @@
+title: DN_0056_linux_auditd_syscall
+author: '@atc_project'
+description: >
+  Linux auditd log of specific system call (syscall)
+loggingpolicy:
+  - LP_0033_linux_auditd_syscall
+references:
+  - https://github.com/linux-audit/audit-documentation
+  - https://github.com/linux-audit/audit-documentation/blob/master/specs/fields/field-dictionary.csv
+  - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/app-audit_reference
+  - https://access.redhat.com/solutions/36278
+  - https://filippo.io/linux-syscall-table/
+category: OS Logs
+platform: Linux
+type: SYSCALL
+channel: auditd
+provider: auditd
+fields:
+  - type    # the audit record's type
+  - msg     # the payload of the audit record
+  - arch    # the elf architecture flags
+  - syscall # syscall number in effect when the event occurred
+  - success # whether the syscall was successful or not
+  - exit    # syscall exit code
+  - a0      # argument of the system call, encoded in hexadecimal notation
+  - a1      # argument of the system call, encoded in hexadecimal notation
+  - a2      # argument of the system call, encoded in hexadecimal notation
+  - a3      # argument of the system call, encoded in hexadecimal notation
+  - items   # the number of path records in the event
+  - ppid    # parent process ID
+  - pid     # process ID
+  - auid    # login user ID
+  - uid     # real user ID of the user who started the analyzed process
+  - gid     # group ID
+  - euid    # effective user ID
+  - suid    # set user ID of the user who started the analyzed process
+  - fsuid   # file system user ID of the user who started the analyzed process
+  - egid    # effective group ID of the user who started the analyzed process
+  - sgid    # set group ID of the user who started the analyzed process
+  - fsgid   # file system group ID of the user who started the analyzed process
+  - tty     # name of the controlling terminal. The value (none) is used if the process has no controlling terminal
+  - ses     # login session ID
+  - comm    # command line program name
+  - exe     # executable name
+  - subj    # SELinux context with which the analyzed process was labeled at the time of execution
+  - key     # administrator-defined string associated with the rule that generated this event in the Audit log
+sample: |
+  type=SYSCALL msg=audit(1529507591.700:304): arch=c000003e syscall=62 success=yes exit=0 a0=829 a1=9 a2=0 a3=829 items=0 ppid=1783 pid=1784 auid=1001 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="bash" exe="/usr/bin/bash" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="kill_rule"

atc-data-new-structure/data/linux/process/DN_0054_linux_auditd_execve.yml

@@ -0,0 +1,26 @@
+title: DN_0054_linux_auditd_execve
+author: '@atc_project'
+description: >
+  Linux auditd log of process (binary) execution (execeve syscall)
+  with command line arguments
+loggingpolicy:
+  - LP_0031_linux_auditd_execve
+references:
+  - https://github.com/linux-audit/audit-documentation
+  - https://github.com/linux-audit/audit-documentation/blob/master/specs/fields/field-dictionary.csv
+  - https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/app-audit_reference
+category: OS Logs
+platform: Linux
+type: EXECVE
+channel: auditd
+provider: auditd
+fields:
+  - type # the audit record's type
+  - msg  # the payload of the audit record
+  - argc # the number of arguments to an execve syscall
+  - a0   # a[[:digit:]+]\[.*\] — the arguments to the execve syscall
+  - a1
+  - a2
+  - a3
+sample: |
+  type=EXECVE msg=audit(1564425065.452:651): argc=3 a0="ls" a1="-l" a2="/var/lib/pgsql"

atc-data-new-structure/data/linux/uncategorised/DN_0091_linux_modsecurity_log.yml

@@ -0,0 +1,21 @@
+title: DN_0091_linux_modsecurity_log
+author: '@atc_project'
+description: >
+  Mod_security (Web Application Firewall) audit/error log
+loggingpolicy:
+  - None
+references:
+  - https://www.nginx.com/blog/modsecurity-logging-and-debugging/
+  - https://www.cryptobells.com/mod_security-json-audit-logs-revisited/
+category: OS Logs
+platform: Linux
+type: modsecurity
+channel: modsecurity
+provider: modsecurity
+fields:
+  - timestamp
+  - hostname
+  - client
+  - uri
+sample: |
+  [Thu Jul 02 04:14:31 2018] [error] [client 190.222.135.100] mod_security: Access denied with code 500. Pattern match "SomePattern" at HEADER("USER-AGENT") [hostname "samplesite.com"] [uri "/some/uri"]

atc-data-new-structure/data/linux/uncategorised/DN_0093_linux_clamav_log.yml

@@ -0,0 +1,22 @@
+title: DN_0093_linux_clamav_log
+author: '@atc_project'
+description: >
+  Linux ClamAV anti-virus logs
+loggingpolicy:
+  - None
+references:
+  - https://www.clamav.net
+  - https://docs.pivotal.io/addon-antivirus/1-4/monitoring-logs.html
+  - https://github.com/ossec/ossec-hids/blob/master/etc/rules/clam_av_rules.xml
+category: AV Alerts
+platform: Linux
+type: None
+channel: ClamAV # required for parsing
+provider: ClamAV
+fields:
+  - Hostname
+  - Signature
+  - FileName
+  - FilePath
+sample: |
+  /var/vcap/data/test.txt: Eicar-Test-Signature FOUND