Maintenance and bug fixes

Alerts

• ./run all now runs data connectors, then violations, then alerts
• deprecated baseline runners and scripts are removed (cf3bc6d, 972c5cd)
• a rule may now declare a single handler without wrapping it in an array (972c5cd)

Data Connectors

• fix bug in AWSIC running on latest EKS in #425 (ty @edulop91)
• AWSIC now records config describe-configuration-recorders results for all available regions (2844b94)
• AWSIC now respects AWS rate limits for Get requests (2844b94)
• AWSIC now handles ServerTimeout errors (no response in 60 seconds) gracefully (2cf48eb)
• fix bug in Jira correlation logic
• add custom Jira starting status via environment variable JIRA_STARTING_STATUS
• JAMF and AzIC scheduling code are moved to generic system via table comments
• fix Azure log to work for with additional log types in 947c394 (ty for #414 @Chaitali-Sonparote)
• minor cleanup of AzIC in 5ccc0f4
• minor fix from gsuite API change e8a58e5
• Okta connector can now use a custom domain and includes a pack for initial data cleanup
• Jamf now handles large inventory sizes better in 5e55b8e

Packs

• Bug fix in snowflake_security_monitoring in a3ad191 (ty to Intact Financial Corporation for the report & fix)
• Basic Okta structures around ingested data (b05fa92)

Handlers

• fix bug in Jira correlation logic #424
• rules can now send arbitrary payloads to ServiceNow handler in bbbb4c2
• Jira handler works with single string source as well as list of multiple sources, and can now link alerts types to a triage repository (2d345aa)
• SMTP handler can now pass host, user, port, and password as params (d452139)

WebUI

• fix minor UX bugs and bump dependencies with security detections (425cdb6)

Deprecate Ingestion Scripts

• ZenGRC is decommissioned and will be re-introduced as a pack in a future version (#436)
• Agari have been decommissioned without plans for re-introduction (79c3702)

New ServiceNow Handler, Duo Data Connector, CIS Rules, and more

Packs

• minor cleanup in AWS CIS 1.1, 1.13, and 1.12
• added Azure CIS 1.1, 1.2, 3.3, 4.1, 7.3, 7.4, 8.1, and 8.2
• fixed errors in Azure CIS 7.1, 7.2

Data Connectors (DC)

• new Duo Admins Inventory Connector
• DC schedule can now be generally specified in any connector comment
• multiple DC's can now be scheduled to run from one CLI command
• improvement to make Azure log connector more robust to different log types (#414)

Azure Inventory and Configuration (AzIC) Connector

• adds groups_members, role_assignments, queue_services, queue_services_properties, sql_servers, and sql_servers_auditing_settings collection
• includes updated values from new Graph API groups, role_definitions, and service_principals endpoints
• adds mechanism to save arbitrary values as API's change
• fixed GovCloud authentication bug
• fixes minor misnamed columns
• greatlyimproves runtime and reliability

AWS Inventory and Configuration (AWSIC) Connector

• adds iam_list_groups, iam_list_attached_group_policies, and ec2_describe_route_tables tables
• adds error column for tracking failed API responses as in AzIC
• removes vestigial region columns from tables that did not end up populating them (per boto3 client's describe_regions)
• improves error handling and logging in API retries
• fixes session expiration errors

AWS CloudTrail Connector

• fixes timezone translation bug in accounts with default LTZ set to zone other than UTC (#416)

Alert Runner and Processor

• adds support to run multiple alerts from CLI (#413)
• adds FROM_TIME env variable that can be used to specify alerting period explicitly instead of relative to the end time (#416)
• fixes alert deduplication logic bug
• fixes logging on invalid credentials (#379)
• fixes handling of to alert queries with lists in actor field

WebUI

• adds custom db / warehouse / role so a single WebUI deployment can support multiple SnowAlert installations
• fix buggy data connector form validation
• dependency updates

Handlers

• added ServiceNow handler
• added SQL-based blocks to Slack handler (making UDF use optional)
• fixed Slack handler exception handling (#401)

Scripts & minor fixes

• more robust Jira bulk change script
• pyYAML and pandas vuln updates
• explicitly empty default region sets to default

External Contributors

Thanks to @bhasampa, @carolinepotts, @Chaitali-Sonparote, @mikeurbanski1, and @GalGreenfield for all your great and minor contributions to this release!

Azure CIS, Data Connectors, and Whitesource

Alert Query Runners

• fixes bug which broke single-slice deduplication (thanks, @ mikeurbanski1!)

Query Packs

• adds Azure CIS Rules from sections 2, 3, 5, 6, and rules 7.1 + 7.2

Data Connectors

• adds "Diagnostic Settings", "Activity Log Alerts", "VM Instance View", "VM Extensions" and updated fields to existing Azure Inventory & Configuration Connector
• fixes bug in Azure Log Connector mis-handling JSON decode errors
• adds error logging to AWS I&C Connector
• fixes typo in AWS I&C Connector (thanks for noticing this, @Chaitali-Sonparote!)

Handlers

• adds new Pager Duty handler added in 2f2581b (thanks, @olegg!)
• improves Jira handler with per-alert Jira ticket type setting in 02d3ac2
• fies Slack template selector not using params (thanks, @sfc-gh-anezvigin!)

WebUI

• adds custom "db role" option when logging into WebUI
• updates WebUI dependencies and cleans up related tech debt

Misc

• fixes bug in Baselines with "no-zeros" option
• add Whitesource for dependency security scanning
• deletes Okta ingest script

Thanks

Thanks to Mike Urbanski at People.ai, our own @sfc-gh-anezvigin, and @Chaitali-Sonparote for making this release better. Your efforts are greatly appreciated :-)

Better Azure Inventory and Configuration, initial CIS Rules, and Baselines

Data Connectors

• Azure Inventory and Configuration are updated, improved, and vetted against GovCloud
• AWS Inventory & Config adds results of aws inspector list-findings and describe-findings (ty @kuannie1 for the contribution!)
• an initial set of Azure CIS SQL rules is included in ./packs/azure_cis.sql
• fixes tenable.io agent collection to include agents not in a group and handling of API downtime
• fixes #376 in Azure Log connector (ty @plakhanpal for spotting this regression)

WebUI

• the URL now includes Alert and Violation id's, for easy linking to a specific rule
• beta features added to menu drop-down, letting you toggle ones not quite ready for prime-time
• (beta) Baselines section contains an initial version of Percentile Baselines have now been added to the WebUI. These help you analyze your data for patterns and alert on abnormalities.

Handlers

• adds custom issue type in Jira handler (ty @plakhanpal and @GalGreenfield for noting the trouble)

Minor & Misc

• various clarifications in AWS Inventory and Configuration docs
• fixes installer bug for accounts where default timezone is either LTZ or NTZ
• adds optional port environment variable running tests locally (#378)
• removes vestigial scripts
• fixes Jamf inventory connector to run every two hours
• makes it simpler to manually re-run alert query runners (ba88d6f)

New Connectors & Improvements

Data Connectors

• the new Azure Inventory & Configuration Connector gathers configuration and inventory into 23 tables for a given Azure Tenant, deprecating the Azure VM and Azure Subscription Connectors. The original set is intended to support upcoming CIS coverage, but #373 adds MS Intune support, as well.
• the new Salesforce Event Log gathers event logs from the Salesforce API, written and documented by @hh-jamesweakley, thank you kindly!
• the AWS Inventory & Configuration Connector has new documentation and some improvements to support collecting logs from multiple organizations, deprecating the AWS Inventory, Config, and Account Connectors. many thanks to @blackstatic ([email protected]) for his tireless help getting the documentation in order.
• the AWS Inventory & Configuration Connector can now collect from a single account (just include it as a "master" account and errors in organizations list-accounts fall back to collecting from that account alone)
• 4e88c05 fixes the Jamf Connector to ignore intermittent errors from the API

WebUI

• a9008b6 adds the ability for the web operator to set a default user role and database instead of using the individual users' Snowflake defaults, creating a smoother onboarding experience. Many thanks to @jamesweakley for both thinking of this improvement, implementing, and documenting it!
• large chunk of tech debt has been processed, with all deprecated frontend dependencies upgraded or replaced
• minor UX tweaks like b2795ac to fix text overflow in rule titles

Infra, AWS Collection, and WebUI improvements

Infra

• CloudFormation Templates are now included in ./infra/cfn (#362)

Thank you @maestro-jamesweakley (aka @jamesweakley) for contributing this to the release!

Connectors

The AWS Collection Connector has gotten a performance and reliability boost, designed to handle collection for even the largest organizations.

• AWS Collector has more accurate table name (efac7a6)
• AWS Collection bug fixes & speed optimizations (#363)
• AWS Collector iterates over all regions (7760e2d)

WebUI

The WebUI is moving out of a "work in progress" and into a "designed for production" state.

• Adds redirect-to-login fallback on auth errors in WebUI (3fe911e)
• Fixes server crash on invalid OAuth tokens (eb2984e)
• Use user's default db instead of environment variable in WebUI (ce42b77)
• Fix db connection cache persisting between requests (cc2ddec)
• Rename REGION envar to SA_REGION (9582d1c)

Runners

Minor maintenance and fixes in runners.

• Fixes Violations runners to properly populate query_name field (7bd53ec)

Installer

• Fixed installer to allow reruns to reset user PK without errors (f31073c)

CIS Benchmarks and AWS Collection

Data Connectors

• Adds AWS Collect Connector (#352)
• Adds CIS Benchmark pack (8de1dad)
• Adds JAMF Inventory Connector (925ca75)
• Adds Data Connector for AirWatch (#357)
• Rename Tenable Settings to Tenable.io Connector (eafcb77)

Deprecation

• Removes IAM Credential Report Ingestion Script (c74c907)

Bug Fixes

• Variety of bug fixes

Snowflake Packs

This minor release adds packs useful for creating monitoring alerts on data tracked by the Snowflake product, and renames ./samples to ./packs.

v1.8.8 Upgraded AWS AMI & Tenable Ingestion, SMTP Handler

Data Connectors

• Adds AMI to AWS Inventory (#346)
• Adds Tenable ingestion of vulns and assets (#348)

Handlers

• Adds unauthenticated SMTP support (d062a8f)
• Fixes bug in AQR caused by NULL::STRING values (5a7814c)
• Fixes bug in time_slices to work with large queries (72c72f1)

Improved Connectors, Handlers, Runners

Data Connectors

• Adds Fire library to the runner, to run connections separately from CLI
• Updates Okta Connector to include deprovisioned users
• Fixed AWS CloudTrail errors when mfaAuthenticated is str (#336)
• Fixes AWS Flow UI Role field name (#330)
• Adds IAM Connection Type to AWS Inventory (#337)
• Adds mypy checking to connectors modules
• Adds gov cloud support to Azure ingestion
• Adds account IDs to AWS Inventory EC2 and ELB ingestion (#344)
• Updates Azure client dependencies

Handlers

• Improves Jira handler's custom fields options
• Adds and fixes SMTP handler

Query Runners

• Adds custom Alert cutoff time via env var
• Fixes Violation runner in single violation run mode

Misc

• Standardizes Python code formatting with Black

Thanks

Thank you @alldoami, @edulop91, and @sf-bhushanchitte for contributions to this release!