More Connectors, New Handlers, and Baseline fixes

General

79f7954 Adds an Airbrake integration to better track exceptions (#323)
ff8176e Adds v1.8.5 which we merged into "latest" but forgot to tag a release for

Ingestion & Data Connectors

230d27b Adds parallelism to the runner (#307)
4acf685 Adds an LDAP Connector (#300)
465936b Adds an Nginx Connector (#295)
9b846f2 Adds an osquery Data Connector (#278)
3ef3d89 Fixes azure_vm & utils.create_metadata_table (#305)
1783677 Added list type and int casting to runner (#306)
c67f8ee Adds Meraki Device Data Connector (#290)
72f572c Improves AWS Inventory Connector (#303)

New Handlers

b5011a0 Adds SES, SNS, Twilio, SMTP, and Stored Procedure handlers

Thanks

Thanks, @kuannie1, @alldoami, @edulop91, and @rdobrik-snowflake, for contributing to this release!

New & improved Data Connectors and Baseline Runner

Installer

1ea9358 Fixes URL parser to add support for Azure URLs

Rule Queries and Handlers

89652d2 Adds assignee and area params to Jira handler
4119db4 Fixes Slack handler templates containing special characters

Data Connectors and Ingestion Scripts

319f9dc Adds Tenable Settings Connector w/ "Users" Connection Type
1742adb Fixes Azure Subscription Inventory Ingestion
f11c643 Removes ingestion/aad_auditlogs.py, now done by Azure AD Logs Data Connector

Baselines

d866a94 Fixes OOM errors by upgrading runner to use a new subprocess for each baseline
d866a94 Fixes minor bugs and removes up debugging statements
ef17ff0 Fixes R returning NaN to no longer break baseline creation
8871eda Fixes Prediction Violation Baseline to show full date

More Data Connectors, touch-ups, and fixes

Runner Changes, Fixes, and Improvements

• ./run all now runs connectors and baselines, as well as Alert and Violation Rules
• fixes Alert Query Runner when running a single Alert Query from CLI
• SnowAlert KMS region is set separately from Snowflake region (ty @nagalakshmisreeram!)
• Alert Queries can now set project parameter in Jira handler

New Collection Infra & fixes

• adds Azure Subscription Inventory Data Connector to collect Subscriptions
• adds Azure VM Data Inventory Data Connector to collect VM data
• adds command line for running a single Data Connector
• adds functionality to run a single ingestion script from CLI
• fixes AWS Config Data Connector to load dates out of filenames
• fixes AWS EC2 Inventory ingestion script to save dates properly
• fixes AWS Asset Data Connector

WebUI changes and bug fixes

• hide Policies behind localStorage feature flag
• includes Gunicorn web server
• fixes bug in saving queries with {...} in the SQL with arbitrary ...

Baseline Runner Improvements

• adds Violation Closeout Date baselines module
• adds Violations Linear Predictor baseline module
• adds MASS, tidyverse, and broom packages

v1.8.1 More Data Connectors and fixes

Data Connectors (DC's)

• AWS Config Data Connector lets you gather AWS configurations
• AWS Inventory Data Connector lets you use AWS API's to inventory cloud entities

Fixes & Improvements

• Auth PK may now be supplied in same format as Snowflake's ALTER USER command
• AWS CloudTrail DC handles malformed timestamps and records collection time
• Azure Connectors use newest API
• Several minor WebUI and installer bugs fixed
• Docs brought to date with code

v1.8.0 Data Connectors, Orchestrator Templates, etc.

Data Connectors

We have added next generation infrastructure and a UI to handle collecting data into Snowflake. Data Connectors have a standardized installation and ingestion interface.

To make sure they work best on existing installations, please add a metadata table —

USE SCHEMA results;
CREATE TABLE IF NOT EXISTS ingestion_metadata(event_time TIMESTAMP_LTZ, v VARIANT);
GRANT INSERT, SELECT ON ingestion_metadata TO ROLE snowalert;

To start, we are providing three Data Connectors: Okta, CloudTrail, and Azure Cloud Logs.

Orchestration Infrastructure

We're excited to see other companies take up SnowAlert and have preliminary data on what orchestration infrastructure our community will be using. To start, we are releasing k8s configuration files written by @cherrera2001. Thanks!

For internal teams, we've deployed SnowAlert to AWS Fargate using a Terraform module, which is included in the infra/ directory, as well. It's minimal for what we needed so pull request are welcome for, e.g. a Scheduled Task for the ingestion runner. Thanks @rdobrik-snowflake!

Improvements and Bug Fixes

• Fixed installer bug for custom passwords including symbol $ (88297a8)
• Fixed installer bug on accounts w/o data sharing (49c7856)
• Fixes behavior when Alert explicitly declares empty handler list (b98a56b)
• Added --no-samples options to installer (30453fc)
• Updates documentation and query packs to match latest functionality (607e816, d2fdeb6)
• Fixes UDTF privilege grant in installer (a904207)
• Move Violations on/off switch to top of page and label it (2353b50)
• WebUI dependencies updated and TS lib version bumped to es2019

v1.7.2 More Ingestion Scripts, bug fixes

New and Improved Ingestion Scripts

• new Agari Ingestion Script
• new AAD Ingestion Script
• improved EC2 inventory ingestion

Minor bug fixes and improvements

• fixes crash when using unencrypted passphrase in env var (cd66e77)
• alert query and suppression failures create better Jira tickets (ec6440f)
• fixes WebUI OAuth to non-default region accounts (ty @cherrera2001 for ef38471)
• dispatcher dispatches 1000 alerts at a time, instead of 100 (0f4bd91)
• don't drop public schema, for installing over existing DB
• smarter caching in WebUI

ZenGRC Ingestion and misc fixes

• adds ZenGRC Ingestion Script
• fixes a bug in runners unable to decrypt pk passphrase
• fixes a bug in installer crashing when unable to find samples share
• fixes a non-deterministic bug in boto3 returning part of a result

Handlers, Baselines, Security Hardening

Custom Handlers

The Alert Handler is replaced with the Alert Dispatcher with plug-in handlers in the ./src/runners/handlers directory, and the Jira Handler is now complemented by the Slack Handler. How to develop new ones is documented in the refreshed CONTRIBUTING document.

New Ingestion Scripts

We have added three new ingestion scripts —

• ec2_describe_instances.py
• iam_credential_report.py
• list_aws_accounts.py

Installer Improvements

The installer has several minor bug fixes, as well as new parameters which let you customize what it does —

--admin-role to set a SnowAlert administrator other than ACCOUNTADMIN
--nosamples to install SnowAlert without sample rules
--config-account to auth with a named account inside your `~/.snowsql/config`

Optimized Alert Query Runner (AQR)

We've gone through a performance review and have begun optimizing how long alerts execute to best utilize your Snowflake resources, we've also added multithreading to the AQR. Because the queries still all MERGE into the same table, the bulk of the benefit from this optimization is expected in a future release.

Easier Contribution

We've rewritten the CONTRIBUTING intro and have created a more thorough automated test suite to make it easier to test changes and improvements.

(beta) Baselines

We've added an R installation to our main container and a Baseline Runner that lets you define tables you'd like populate with statistical baselines that would take too long to include in Alert Queries on-the-fly. We've yet to sand down the corners on this feature, document the functionality, and add it to the WebUI, but we are using it internally so feel free to read the code or reach out at [email protected] if you'd like to help us test or develop it.

(beta) WebUI moving towards launch

While the WebUI is still heavily a work-in-progress, it went through user feedback and an initial security hardening. The WebUI and the runners have been patched for a minor SQL injection vulnerability and other fixes that will prepare us to launch soon.

Polish and Fixes

Polish

• Adds owner field to Violations, default values for missing fields
• Adds config account selection option to installer
• Adds data views for querying rules by tag
• Adds query_name set in Alert Query Runners instead of AQ's

Fixes

• Fixes default value of alert event_time in WebUI
• Fixes bugs in WebUI, installer, and ingestion script
• Fixes bugs in VQ runner metadata run and error recording

Alert Correlations, Rule Tags, fixe

Core Functionality

There's a variety of new features and improvements in this release —

• Correlation functionality helps organize Alerts, by adding an ID when a chain of two or more alerts close in time have the same (actor, object) or (actor, action) pair;
• Tags can help you organize Alert and Violation Rules; and
• Alert Query and Violation Suppression performance, stability, and logging is all improved.

Introducing Ingestion

One of the biggest pain points we're looking to address is the lack of coordination in creating rules, and a very similar problem exists in upstream collection of data into Snowflake to start the whole process. Since this is a problem a lot of people are solving and re-solving, SnowAlert is going to help. This is not yet a "fully baked" feature that's part of the official runners or integrated with the WebUI, but it's a hint at what is to come. We are starting with two ingestion scripts —

• Okta log ingestion script, and
• AWS Accounts ingestion script.

WebUI

The SAM-UI is now called the WebUI and we're moving towards taking it out of "in progress" to "first rate" soon. This release —

• fixes UX and functionality bugs in WebUI
• adds support for installer being run by DB admin separate from account admin
• adds installer flags allowing for simple unattended installation