Logging Gatekeeper Policy Constraint Violations (Warn)

[MarkVLK]

Ahead of enabling new constraints, I'd like to use the warn enforcementAction and then query logs I'm capturing to identify how frequently, if at all, various warn constraint violations are occurring. However, I'm not seeing any warn logs and I'm unsure if it's because I'm not capturing the correct logs, or there's just no resources violating the constraints I have set to warn.

I see in this documentation that there should be a constraint_audited event type, but I wasn't sure if that fired for warning events too or only if Gatekeeper is actively blocking resources not passing constraints (deny enforcement action). I don't see any constraint_audited events in logs, but I do see many audit_started and audit_finished events.

I also noticed the same docs say

The audit pod emits JSON-formatted audit logs to stdout.

but my logs curiously seem to only be seeing things from the stderr stream for Gatekeeper. Again, I'm not sure if this is due to a misconfiguration on my end or if there's just nothing being printed to stdout to capture in logs.

Any pointers would be greatly appreciated!

[ritazh]

@MarkVLK There shouldn't be any difference for enforcementAction: warn constraints.

There are few ways to troubleshoot this.

1, You can see how many violations your warn constraint is generating from the audit pod logs with
kubectl logs gatekeeper-audit-5bd959c795-vhq7f -n gatekeeper-system | grep constraint_audited does constraint_violations return > 0?

{"level":"info","ts":1667875283.0518973,"logger":"controller","msg":"audit results for constraint","process":"audit","audit_id":"2022-11-08T02:41:21Z","event_type":"constraint_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sAllowedRepos","constraint_name":"prod-repo-is-openpolicyagent","constraint_namespace":"","constraint_action":"warn","constraint_status":"enforced","constraint_violations":"10"}

2. You can also look for specific violating resource for this constraint:
   kubectl logs gatekeeper-audit-5bd959c795-vhq7f -n gatekeeper-system | grep opa does it return a violation mapped to the warn constraint?

{"level":"info","ts":1667875342.9665437,"logger":"controller","msg":"container <opa> has an invalid image repo <gcr.io/smythe-kpc/testbuilds/opa:0.9.2>, allowed repos are [\"openpolicyagent\"]","process":"audit","audit_id":"2022-11-08T02:42:21Z","details":{},"event_type":"violation_audited","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sAllowedRepos","constraint_name":"prod-repo-is-openpolicyagent","constraint_namespace":"","constraint_action":"warn","resource_group":"","resource_api_version":"v1","resource_kind":"Pod","resource_namespace":"default","resource_name":"opa"}

3. You can use https://open-policy-agent.github.io/gatekeeper/website/docs/violations/#log-denies to log admission events to see if users are admitting violating requests.

Here is an example output in the log:

{"level":"info","ts":1667875112.8724792,"logger":"webhook","msg":"denied admission","process":"admission","event_type":"violation","constraint_name":"prod-repo-is-openpolicyagent","constraint_group":"constraints.gatekeeper.sh","constraint_api_version":"v1beta1","constraint_kind":"K8sAllowedRepos","constraint_action":"warn","resource_group":"","resource_api_version":"v1","resource_kind":"Pod","resource_namespace":"default","resource_name":"opa","request_username":"kubernetes-admin"}

[MarkVLK]

@ritazh sorry for the severely delayed response!

After using the commands you suggested above, replacing the pod name with my own pod name, all of the grep commands came back with no responses. When grepping for keywords from constraints I've created, I do see some output related to watcher registry, template status being updated, etc. but no warnings or alerts or anything about resources being audited.

A while ago we did try to set the log-denies flag to true (we deployed via Helm and Terraform, using a helm_release resource) like so:

resource "helm_release" "gatekeeper" {
...

set {
    name  = "logDenies"
    value = "true"
  }
}

This was based on the parameters listed at https://github.com/open-policy-agent/gatekeeper/blob/master/charts/gatekeeper/README.md#parameters

[maxsmythe]

Are you looking to see how many resources currently exist on the cluster that violate a policy with a "warn" enforcement action, or are you looking to see how many requests to the API server triggered a "warn" response from the webhook?

Knowing which you're looking for will help determine where to look and whether/if/how anything could be going wrong.