v0.24.0

This release contains a number of small enhancements and bug fixes.

Bundle Persistence

This release adds support for persisting bundles for recovery purposes. When persistence is enabled, OPA will save activated bundles to disk. On startup, OPA checks for persisted bundles and activates them immediately. This allows OPA to startup if the bundle server is unavailable (#2097). For more information see the Bundle documentation.

Built-in Functions

This release includes a few new built-in functions:

• base64.is_valid for testing if strings are valid base64 encodings (#2690) authored by @carlpett
• net.cidr_merge function for merging sets of IPs and CIDRs (#2692)
• urlquery.decode_object for parsing URL query parameters into objects (#2647) authored by @GBrawl

In addition, http.send has been enhanced to support caching overrides and in-band error handling (#2666 and #2187).

Fixes

• Fix opa build to support custom built-in functions (#2738) authored by @gshively11
• Fix for file watching volume mounted configmaps (#2588) authored by @drewwells
• Fix discovery plugin to set last request and last successful request timestamps in status updates (#2630)
• Fix planner crash on virtual document iteration (#2601)
• Fix decision logger to requeue failed chunks (#2724 authored by @anderseknert)
• Fix object/set implementation in WASM-C library to avoid resizing.
• Fix JSON parser in WASM-C library to copy memory for strings and numbers.
• Improve WASM-C library to recycle object and set element structures while growing.

In addition, this release contains several fixes for panics identified by fuzzing:

• ast: Fix compiler to expand exprs in rule args (#2649)
• ast: Fix output var analysis to accept refs with non-var heads (#2678)
• ast: Fix panic during local var rewriting (#2720)
• ast: Fix panic in local var rewriting caused by object corruption (#2661)
• ast: Fix panic in parser post-processing of expressions (#2714)
• ast: Fix parser to ignore rules with args and key in head (#2662)
• ast: Fix object corruption during safety reordering
• types: Fix panic on reference to object with composite key (#2648)

Backwards Compatibility

• Renamed timer_rego_builtin_http.send_ns to timer_rego_builtin_http_send_ns to avoid issues with periods in metric keys.
• Removed deprecated watch package (#2265)

Miscellaneous

• Add support for H2C on HTTP listener (#2739 thanks @srenatus!).
• Add Go version information to opa version output (thanks @srenatus!)
• The official OPA build has been updated to Go v1.14.9. Previously it was using v1.13.7 which is no longer supported (thanks @srenatus!)

v0.23.2

This release contains a fix for a regression in v0.23.1 around bundle downloading. The bug caused OPA to cancel bundle downloads prematurely. Users affected by this issue would see the following error message in the OPA logs:

[ERROR] Bundle download failed: bundle read failed: archive read failed: context canceled
  plugin = "bundle"
  name = <bundle name>

v0.23.1

Fixes

• plugins/discovery: Set the last request and last successful request in discovery status (#2630)

Miscellaneous

• plugins/rest: Add response header timeout for REST client

v0.23.0

http.send Caching

The http.send built-in function now supports caching across policy queries. The caching.inter_query_builtin_cache.max_size_bytes configuration setting places a limit on the amount of memory that will be used for built-in function caching. By default, not limit is set. For http.send, cache duration is controlled by HTTP response headers. For more details see the http.send documentation.

Capabilities

OPA now supports a capabilities check on policies. The check allows callers to restrict the built-in functions that policies may depend on. If the policies passed to OPA require built-ins not listed in the capabilities structure, an error is returned. The capabilities check is currently supported by the check and build sub-commands and can be accessed programmatically on the ast.Compiler structure. The repository also includes a set of capabilities files for previous versions of OPA under the capabilities/ directory.

For example, given the following policy:

package example

deny["missing semantic version"] {
  not valid_semantic_version_tag
}

valid_semantic_version_tag {
  semver.is_valid(input.version)
}

We can check whether it is compatible with different versions of OPA:

# OK!
$ opa build ./policies/example.rego --capabilities ./capabilities/v0.22.0.json

# ERROR!
$ opa build ./policies/example.rego --capabilities ./capabilities/v0.21.1.json

Built-in Functions

This release includes a new built-in function to test if a string is a valid regular expression: regex.is_valid.

WebAssembly

• Host environments no longer have to provide the opa_println function when instantiating compiled policy modules.
• SDKs no longer have to set the heap top address during initialization.

Fixes

• Add a new inter-query cache to cache responses across queries (#1753)
• Fix opa CLI flags to match documentation (#2586) authored by @OmegaVVeapon
• Fix rule indexing when multiple glob.match mappers are required (#2617)
• Fix AST to marshal non-string object keys (#516)
• Fix signature calculation to include port if necessary (#2568)
• Fix partial evaluation to check function output for false values (#2573)

Miscellaneous

• Add http.send latency to query metrics (#2034)
• Add support for opa build unknowns under data (#2581)
• Add support to wait for plugin readiness before starting server
• Add parameter to set wall clock time during evaluation for replay purposes
• Fix groundness bit on objects during update
• Fix x509 built-in functions to parse PEM or DER inputs
• Fix bundle signing and verification to use standard JWT key ID header
• Optimize AST collections to cache hash values
• Optimize object iteration to avoid hashing
• Optimize evaluator by removing unnecessary term copying

Deprecations

• The watch query parameter on the Data API has been deprecated. The query watch feature was unused and the lack of incremental evaluation would have introduced scalability issues for users. The feature will be removed in a future release.

• The partial query parameter on the Data API has been deprecated. Note, this only applies to the partial query parameter that the Data API supports, not Partial Evaluation itself. The partial parameter allowed users to lazily trigger Partial Evaluation (for optimization purposes) during a policy query. While this is useful for kicking the tires in a development environment, putting optimization into the policy query path is not recommended. If users want to kick the tires with Partial Evaluation, we recommend running the opa build command.

Backwards Compatibilty

• The storage.Indexing interface has been removed. Storage indexing has not been supported since 0.5.12. It was time to remove the interface. Custom store implementations that may have included no-op implementations of the interface can be updated.

• The ast.Array type has been redefined a struct. Previously ast.Array was a type alias for []*ast.Term. This change is backwards incompatible because slice operations can no longer be performed directly on values of type ast.Array. To accomodate, the ast.Array type now exports functions for the same operations. This change decouples callers from the underlying array implementation which opens up room for future optimizations.

v0.22.0

Bundle Signing

OPA now supports digital signatures for policy bundles. Specifically, a signed bundle is a normal OPA bundle that includes a file named ".signatures.json" that dictates which files should be included in the bundle, what their SHA hashes are, and of course is cryptographically secure. When OPA receives a new bundle, it checks that it has been properly signed using a key that OPA has been configured with out-of-band. Only if that verification succeeds does OPA activate the new bundle; otherwise, OPA continues using its existing bundle and reports an activation failure via the status API and error logging. For more information see https://openpolicyagent.org/docs/latest/management/#signing. Many thanks to @ashish246 who co-designed the feature and provided valuable input to the development process with his proof-of-concept #1757.

Optimization Levels

opa build now supports multiple optimization levels. The first level (--optimize=1) enables constant folding (based on partial evaluation) that only inlines values that can be computed entirely at build time. The second level (--optimize=2) enables the existing (more aggressive) version of partial evaluation that eagerly inlines as much of the policy as possible. For more information on the optimization levels see the Optimization Levels section in the documentation.

Built-in Functions

• numbers.range (#2479) was added to support policies that need to generate a range of integers (e.g., a network port range).
• semver.is_valid and semver.compare (#2538) was added to support policies that need to validate semantic version numbers (authored by @charlieegan3).

WebAssembly

• All String built-in functions (except sprintf) are now implemented natively inside of Wasm-compiled policies.

Fixes

• A few small issues in the Go integration and rego package examples have been resolved (#2294) and #2367) authored by @gaga5lala.
• The Kubernetes Admission Controller tutorial as been updated to work with recent versions of Kubernetes (#2467 authored by @gaga5lala).
• A few issues in partial evaluation around negation inlining and partial rules have been resolved (e.g., #2492, #2491).

Miscellaneous

• OPA now supports IMDSv2 for the AWS metadata service. This improves the security posture of OPA deployments in AWS (#2482) authored by @nhw76.
• Several improvements to the project documentation including a policy style discussion, an integration option comparison, and discussion of bootstrapping and fail-open versus fail-closed modes.
• The project's CI/CD infrastructure has been migrated to GitHub Actions. The new CI/CD infrastructure is designed and implemented to be portable and includes a number of quality-of-life improvements.
• End-to-end query latency with decision logging enabled has been improved by 10%-15% in real-world cases.

Backwards Compatibility

• The rego.Tracer and rego.EvalTracer API's have been deprecated in favor of
  the newer rego.QueryTracer and rego.EvalQueryTracer API.
• The tester.Runner#SetCoverageTracer API has been deprecated in favor of the
  newer test.Runner#SetCoverageQueryTracer API.

v0.21.1

This release fixes #2497 where the comprehension indexing optimization produced incorrect results for nested comprehensions that close over variables in the outer scope. This issue only affects policies containing nested comprehensions that are recognized by the indexer (which is a relatively small percentage).

This release also backports the GitHub Actions migration and a fix to the Wasm library build step.

v0.21.0

Features

• Decision log masks can now mutate decision log events. Previously, the masks could only erase data in the events. With this change, users can implement masks that obfuscate or add information to the decision log events before they are emitted. Thanks to @dkiser for implementing this feature #2379)!

• This release contains a new built-in function for parsing X.509 Certificate Signing Requests (crypto.x509.parse_certificate_request). Thanks to @vivekbagade for implementing this feature #2402!

• This release adds support for aggregation and bit arithmetic operations for WebAssembly compiled policies. These functions no longer have to be provided by the host environment.

Fixes

• cmd: Fix bug in --disable-inlining option parsing (#2196) authored by @Syn3rman
• docs: Improve terraform example to incorporate child_modules (#1772)
• server: Fix panic caused by compiler misuse with bundles (#2197)
• topdown: Fix incorrect memoization during partial evaluation (#2455)
• topdown: Fix loss of precision in arithmetic and aggregate builtins (#2469)

Miscellaneous

• Thanks to @Syn3rman for implementing an improvement to our release process to automatically tag external contributors (#2323)!

• The coverage and profiling tracers no longer require variable values from the evaluator. This change improves perfomance significantly when coverage or profiling is enabled and policies inspect large data sets. Benchmarks show anywhere from 0.5x to over 30x speedup depending on the policy.

Backwards Compatibility

• topdown.Tracer has been deprecated in favor of a newer interface
  topdown.QueryTracer.
• All tracers (regardless of interface implementation) will now only be checked
  for being enabled at the beginning of query evaluation rather than on a
  per-event basis.
• topdown.BuiltinContext#Tracers has been deprecated in favor of
  topdown.BuiltinContext#QueryTracers. The older Tracers field will be nil
  starting this release, and eventually removed.

v0.20.5

Fixes

• compile: Change name of result var for wasm binary (#2441)
• format: Deep copy inputs to avoid mutating the caller's copy (#2439)

Miscellaneous

• docs: Add opa_println to wasm required imports

v0.20.4

0.20.4

Fixes

• format: Refactor wildcard names to rewrite early (#2430)

v0.20.3

Fixes

• docs/content small output correction on terraform page (#1772)
• format: Fix wildcards in nested refs