Partial evaluation response simplification

[srlk]

Hello

We have a use case where we create some policies those contain somehow complex operators as

• Filtering based on regex and/or some conditions on multiple collections
• Allow/deny decision based on intersection/difference of these collections.

To visualise the situation, let's say we have multiple complex operations custom_business_condition1, custom_business_condition2 ... custom_business_condition...
With a theorical REGO policy

package pkg

custom_business_condition1(collection1) {
    lhs := { x |
        x := collection1[_].value
        x.important
    }
    count(lhs) == 0
}

custom_business_condition2(collection1, collection2) {
    lhs := { x |
        x := collection1[_].value
        x.important
    }
    rhs := { x | x := collection2[_].value}
    count(lhs) == count(lhs & rhs)
}

allow {
    custom_business_condition1(input.col1)
    custom_business_condition2(input.col1, input.col2)
}

And then this policy (among other similar policies) bundled and the policies need to be called by different services via Compile API.
And those services are backed by different data store, let's say SQL, Elastic Search, MongoDB etc

As it is now, with an example input

{
  "col2": [
    {
      "value": "a"
    }
  ]
}

Partial evaluation result will be as below

# Query 1
__local2__2 = {__local1__2 | __local1__2 = input.col1[_].value; __local1__2.important}

count(__local2__2, 0)
__local6__3 = {__local5__3 |
        __local5__3 = input.col1[_].value
        __local5__3.important
}

count(__local6__3 & {"a"}, count(__local6__3))

The resulting rego unwraps and tries to evaluate content of the function, which is great but very difficult to translate into predicates for multiple dialects. All services calling Compile API would need to implement all operations that make up custom_business_condition... and some steps may even not be practical to translate due to how we express them in REGO.

We are looking into a way that OPA sends directly the function name in the partial evaluation response as custom_business_condition... as a predicate if they are dependent on unknowns, rather than providing all steps how custom_business_condition... should be calculated.

Therefore, instead of having to deal with the low level rego conditions, the expected partial evaluation response in Rego format could be simplified to this:

# Query 1
custom_business_condition1(input.col1)
custom_business_condition2(input.col1, [{"value": "a"}])

The services that call OPA will then be able to create and optimize the condition (eg. custom_business_condition1) in their own dialect and would be abstracted from the REGO low level implementation.

One possible solution we have looked into is creating these complex operations as built-in functions and compile our own dialect of OPA. This seems to work fine but also comes with a con that we would need to build our own instance of OPA executable.

I was wondering if there's a way to achieve this without extending OPA or a feature in the road-map? Maybe some kind of annotation for the functions to hint OPA for this behavior?

# @compile_api_unwrap(false)
custom_business_condition1(collection1) {
...
}

Looking forward to hear your ideas and suggestions
Thanks in advance!

[srenatus]

There are more comprehensive inlining controls available when using the golang library. Would those do the trick for you? Perhaps exposing some more of those via the compile API would be a way forward 🤔

[srlk]

Hey @srenatus

Thanks for the tip, I've looked up inlining controls and disableInlining seems to do the trick. The only con is that it is not exposed yet via Compile API as far as I could see.

# Query 1
data.partial.func.custom_business_condition1(input.col1)
data.partial.func.custom_business_condition2(input.col1, [{"value": "a"}])

# Module 1
package partial.func

custom_business_condition1(__local0__2) {
        __local2__2 = {__local1__2 |
                __local1__2 = __local0__2[_].value
                __local1__2.important
        }

        count(__local2__2, __local9__2)
        __local9__2 = 0
}

custom_business_condition2(__local3__3, __local4__3) {
        __local6__3 = {__local5__3 |
                __local5__3 = __local3__3[_].value
                __local5__3.important
        }

        __local8__3 = {__local7__3 |
                __local7__3 = __local4__3[_].value
        }

        count(__local6__3, __local10__3)
        __local11__3 = __local6__3 & __local8__3
        count(__local11__3, __local12__3)
        __local10__3 = __local12__3
}

[srenatus]

Thanks for filing the issue 👉 open-policy-agent/opa#4357 (closing the loop)