v1 SDK - full move #7192

johanfylling · 2024-11-25T17:35:25Z

An alternative to the v1 SDK PR and the v0 global-toggle PR.

v1 SDK: Somewhat messy, where some parts of the go API is moved to v1, but some (most) aren't.
v0 global-toggle: Contentious:
- devs need to make code changes to remain on v0 behavior
- global toggle forces lib devs to not rely on default rego-version if there is a risk of competing OPA dependencies that rely on different default
- If default rego-version is set to v0 instead of v1, OPA 1.0 doesn't fully commit to one version, and will be stuck in the past.

This 3rd option moves all (with some exceptions) of OPA proper to the v1 package, and old root packages only contains type aliases and proxy functions.

Pros:

Cleaner interface than the original v1 SDK option
No global toggle; devs select what rego-version is the default through import paths

Cons:

Most packages, types, and functions will now have two options: one in the root for v0; and one under the v1 package for v1. Devs will need to make sure they're not mixing-and-matching their imports improperly. IDE auto-complete can lead you astray.
Somewhat larger maintainer burden (because of above point)

Signed-off-by: Johan Fylling <[email protected]>

for: * compiler * parser * bundle compiler * rego SDK * OPA SDK Signed-off-by: Johan Fylling <[email protected]>

Signed-off-by: Johan Fylling <[email protected]>

johanfylling · 2024-11-25T17:43:20Z

Conflicting files
ast/builtins.go
ast/index.go
ast/internal/scanner/scanner.go
ast/map.go
ast/term.go
debug/debugger.go
format/format.go
rego/rego.go
storage/disk/config_test.go
storage/disk/disk_test.go
storage/disk/partition_test.go
storage/disk/paths_test.go
topdown/builtins.go
topdown/crypto_test.go
topdown/errors_test.go
topdown/eval.go
topdown/eval_test.go
topdown/exported_test.go
topdown/http.go
topdown/http_test.go
topdown/input_test.go
topdown/json_test.go
topdown/jsonschema_test.go
topdown/lineage/lineage_test.go
topdown/object.go
...

Oh bother, this'll be fun 😩 ..

Conflicts: ast/builtins.go ast/index.go ast/map.go ast/term.go debug/debugger.go format/format.go rego/rego.go topdown/builtins.go topdown/http.go topdown/query.go types/types.go

Signed-off-by: Johan Fylling <[email protected]>

johanfylling · 2024-12-02T11:08:37Z

v1/doc.go

+// Package v1 implements the v1 API for the Open Policy Agent (OPA).
+// The v1 API defaults to enforcing the v1 Rego syntax ([github.com/open-policy-agent/opa/v1/ast.RegoV1]).
+// Most packages outside the v1 API are deprecated. These constitute the older v0 API, which defaults to the v0 Rego syntax ([github.com/open-policy-agent/opa/v1/ast.RegoV0]).
+// The v0 API is provided as a means to ease transition to OPA 1.0 for 3rd party integrations, see [TODO: LINK TO V0 MIGRATION GUIDE].


The v0 migration guide isn't completed yet. Whichever makes it into main last will need to add the link.

johanfylling · 2024-12-02T11:16:15Z

v1/doc.go

Signed-off-by: Johan Fylling <[email protected]>

johanfylling · 2024-12-02T14:10:00Z

v1/ast/parser.go

+const DefaultRegoVersion = RegoV1
+
+const (
+	RegoUndefined RegoVersion = iota


This rego-version is the default value for any ast.RegoVersion struct field, e.g. ast.ParserOptions.RegoVersion and ast.Module.regoVersion. Which allows us to select whatever rego-version we should actually default to based on the user's imports.
E.g.:

opa/sdk/opa.go

Line 33 in 31b5f17

if opts.RegoVersion == ast.RegoUndefined {

opa/v1/ast/compile.go

Line 1757 in 8b531ca

if mod.regoVersion == RegoUndefined {

johanfylling · 2024-12-02T14:28:55Z

ast/compile.go

-
-	_, errs := checker.CheckTypes(c.TypeEnv, elems, as)
-	return errs
+	return v1.NewCompiler().WithDefaultRegoVersion(DefaultRegoVersion)


So, ast.DefaultRegoVersion == ast.RegoV0 in this package. I wonder if we should use ast.RegoV0 here instead (and other places in the v0 API), as that is a bit more explicit ..

johanfylling · 2024-12-02T14:36:07Z

v1/ast/builtins.go

This GH diff makes it look like this is a completely new file, but it looks like git actually knows better:

$ git blame v1/ast/builtins.go 4dc1c56552 ast/builtins.go (Torin Sandall 2016-05-16 14:21:32 -0700 1) // Copyright 2016 The OPA Authors. All rights reserved. 4dc1c56552 ast/builtins.go (Torin Sandall 2016-05-16 14:21:32 -0700 2) // Use of this source code is governed by an Apache2 4dc1c56552 ast/builtins.go (Torin Sandall 2016-05-16 14:21:32 -0700 3) // license that can be found in the LICENSE file. 4dc1c56552 ast/builtins.go (Torin Sandall 2016-05-16 14:21:32 -0700 4) 4dc1c56552 ast/builtins.go (Torin Sandall 2016-05-16 14:21:32 -0700 5) package ast 4dc1c56552 ast/builtins.go (Torin Sandall 2016-05-16 14:21:32 -0700 6) ea2ea9b12b ast/builtins.go (Torin Sandall 2017-10-03 14:15:07 -0700 7) import ( ea2ea9b12b ast/builtins.go (Torin Sandall 2017-10-03 14:15:07 -0700 8) "strings" ea2ea9b12b ast/builtins.go (Torin Sandall 2017-10-03 14:15:07 -0700 9) 67e8954b57 v1/ast/builtins.go (Johan Fylling 2024-11-21 16:35:26 +0100 10) "github.com/open-policy-agent/opa/v1/types" ea2ea9b12b ast/builtins.go (Torin Sandall 2017-10-03 14:15:07 -0700 11) ) bad1c9a2b2 ast/builtins.go (Torin Sandall 2017-03-15 16:06:00 -0700 12) 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 13) // Builtins is the registry of built-in functions supported by OPA. 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 14) // Call RegisterBuiltin to add a new built-in. 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 15) var Builtins []*Builtin 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 16) 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 17) // RegisterBuiltin adds a new built-in function to the registry. 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 18) func RegisterBuiltin(b *Builtin) { 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 19) Builtins = append(Builtins, b) 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 20) BuiltinMap[b.Name] = b 95bdd4c824 ast/builtins.go (Torin Sandall 2017-02-09 16:59:44 -0800 21) if len(b.Infix) > 0 { 95bdd4c824 ast/builtins.go (Torin Sandall 2017-02-09 16:59:44 -0800 22) BuiltinMap[b.Infix] = b 95bdd4c824 ast/builtins.go (Torin Sandall 2017-02-09 16:59:44 -0800 23) } 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 24) } 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 25) 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 26) // DefaultBuiltins is the registry of built-in functions supported in OPA 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 27) // by default. When adding a new built-in function to OPA, update this 4dc1c56552 ast/builtins.go (Torin Sandall 2016-05-16 14:21:32 -0700 28) // list. 480281741b ast/builtins.go (Torin Sandall 2016-05-25 09:39:12 -0700 29) var DefaultBuiltins = [...]*Builtin{ ab91356d5b ast/builtins.go (Torin Sandall 2018-01-27 09:44:08 -0800 30) // Unification/equality ("=") 4dc1c56552 ast/builtins.go (Torin Sandall 2016-05-16 14:21:32 -0700 31) Equality, 6f90081ea3 ast/builtins.go (Torin Sandall 2016-11-25 10:53:21 -0800 32) ab91356d5b ast/builtins.go (Torin Sandall 2018-01-27 09:44:08 -0800 33) // Assignment (":=") ab91356d5b ast/builtins.go (Torin Sandall 2018-01-27 09:44:08 -0800 34) Assign, ab91356d5b ast/builtins.go (Torin Sandall 2018-01-27 09:44:08 -0800 35) f8286be64e ast/builtins.go (Stephan Renatus 2021-10-14 09:02:38 +0200 36) // Membership, infix "in": `x in xs` f8286be64e ast/builtins.go (Stephan Renatus 2021-10-14 09:02:38 +0200 37) Member, f8286be64e ast/builtins.go (Stephan Renatus 2021-10-14 09:02:38 +0200 38) MemberWithKey, f8286be64e ast/builtins.go (Stephan Renatus 2021-10-14 09:02:38 +0200 39) 6f90081ea3 ast/builtins.go (Torin Sandall 2016-11-25 10:53:21 -0800 40) // Comparisons 0f8c6798f2 ast/builtins.go (Torin Sandall 2018-02-16 09:22:16 -0800 41) GreaterThan, 0f8c6798f2 ast/builtins.go (Torin Sandall 2018-02-16 09:22:16 -0800 42) GreaterThanEq, 0f8c6798f2 ast/builtins.go (Torin Sandall 2018-02-16 09:22:16 -0800 43) LessThan, 0f8c6798f2 ast/builtins.go (Torin Sandall 2018-02-16 09:22:16 -0800 44) LessThanEq, 0f8c6798f2 ast/builtins.go (Torin Sandall 2018-02-16 09:22:16 -0800 45) NotEqual, 0f8c6798f2 ast/builtins.go (Torin Sandall 2018-02-16 09:22:16 -0800 46) Equal, 6f90081ea3 ast/builtins.go (Torin Sandall 2016-11-25 10:53:21 -0800 47)

johanfylling · 2024-12-02T16:26:13Z

ast/doc.go

-//	                   +--- Expression (Term | Terms | Variable Declaration)
-//
-// At query time, the policy engine expects policies to have been compiled. The compilation stage takes one or more modules and compiles them into a format that the policy engine supports.
+// Deprecated: Use [github.com/open-policy-agent/opa/v1/ast] instead.


This will put a deprecation message at the top of the doc page for the package:

and it will cause linters (such as staticcheck used in OPA) to warn about usage of deprecated functions and types when those are referenced.

However, it won't put deprecated notices on anything inside the deprecated package in the generated docs. For this, we need to explicitly tag each type, const, var, and function. We could brute force it by just adding a generic

// // Deprecated: Use [github.com/open-policy-agent/opa/v1] instead.

everywhere. But before doing that to thousands of places, we should determine if it's necessary.

Can web perhaps add some words, like "this package isn't going to go away but new projects are advised to start with v1"? If that's accurate. Just to avoid people panicking.

You mean "web" as in what's generated from here, or as in the separate v0-migration guide we're working on for the docs?

How about:

Deprecated: This package is intended for older projects transitioning from OPA v0.x and will remain for the lifetime of OPA v1.x, but its use is not recommended. For newer features and behaviours, such as defaulting to the Rego v1 syntax, use [github.com/open-policy-agent/opa/v1/ast] instead.

Can web perhaps add some words

typing with gloves on in harsh weather, sorry 😅 "Can we perhaps add..." is what I had meant.

Signed-off-by: Johan Fylling <[email protected]>

johanfylling added 10 commits November 21, 2024 16:35

Moving (most) source to v1 root dir

67e8954

Signed-off-by: Johan Fylling <[email protected]>

Making v1 the default rego-version

0f17753

Signed-off-by: Johan Fylling <[email protected]>

Adding back v0 API to root

c22d553

Signed-off-by: Johan Fylling <[email protected]>

Adding tests

31b5f17

for: * compiler * parser * bundle compiler * rego SDK * OPA SDK Signed-off-by: Johan Fylling <[email protected]>

Adding default rego-version to test runner

f607a52

Signed-off-by: Johan Fylling <[email protected]>

Fixing linter issues

8b86b19

Signed-off-by: Johan Fylling <[email protected]>

Adding deprecation notes to packages

9ecc0d8

Signed-off-by: Johan Fylling <[email protected]>

Adding all built-in declarations to ast

42091b8

Signed-off-by: Johan Fylling <[email protected]>

Adding topdown.NewPrintHook()

ab1fef3

Signed-off-by: Johan Fylling <[email protected]>

Applying default rego-version to format.SourceWithOpts()

893d58f

Signed-off-by: Johan Fylling <[email protected]>

johanfylling added 6 commits November 25, 2024 19:18

Merge branch 'main' into rego-v1/sdk_full_move

713d18d

Conflicts: ast/builtins.go ast/index.go ast/map.go ast/term.go debug/debugger.go format/format.go rego/rego.go topdown/builtins.go topdown/http.go topdown/query.go types/types.go

Updating Wasm tests

3698c00

Signed-off-by: Johan Fylling <[email protected]>

Updating check-yaml-tests location

26dad00

Signed-off-by: Johan Fylling <[email protected]>

Adding staticcheck exclusions for SA1019 deprecation warnings

2556737

Signed-off-by: Johan Fylling <[email protected]>

Adding missing bundle for test

8b4aed0

Signed-off-by: Johan Fylling <[email protected]>

Fixing TestWasmE2E

e9bf88a

Signed-off-by: Johan Fylling <[email protected]>

johanfylling force-pushed the rego-v1/sdk_full_move branch 3 times, most recently from e66a1f6 to 661a37b Compare November 27, 2024 14:39

Updating moke tests

525527a

Signed-off-by: Johan Fylling <[email protected]>

johanfylling force-pushed the rego-v1/sdk_full_move branch from 661a37b to 525527a Compare November 27, 2024 14:48

johanfylling added 3 commits November 27, 2024 16:24

Adding ast/compilehelper tests

5bbce70

Signed-off-by: Johan Fylling <[email protected]>

Updating cmd tests

1318f4b

Signed-off-by: Johan Fylling <[email protected]>

Adding back v0 repl package

8b531ca

Signed-off-by: Johan Fylling <[email protected]>

johanfylling force-pushed the rego-v1/sdk_full_move branch from fff4204 to 8b531ca Compare November 28, 2024 17:21

Updating tests

c31fba7

Signed-off-by: Johan Fylling <[email protected]>

johanfylling commented Dec 2, 2024

View reviewed changes

v1/doc.go

Copy link

Contributor Author

johanfylling Dec 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding doc to v1 package

fdb173d

Signed-off-by: Johan Fylling <[email protected]>

johanfylling force-pushed the rego-v1/sdk_full_move branch from 544a26a to fdb173d Compare December 2, 2024 11:31

johanfylling commented Dec 2, 2024

View reviewed changes

johanfylling requested review from srenatus and ashutosh-narkar December 2, 2024 15:21

johanfylling commented Dec 2, 2024

View reviewed changes

Fixing ast.TypeEnv type alias

a6b2a0e

Signed-off-by: Johan Fylling <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1 SDK - full move #7192

v1 SDK - full move #7192

johanfylling commented Nov 25, 2024

johanfylling commented Nov 25, 2024

johanfylling Dec 2, 2024

johanfylling Dec 2, 2024

johanfylling Dec 2, 2024

johanfylling Dec 2, 2024

johanfylling Dec 2, 2024

johanfylling Dec 2, 2024

srenatus Dec 2, 2024

johanfylling Dec 2, 2024

srenatus Dec 2, 2024

v1 SDK - full move #7192

Are you sure you want to change the base?

v1 SDK - full move #7192

Conversation

johanfylling commented Nov 25, 2024

johanfylling commented Nov 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment