docs/concept: working with incomplete CUE

This adds a concept guide explaining the basics of working with
incomplete CUE using both the cue command and the Go API.

Fixes cue-lang/docs-and-content#180.

Preview-Path: /docs/concept/working-with-incomplete-cue/
Signed-off-by: Paul Jolly <[email protected]>
Change-Id: If2fdd49e42c96cd098a19464038827ad2000e951
Dispatch-Trailer: {"type":"trybot","CL":1203092,"patchset":17,"ref":"refs/changes/92/1203092/17","targetBranch":"master"}

content/docs/concept/working-with-incomplete-cue/en.md

@@ -0,0 +1,163 @@
+---
+title: "Working with incomplete CUE"
+toc_hide: true
+authors: [myitcv, jpluscplusm]
+tags: [go api]
+---
+
+{{{with _script_ "en" "set caches to speed up re-running"}}}
+export GOMODCACHE=/caches/gomodcache
+export GOCACHE=/caches/gobuild
+{{{end}}}
+
+In general, CUE can handle references to fields that don't yet exist, or where
+a value can't be calculated because there's insufficient information.
+CUE that contains such references or values is referred to as *incomplete* CUE
+ -- but only if it's otherwise *valid* CUE.
+
+Here's an example of incomplete CUE. The value of `connectionString` can't be
+calculated given the information in `database.cue` alone because the value of
+the `password` field isn't concrete - it's only a `string` constraint.
+
+{{{with upload "en" "cue"}}}
+-- database.cue --
+package database
+
+connectionString: "\(system)://\(user):\(password)@\(host):\(port)/\(database)"
+
+system:   "postgres"
+host:     "prod.db.example.com"
+user:     "alex"
+port:     "5432"
+database: "transactions"
+password: string
+{{{end}}}
+
+CUE allows evaluations to be augmented by extra information introduced through
+{{<linkto/inline "tour/basics/unification" >}}.
+If we introduce information into an evaluation of the `database.cue` file that
+provides a concrete `string` value for the `password` field then the value of
+the `connectionString` field can be calculated.
+But until that happens, this otherwise valid CUE is referred to as "incomplete"
+because (in isolation) it doesn't contain sufficient information to permit a
+complete evaluation.
+
+Note that because incomplete CUE is valid CUE it *can* be evaluated ...
+
+{{{with script "en" "cue eval succeeds"}}}
+cue eval database.cue
+{{{end}}}
+
+... but it *can't* be exported:
+
+{{{with script "en" "cue export fails"}}}
+! cue export database.cue
+{{{end}}}
+
+Issue {{<issue 2120>}}#2120{{</issue>}} tracks some proposed changes to `cue
+vet` that would allow it to be used in place of `cue eval`, above, to vet the
+incomplete CUE silently.
+
+A configuration that results in incomplete values can be made *complete* by
+unifying it with the right information. This means that every field that
+contributes to the configuration being emitted must be able to be calculated.
+
+Here's some YAML data that will do this for our example and "fill in the gaps"
+in our incomplete CUE by providing the `password` secret:
+
+{{{with upload "en" "data"}}}
+-- secrets.yaml --
+password: "Ch^ngeMeBef0r3GoL!ve"
+{{{end}}}
+
+Unifying our incomplete CUE with this data makes the configuration complete,
+and allows us to export the result:
+
+{{{with script "en" "cue export succeeds"}}}
+cue export database.cue secrets.yaml
+# Export just the data source name as a text value.
+cue export database.cue secrets.yaml -e connectionString
+{{{end}}}
+
+### Using the Go API
+
+The Go API is also able to handle incomplete CUE.
+
+To demonstrate the Go API in action we start by initializing a Go module:
+
+{{{with script "en" "go mod init"}}}
+#ellipsis 0
+go mod init go.example
+{{{end}}}
+
+We place this example Go code in `main.go`.
+Comments explain what is being done at each step.
+
+{{{with upload "en" "go"}}}
+-- main.go --
+package main
+
+import (
+	"fmt"
+	"log"
+
+	"cuelang.org/go/cue"
+	"cuelang.org/go/cue/cuecontext"
+	"cuelang.org/go/cue/load"
+	"cuelang.org/go/encoding/yaml"
+)
+
+func main() {
+	ctx := cuecontext.New()
+
+	// Step #1: load the CUE package in the current directory.
+	// It contains a single file - "database.cue", as shown above.
+	bis := load.Instances([]string{"."}, nil)
+	step1 := ctx.BuildInstance(bis[0])
+	fmt.Printf("step1: %v\n", step1)
+
+	// Step #2: load the "secrets.yaml" file shown above.
+	step2File, err := yaml.Extract("secrets.yaml", nil)
+	if err != nil {
+		log.Fatal(err)
+	}
+	step2 := ctx.BuildFile(step2File)
+	fmt.Printf("step2: %v\n", step2)
+
+	// Ensure that the result of unifying the two steps is both
+	// valid and concrete - and thus could be exported as data:
+	result := step1.Unify(step2)
+	if err := result.Validate(cue.Concrete(true)); err != nil {
+		log.Fatal(err)
+	}
+
+	// Display the resulting CUE:
+	fmt.Printf("result: %v\n", result)
+}
+{{{end}}}
+
+We fetch the latest version of CUE, and tidy our Go module:
+
+{{{with script "en" "go get etc"}}}
+#ellipsis 0
+go get cuelang.org/go@${CUELANG_CUE_LATEST}
+#ellipsis 0
+go mod tidy
+{{{end}}}
+
+When we run our Go code, it behaves the same as the `cue export` command above
+\- except that it *also* displays the interim `step1` and `step2` values:
+
+{{{with script "en" "go run"}}}
+go run .
+{{{end}}}
+
+{{{with _script_ "en" "https://github.com/cue-lang/cue/issues/3496"}}}
+go vet ./...
+#ellipsis 0
+go run honnef.co/go/tools/cmd/[email protected] ./...
+{{{end}}}
+
+## Related content
+
+- Tag: {{<tag "go api">}} -- Guides exploring CUE's Go API

content/docs/concept/working-with-incomplete-cue/gen_cache.cue

@@ -0,0 +1,150 @@
+package site
+{
+	content: {
+		docs: {
+			concept: {
+				"working-with-incomplete-cue": {
+					page: {
+						cache: {
+							upload: {
+								cue:  "o7fv3WIZQZf9ZvcPQG0AgTb3HliyXY1A3m+ISjZc1Ng="
+								data: "zFddlyP+5ljb/rgQuQvSIj63WHuP/HEFEJLRt+IvlFc="
+								go:   "jJD0SLVBLo46ZI9UsmrgNnDsFH62h0Iwqp9QUIqodo8="
+							}
+							multi_step: {
+								hash:       "CVDM8AJKKI0BSIBSVNKHLI5A9237NNF583ESBRN3CG4R5AO2QSM0===="
+								scriptHash: "84I3SKVHB10K1UE2EBQM7NBU6TOAKB33LDB4MAK4U5EAG83G68K0===="
+								steps: [{
+									doc:      ""
+									cmd:      "export GOMODCACHE=/caches/gomodcache"
+									exitCode: 0
+									output:   ""
+								}, {
+									doc:      ""
+									cmd:      "export GOCACHE=/caches/gobuild"
+									exitCode: 0
+									output:   ""
+								}, {
+									doc:      ""
+									cmd:      "cue eval database.cue"
+									exitCode: 0
+									output: """
+											connectionString: "\\(system)://\\(user):\\(password)@\\(host):\\(port)/\\(database)"
+											system:           "postgres"
+											host:             "prod.db.example.com"
+											user:             "alex"
+											port:             "5432"
+											database:         "transactions"
+											password:         string
+
+											"""
+								}, {
+									doc:      ""
+									cmd:      "cue export database.cue"
+									exitCode: 1
+									output: """
+											password: incomplete value string:
+											    ./database.cue:10:11
+											connectionString: invalid interpolation: non-concrete value string (type string):
+											    ./database.cue:3:19
+											    ./database.cue:10:11
+
+											"""
+								}, {
+									doc:      ""
+									cmd:      "cue export database.cue secrets.yaml"
+									exitCode: 0
+									output: """
+											{
+											    "connectionString": "postgres://alex:Ch^[email protected]:5432/transactions",
+											    "system": "postgres",
+											    "host": "prod.db.example.com",
+											    "user": "alex",
+											    "port": "5432",
+											    "database": "transactions",
+											    "password": "Ch^ngeMeBef0r3GoL!ve"
+											}
+
+											"""
+								}, {
+									doc:      "# Export just the data source name as a text value."
+									cmd:      "cue export database.cue secrets.yaml -e connectionString"
+									exitCode: 0
+									output: """
+											"postgres://alex:Ch^[email protected]:5432/transactions"
+
+											"""
+								}, {
+									doc:      "#ellipsis 0"
+									cmd:      "go mod init go.example"
+									exitCode: 0
+									output: """
+											...
+
+											"""
+								}, {
+									doc:      "#ellipsis 0"
+									cmd:      "go get cuelang.org/[email protected]"
+									exitCode: 0
+									output: """
+											...
+
+											"""
+								}, {
+									doc:      "#ellipsis 0"
+									cmd:      "go mod tidy"
+									exitCode: 0
+									output: """
+											...
+
+											"""
+								}, {
+									doc:      ""
+									cmd:      "go run ."
+									exitCode: 0
+									output: """
+											step1: {
+											\tconnectionString: "\\(system)://\\(user):\\(password)@\\(host):\\(port)/\\(database)"
+											\tsystem:           "postgres"
+											\thost:             "prod.db.example.com"
+											\tuser:             "alex"
+											\tport:             "5432"
+											\tdatabase:         "transactions"
+											\tpassword:         string
+											}
+											step2: {
+											\tpassword: "Ch^ngeMeBef0r3GoL!ve"
+											}
+											result: {
+											\tconnectionString: "postgres://alex:Ch^[email protected]:5432/transactions"
+											\tsystem:           "postgres"
+											\thost:             "prod.db.example.com"
+											\tuser:             "alex"
+											\tport:             "5432"
+											\tdatabase:         "transactions"
+											\tpassword:         "Ch^ngeMeBef0r3GoL!ve"
+											}
+
+											"""
+								}, {
+									doc:      ""
+									cmd:      "go vet ./..."
+									exitCode: 0
+									output:   ""
+								}, {
+									doc:      "#ellipsis 0"
+									cmd:      "go run honnef.co/go/tools/cmd/[email protected] ./..."
+									exitCode: 0
+									output: """
+											...
+
+											"""
+								}]
+							}
+						}
+					}
+				}
+			}
+		}
+	}
+}

content/docs/concept/working-with-incomplete-cue/page.cue

@@ -0,0 +1,3 @@
+package site
+
+content: docs: concept: "working-with-incomplete-cue": page: _