From c2b865f75d9d7d6da8fc538f5f95673490df743e Mon Sep 17 00:00:00 2001 From: Jonathan Matthews Date: Tue, 21 May 2024 15:38:31 +0100 Subject: [PATCH] docs/introduction: rewrite introduction This splits the ideas and features mentioned on the first page of /docs/introduction/ into a set of five pages that gives the absolute CUE newcomer enough information and context that they'll be motivated to continue learning by exploring the rest of the site. The pages are rewritten in line with the following aims: /docs/introduction/: - be friendly and approachable, as perhaps the first page that a CUE newcomer might read. - avoid CompSci terms, and any jargon that isn't essential. - immediately put some self-explanatory CUE in front of the reader, so they can get a hint of its capabilities, and aren't left wondering if it's some opaque, binary format; or if it's worse to handle than XML. - provide high-level scoping information for what the CUE project is, what resources it produces, and where those resources can be found. - communicate high-level information about what the user can achieve with CUE today, but without being exhaustively verbose. - include information and links to integration- and capability-specific pages, but without insisting that the reader follow those links immediately (i.e. "don't worry - they're there when you need them"). - give the reader multiple reasons to continue with the introduction by whetting their appetite with CUE characteristics and features. - hint at the transformative effect that CUE can have on folks' data and schema handling (whilst keeping it friendly and lighthearted!). /docs/introduction/cue-is-familiar/: - build on the positive "I want to do *that*!" feelings engendered by the first page - plant the seed of "I already know how to read&write CUE" in the reader's head. - reassure the reader that, whilst they're about to learn of some aspects of CUE that appear slightly different all these concepts exist within the context of a familiar-feeling language. - link to concept guides as appropriate, but infrequently (so the reader doesn't feel they're being strongly nudged to jump away from the intro before reaching the the "Why CUE?" convincer. /docs/introduction/cue-is-different/: - show the reader some concepts and new syntax: order irrelevance & unification; immutable data; types & constraints are values; and push don't pull constraints. - include multiple promises that the payoffs for these differences will be shown on the next page. /docs/introduction/why-cue/: - develops the concepts from cue-is-different through to their real-world impacts in a single, more fully-formed example. - demonstrate some of the concrete benefits that CUE users often reap and the reasons that they value adopting/using the language. - link to concept guides appropriately. /docs/introduction/effective-cue/: - take some of the unused "principles" from the original introduction page and ensure their lessons and impacts don't go unmentioned. It also adds a placeholder for the final page of the intro: "Next Steps" (to be completed in a follow-up). For cue-lang/docs-and-content#93 Preview-Path: /docs/introduction/ Preview-Path: /docs/introduction/cue-is-familiar/ Preview-Path: /docs/introduction/cue-is-different/ Preview-Path: /docs/introduction/why-cue/ Preview-Path: /docs/introduction/effective-cue/ Signed-off-by: Jonathan Matthews Change-Id: I84d24ba4218056957bc435fcc49537cb9f9ceef8 Dispatch-Trailer: {"type":"trybot","CL":1195016,"patchset":24,"ref":"refs/changes/16/1195016/24","targetBranch":"master"} --- content/docs/introduction/_en.md | 363 ++++-------------- .../docs/introduction/cue-is-different/en.md | 307 +++++++++++++++ .../cue-is-different/gen_cache.cue | 23 ++ .../introduction/cue-is-different/page.cue | 3 + .../docs/introduction/cue-is-familiar/en.md | 109 ++++++ .../cue-is-familiar/gen_cache.cue | 18 + .../introduction/cue-is-familiar/page.cue | 3 + content/docs/introduction/effective-cue/en.md | 130 +++++++ .../introduction/effective-cue/gen_cache.cue | 30 ++ .../docs/introduction/effective-cue/page.cue | 3 + content/docs/introduction/gen_cache.cue | 9 +- content/docs/introduction/history/en.md | 4 +- content/docs/introduction/installation/en.md | 4 +- content/docs/introduction/next-steps/en.md | 6 + content/docs/introduction/next-steps/page.cue | 3 + content/docs/introduction/why-cue/en.md | 283 ++++++++++++++ .../docs/introduction/why-cue/gen_cache.cue | 99 +++++ content/docs/introduction/why-cue/page.cue | 3 + hugo/content/en/docs/introduction/_index.md | 358 ++++------------- .../introduction/cue-is-different/index.md | 315 +++++++++++++++ .../introduction/cue-is-familiar/index.md | 109 ++++++ .../docs/introduction/effective-cue/index.md | 126 ++++++ .../en/docs/introduction/history/index.md | 4 +- .../docs/introduction/installation/index.md | 4 +- .../en/docs/introduction/next-steps/index.md | 6 + .../en/docs/introduction/why-cue/index.md | 297 ++++++++++++++ 26 files changed, 2042 insertions(+), 577 deletions(-) create mode 100644 content/docs/introduction/cue-is-different/en.md create mode 100644 content/docs/introduction/cue-is-different/gen_cache.cue create mode 100644 content/docs/introduction/cue-is-different/page.cue create mode 100644 content/docs/introduction/cue-is-familiar/en.md create mode 100644 content/docs/introduction/cue-is-familiar/gen_cache.cue create mode 100644 content/docs/introduction/cue-is-familiar/page.cue create mode 100644 content/docs/introduction/effective-cue/en.md create mode 100644 content/docs/introduction/effective-cue/gen_cache.cue create mode 100644 content/docs/introduction/effective-cue/page.cue create mode 100644 content/docs/introduction/next-steps/en.md create mode 100644 content/docs/introduction/next-steps/page.cue create mode 100644 content/docs/introduction/why-cue/en.md create mode 100644 content/docs/introduction/why-cue/gen_cache.cue create mode 100644 content/docs/introduction/why-cue/page.cue create mode 100644 hugo/content/en/docs/introduction/cue-is-different/index.md create mode 100644 hugo/content/en/docs/introduction/cue-is-familiar/index.md create mode 100644 hugo/content/en/docs/introduction/effective-cue/index.md create mode 100644 hugo/content/en/docs/introduction/next-steps/index.md create mode 100644 hugo/content/en/docs/introduction/why-cue/index.md diff --git a/content/docs/introduction/_en.md b/content/docs/introduction/_en.md index e89d4a4490..72132e01b4 100644 --- a/content/docs/introduction/_en.md +++ b/content/docs/introduction/_en.md @@ -8,289 +8,84 @@ aliases: - /docs/about --- -## Welcome! - -CUE is an open-source data validation language and inference engine -with its roots in logic programming. -Although the language is not a general-purpose programming language, -it has many applications, such as -data validation, data templating, configuration, querying, -code generation and even scripting. -The inference engine can be used to validate -data in code or to include it as part of a code generation pipeline. - -A key thing that sets CUE apart from its peer languages -is that it merges types and values into a single concept. -Whereas in most languages types and values are strictly distinct, -CUE orders them in a single hierarchy (a lattice, to be precise). -This is a very powerful concept that allows CUE to do -many fancy things. -It also simplifies matters. -For instance, there is no need for generics, and enums, sum types -and null coalescing are all the same thing. - - -## Applications - -CUE's design ensures that combining CUE values in any -order always gives the same result -(it is associative, commutative and idempotent). -This makes CUE particularly well-suited for cases where CUE -constraints are combined from different sources: - -- Data validation: different departments or groups can each -define their own constraints to apply to the same set of data. - -- Code extraction and generation: extract CUE definitions from -multiple sources (Go code, Protobuf), combine them into a single -definition, and use that to generate definitions in another -format (e.g. OpenAPI). - -- Configuration: values can be combined from different sources - without one having to import the other. - -The ordering of values also allows set containment analysis of entire -configurations. -Where most validation systems are limited to checking whether a concrete -value matches a schema, CUE can validate whether any instance of -one schema is also an instance of another (is it backwards compatible?), -or compute a new schema that represents all instances that match -two other schema. - -## Philosophy and principles - -### Types are Values - -CUE does not distinguish between values and types. -This is a powerful notion that allows CUE to define ultra-detailed -constraints, but it also simplifies things considerably: -there is no separate schema or data definition language to learn -and related language constructs such as sum types, enums, -and even null coalescing collapse onto a single construct. - -Below is a demonstration of this concept. -On the left one can see a JSON object (in CUE syntax) with some properties -about the city of Moscow. -The middle column shows a possible schema for any municipality. -On the right one sees a mix between data and schema as is exemplary of CUE. - -{{< columns >}} -Data -{{{with code "en" "data"}}} --- in.cue -- -moscow: { - name: "Moscow" - pop: 11.92M - capital: true -} -{{{end}}} -{{< columns-separator >}} -Schema -{{{with code "en" "schema"}}} --- in.cue -- -municipality: { - name: string - pop: int - capital: bool -} +### Welcome to CUE! + +CUE is an +open-source +data validation language with its roots in logic programming. +It combines succinct yet clear syntax with powerful, flexible constraints that +enable data, schema, policy, and constraints to coexist seamlessly: + +{{{with code "en" "example"}}} +#location left right +! exec cue vet example.cue +cmp stderr out +-- example.cue -- +length: 20 & int +width: 10.1 & >10 // Must be greater than 10 +area: length * width +area: <=100 // Must be less than or equal to 100 +-- out -- +area: invalid value 202.0 (out of bound <=100): + ./example.cue:4:9 + ./example.cue:3:9 {{{end}}} -{{< columns-separator >}} -CUE -{{{with code "en" "CUE"}}} --- in.cue -- -largeCapital: { - name: string - pop: >5M - capital: true -} -{{{end}}} -{{< /columns >}} - -In general, in CUE one starts with a broad definition of a type, describing -all possible instances. -One then narrows down these definitions, possibly by combining constraints -from different sources (departments, users), until a concrete data instance -remains. - - -### Push, not pull, constraints - -CUE's constraints act as data validators, but also double as -a mechanism to reduce boilerplate. -This is a powerful approach, but requires some different thinking. -With traditional inheritance approaches one specifies the templates that -are to be inherited from at each point they should be used. -In CUE, instead, one selects a set of nodes in the configuration to which -to apply a template. -This selection can be at a different point in the configuration altogether. - -Another way to view this, a JSON configuration, say, can be -defined as a sequence of path-leaf values. -For instance, -{{{with code "en" "json"}}} --- in.json -- -{ - "a": 3, - "b": { - "c": "foo" - } -} -{{{end}}} - -could be represented as - -{{{with code "en" "cue form of json"}}} --- in.cue -- -"a": 3 -"b": "c": "foo" -{{{end}}} - -All the information of the original JSON file is retained in this -representation. - -CUE generalizes this notion to the following pattern: -{{{with code "en" "nodes"}}} -#nofmt --- nodes.cue -- -: -{{{end}}} - -Each field declaration in CUE defines a set of nodes to which to apply -a specific constraint. -Because order doesn't matter, multiple constraints can be applied to the -same nodes, all of which need to apply simultaneously. -Such constraints may even be in different files. -But they may never contradict each other: -if one declaration says a field is `5`, another may not override it to be `6`. -Declaring a field to be both `>5` and `<10` is valid, though. - -This approach is more restricted than full-blown inheritance; -it may not be possible to reuse existing configurations. -On the other hand, it is also a more powerful boilerplate remover. -For instance, suppose each job in a set needs to use a specific -template. -Instead of having to spell this out at each point, -one can declare this separately in a one blanket statement. - -So instead of - -{{{with code "en" "non-dry"}}} --- in.cue -- -jobs: { - foo: acmeMonitoring & {...} - bar: acmeMonitoring & {...} - baz: acmeMonitoring & {...} -} -{{{end}}} - -one can write - -{{{with code "en" "dry"}}} --- in.cue -- -jobs: [string]: acmeMonitoring - -jobs: { - foo: {...} - bar: {...} - baz: {...} -} -{{{end}}} - -There is no need to repeat the reference to the monitoring template for -each job, as the first already states that all jobs _must_ use `acmeMonitoring`. -Such requirements can be specified across files. - -This approach not only reduces the boilerplate contained in `acmeMonitoring` -but also removes the repetitiveness of having to specify -this template for each job in `jobs`. -At the same time, this statement acts as a type enforcement. -This dual function is a key aspect of CUE and -typed feature structure languages in general. - -This approach breaks down, of course, if the restrictions in -`acmeMonitoring` are too stringent and jobs need to override them. -To this extent, CUE provides mechanisms to allow defaults, opt-out, and -soft constraints. - - -### Separate configuration from computation - -There comes a time that one (seemingly) will need do complex -computations to generate some configuration data. -But simplicity of a configuration language can be paramount when one quickly -needs to make changes. -These are obviously conflicting interests. - -CUE takes the stance that computation and configuration should -be separated. -And CUE actually makes this easy. -The data that needs to be computed can be generated outside of CUE -and put in a file that is to be mixed in. -The data can even be generated in CUE's scripting layer and automatically -injected in a configuration pipeline. -Both approaches rely on CUE's property that the order in which this data gets -added is irrelevant. - - - -### Be useful at all scales - -The usefulness of a language may depend on the scale of the project. -Having too many different languages can put a cognitive strain on -developers, though, and migrating from one language to another as -scaling requirements change can be very costly. -CUE aims to minimize these costs -by covering a myriad of data- and configuration-related tasks at all scales. - -**Small scale** -At small scales, reducing boilerplate in configurations is not necessarily -the best thing to do. -Even at a small scale, however, repetition can be error prone. -For such cases, CUE can define schema to validate otherwise -typeless data files. - -**Medium scale** -As soon the desire arises to reduce boilerplate, the `cue` tool can -help to automatically rewrite configurations. -See the Quick and Dirty section of the -[Kubernetes tutorial](https://github.com/cue-labs/cue-by-example/blob/main/003_kubernetes_tutorial/README.md) -for an example using the `import` and `trim` tool. -Thousands of lines can be obliterated automatically using this approach. - -**Large scale** -CUE's underlying formalism was developed for large-scale configuration. -Its import model incorporates best practices for large-scale engineering -and it is optimized for automation. -A key to this is advanced tooling. -The mathematical model underlying CUE's operations allows for -automation that is intractable for most other approaches. -CUE's `trim` command is an example of this. - - -### Tooling - -Automation is key. -Nowadays, a good chunk of code gets generated, analyzed, reformatted, -and so on by machines. -The CUE language, APIs, and tooling have been designed to allow for -machine manipulation. -Aspects of this are: - -- make the language easy to scan and parse, -- restrictions on imports, -- allow any piece of data to be split across files and generated - from different sources, -- define packages at the directory level, -- and of course its value and type model. - -The order independence also plays a key role in this. -It allows combining constraints from various sources without having -to define any order in which they are to be applied to get -predictable results. - - +In its mission to support people using the language and to promote its +adoption, the CUE project develops and publishes a variety of documentation and +tools, including: + +{{< table >}} +| Resource | Description +| --- | --- +| [The `cue` command]({{< relref "installation" >}}) | A command line tool that evaluates CUE, optionally combining it with structured data and other schema formats to validate, transform, and output data and constraints. +| [`cuelang.org/go` APIs](https://pkg.go.dev/cuelang.org/go/cue#section-documentation) | Go APIs that enable CUE's capabilities to be embedded in Go programs. +| [The CUE Language Specification]({{< ref "docs/reference/spec" >}}) | The formal specification of CUE that defines how implementations of the language should behave. +| [cuelang.org](/) | This website, including a foundational [tour through the language]({{< relref "/docs/tour" >}}), hands-on [tutorials]({{< relref "/docs/tutorial" >}}) and [how-to guides]({{< relref "/docs/howto" >}}), and informative [concept guides]({{< relref "/docs/concept" >}}). +| [The CUE Playground](/play/) | A browser-based tool that lets you try out CUE without installing anything. +{{< /table >}} +
+ +By design, CUE isn't a general-purpose programming language, +but its power and flexibility drive its use across a wide range of +applications. It's often used to define, validate and generate +[configuration]({{< relref "/docs/concept/how-cue-enables-configuration" >}}). +CUE also excels at +[validating data]({{< relref "/docs/concept/how-cue-enables-data-validation" >}}) +(such as JSON and YAML) against CUE schemas and policies, whilst also allowing schemas encoded in a variety of +[other formats]({{< relref "/docs/integration/" >}}) +(such as JSON Schema, Protobuf, and OpenAPI) to be used simultaneously. + +CUE's language features enables you to +[template data]({{< relref "/docs/tour/types/templates" >}}), +reducing boilerplate by specifying fields in bulk and allowing data's +important characteristics to stand out prominently. It's also used for +[code generation]({{< relref "/docs/concept/code-generation-and-extraction-use-case" >}}), +and to leverage existing schemas defined in formats such as +[Protobuf]({{< relref "/docs/concept/how-cue-works-with-protocol-buffers" >}}), +[JSON Schema]({{< relref "/docs/concept/how-cue-works-with-json-schema" >}}), +and +[Go]({{< relref "/docs/concept/how-cue-works-with-go" >}}) types. + +Over the next few pages you'll learn about some unique properties of the CUE +language, including: + +- why the merged concepts of "types" and "values" enable succinct and clear + constraints +- how some of CUE's core design principles combine so that the source of each + specific value is never in doubt -- no more hunting through confusing layers + of "overrides" to figure out which files disagree about a particular value +- the advanced tooling that's made possible by CUE's careful design, including + automated boilerplate removal! + + + +{{< warning >}} +Welcome to +}}" style="font-weight: normal;">the CUE community +-- but be warned ... \ +**Prolonged exposure to CUE can seriously affect how you approach data and configuration - for good!** +{{< /warning >}} + +*Next page:* [A Familiar Look and Feel]({{< relref "cue-is-familiar" >}}) diff --git a/content/docs/introduction/cue-is-different/en.md b/content/docs/introduction/cue-is-different/en.md new file mode 100644 index 0000000000..328b35f21d --- /dev/null +++ b/content/docs/introduction/cue-is-different/en.md @@ -0,0 +1,307 @@ +--- +title: Some Unique Differences +weight: 20 +--- + +As you just saw in +[*A Familiar Look and Feel*]({{< relref "cue-is-familiar" >}}), +CUE can be used to encode data using a friendlier, more convenient syntax than +JSON - and some folks do just that. However many teams also rely on the +language's advanced capabilities to validate and secure their data and +configurations, and these features build on some rather unique characteristics. + +Let's take a look at some aspects of CUE that you might not have +experienced in a language before ... + +### Order Doesn't Matter + +Most languages require variables, fields, or data to be declared before they're +used - especially when one data value depends on the value of another. + +**CUE is different:** in CUE, *fields can be defined in any order*. + +This property drives many of CUE's most powerful features, and is referred to +as *order irrelevance*. It applies at all levels of granularity: + +- within the fields of each data *struct* (which is what JSON calls an "object"), +- across the fields and structs defined inside a single `.cue` file, +- when merging multiple `.cue` files that make up a CUE *package*. + +Order irrelevance flows from the rules of CUE's most fundamental operation, +called **unification**. Unification is the process by which CUE determines if +values are compatible with one another. It occurs when you explicitly use +the `&` operator, or implicitly when you mention a field multiple times. + +In formal terms, unification is defined so that the operation is associative, +commutative, idempotent, and recursive. This means that when values are +unified, CUE guarantees that every possible order in which they *might* be +combined produces the same underlying data structure, and therefore it doesn't +matter which *specific* ordering is chosen - even when unifying deeply nested +data. + +In practical terms, unification's rules mean that: + +- Data is immutable: if a field is made concrete by assigning it a specific + value, that value is fixed and cannot be changed. (This might appear + restrictive at first glance, but in reality CUE gives you plenty of options + to cater for the different problematic situations you might be imagining!) +- Data and constraints can be combined from multiple sources predictably and + efficiently, optionally using a convenient shorthand form for specifying + sparsely-populated structs. +- If a field is declared more than once, then all its assigned values must be + compatible with each other. When only specifying concrete data, this + simplifies down: all the assigned values must be *identical*. + +In this example, `A` is specified using implicit unification, and `B` is +specified using explicit unification: + +{{{with code "en" "simple unification"}}} +exec cue export data.cue --out yaml +cmp stdout out +-- data.cue -- +A: 1 +B: 2 & 2 +A: 1 +-- out -- +A: 1 +B: 2 +{{{end}}} + +In this example you might be wondering if these unifications have any point - +surely no-one would *actually* specify\ +`B: 2 & 2`? Wouldn't `B: 2` be sufficient? + +In the case of the value of `B`, you'd be right to wonder - this is simply a +demonstration of explict unification so that you can recognise it, when it's +used later in more interesting situations. +Implicit unification, however (as with field `A`), becomes more interesting when +we learn that, by default, *the `cue` command implicitly unifies the top level +of all the data it's given*. +This means that when we invoke the `cue` command and tell it about different +sources of data, the evaluation result is the unification of all those sources. + +To see what this means in practice, we'll unify some CUE, JSON, and YAML, +simply by mentioning their three data files: + +{{{with code "en" "multi-file unification"}}} +#location top-left top-right bottom-left bottom-right +exec cue export data.cue data.yml data.json --out json +cmp stdout out +-- data.cue -- +CUE: true +a: b: c: 1 +-- data.yml -- +YAML: true +a: + b: + e: 3 +-- data.json -- +{ + "JSON": true, + "a": { + "b": { + "d": 2 + } + } +} +-- out -- +{ + "JSON": true, + "YAML": true, + "CUE": true, + "a": { + "b": { + "d": 2, + "e": 3, + "c": 1 + } + } +} +{{{end}}} + +But watch what happens when we try and specify a field with two incompatible +values - the `cue` command emits an error message pointing out the +incompatibility and the evaluation fails: + +{{{with code "en" "unification failure"}}} +! exec cue export data.cue data.json +cmp stderr out +-- data.cue -- +source: "CUE" +-- data.json -- +{ + "source": "JSON" +} +-- out -- +source: conflicting values "CUE" and "JSON": + ./data.cue:1:9 + ./data.json:2:15 +{{{end}}} + +CUE's design ensures that combining values in any order always gives the same +result. Later, in *Why CUE?*, we'll see some of the situations and use cases +that have elegant solutions enabled by this property. + +### Types Are Values + +In many languages there's a strong distinction between the concrete "values" +that a variable or field can be assigned (e.g. `"foo"`, `4.2`, or `true`) +and the "types" that describe *sets* of permissible values (e.g. strings, +floating-point numbers, or booleans, respectively). Most languages don't +provide a first-class mechanism to constrain values more precisely, leaving any +nuanced value constraints for the user to implement in code with explicit +runtime checks. + +**CUE is different:** *CUE doesn't differentiate between values and types*. + +This is a powerful concept that, as you'll see shortly, allows you to define +detailed constraints - but it also simplifies the process of learning and using +CUE. Because the language has a single, unified syntax for data and for +constraints, there is no separate schema or data definition language to learn. +The syntax even encompasses concepts such as sum types and enums - even null +coalescing is reduced down to this single construct. + +Whilst CUE does provide a type hierarchy that includes `string`, `float`, +and `bool`, along with `bytes`, `int`, `number`, `null`, `[...]` (list), and +`{...}` (struct), these are simply well-known names for constraints that limit +a field's value to well-defined sets or ranges. +The power of CUE comes from constructing precise constraints using these types +as a starting point, progressively layering and unifying additional constraints +built with CUE's rich set of primitives and built-in functions. + +Here's an example that uses +[bounds]({{< relref "docs/tour/types/bounds" >}}) (`>`, `<=`, etc) +to construct constraints that limit a number's value, unified with additional +constraints from a +[disjunction]({{< relref "/docs/tour/types/disjunctions" >}}) +("`|`") that requires a value to be one option from a prescribed set: + +{{{with code "en" "constraints are values too"}}} +#location left right +exec cue eval constraints.cue +cmp stdout out +-- constraints.cue -- +#over10: >10 +#under50: <50 +#from5To40: >=5 & <=40 +#options: 9 | 10 | 11 | 39 | 40 | 41 + +myNumber: int & #over10 & #under50 +myNumber: #from5To40 +myNumber: #options +-- out -- +#over10: >10 +#under50: <50 +#from5To40: >=5 & <=40 +#options: 9 | 10 | 11 | 39 | 40 | 41 +myNumber: 11 | 39 | 40 +{{{end}}} + +In the `cue eval` output, notice how CUE is able to *simplify* the constraints +that apply to the `myNumber` field by ruling out options that definitely aren't +possible, due to the implicit and explicit unification of the constraints that +we defined. + +Later, in *Why CUE?*, we'll see how CUE's merging of types, values, and +constraints into a single concept enables effective and concise schema and +policy validation. + +### Push Constraints, Don't Pull Them + +CUE's constraints act as data validators: + +{{{with code "en" "constraints validate data"}}} +#location top-left top-left top-right bottom +! exec cue vet schema.cue policy.cue data.yml +cmp stderr out +-- schema.cue -- +A: int +B: float +C: string +-- policy.cue -- +A: <=99 +B: >4.2 +C: !="rm -rf /" +-- data.yml -- +A: 100 +B: 1.1 +C: "rm -rf /" +-- out -- +A: invalid value 100 (out of bound <=99): + ./policy.cue:1:4 + ./data.yml:1:4 +B: invalid value 1.1 (out of bound >4.2): + ./policy.cue:2:4 + ./data.yml:2:4 +C: invalid value "rm -rf /" (out of bound !="rm -rf /"): + ./policy.cue:3:4 + ./data.yml:3:4 + ./schema.cue:3:4 +{{{end}}} + +*Pattern constraints* impose constraints on all the fields whose names match +their pattern. They're written as\ +`[pattern]: value`, where `pattern` must be either a value +of type string, or the wildcard value `_` (called "top"). Here's an +example of pattern constraints in action: + +{{{with code "en" "pattern constraints select fields"}}} +#location top-left top-right bottom +! exec cue vet constraints.cue data.yml +cmp stderr out +-- constraints.cue -- +A: { + // Every field must be an int. + [_]: int + // Every field whose name starts with b must + // be 10 or greater. + [=~"^b"]: >=10 +} +-- data.yml -- +A: + foo: 4.2 + bar: 100 + baz: 5 +-- out -- +A.foo: conflicting values 4.2 and int (mismatched types float and int): + ./constraints.cue:3:7 + ./data.yml:2:8 +A.baz: invalid value 5 (out of bound >=10): + ./constraints.cue:6:12 + ./data.yml:4:8 +{{{end}}} + +Pattern constraints *don't* instantiate every field that their pattern might +match. If they *did*, then the example pattern constraint of `[string]: >10` +would bring every field with a name matching `string` into existence - i.e. +*every possible field*, which would take an unacceptable amount of time to +compute ... no matter how powerful your hardware! +So: a pattern constraint merely acts on fields by unifying its constraints with +*existing* fields that match its pattern. + +Many languages have a form of import, or perhaps inheritance, that allows +lower-level (more deeply nested) components to pull restrictions in from some +other entity by specifying them at the point of use. + +**CUE is different:** *unification and pattern constraints offer a simple way +to impose constraints from above*. + +CUE has a robust +[package]({{< relref "/docs/concept/modules-packages-instances" >}}) system +and a carefully designed +[module]({{< relref "/docs/reference/modules" >}}) system +which support imports and cross-entity data and constraint sharing. +Before reaching for those tools, however, it's worth discovering just how +flexible and powerful pattern constraints and unification can be! + +In *Why CUE?* we'll see how pattern constraints are the backbone of a highly +productive mechanism for reducing boilerplate data and configuration, whilst +also enforcing more centralised policy constraints on nested data. + +{{< warning >}} +**On the next page**: discover how CUE's differences have a real impact on data +handling and validation, and why teams trust it with their configurations at +all scales. +{{< /warning >}} + +*Next page:* [Why CUE?]({{< relref "why-cue" >}}) diff --git a/content/docs/introduction/cue-is-different/gen_cache.cue b/content/docs/introduction/cue-is-different/gen_cache.cue new file mode 100644 index 0000000000..2374848afc --- /dev/null +++ b/content/docs/introduction/cue-is-different/gen_cache.cue @@ -0,0 +1,23 @@ +package site +{ + content: { + docs: { + introduction: { + "cue-is-different": { + page: { + cache: { + code: { + "simple unification": "ERKmiq2ADy1AiDb5xsK7i1JIu0PU1aEJXDYt825HJ2E=" + "multi-file unification": "tbtepD1lrMNpnswBf/kemS2TLAPNm5O2ZVId0SnHdtw=" + "unification failure": "yu+b7SRokzdjRJDV6+/t0z2PI99fML3yzdy1+GzpE7I=" + "constraints are values too": "hztyT4KduSV7OHCx/opQ6OF/nO+20VLiKCgOXzU+4ak=" + "constraints validate data": "gWtkffVaGOum9TGUkKqhJjrEYiPoSOnHKXVaMGadbqQ=" + "pattern constraints select fields": "5FEtxG9Ll23OoFPsKYSn0yyXiZGPK8ubjr2HU/F2iQY=" + } + } + } + } + } + } + } +} diff --git a/content/docs/introduction/cue-is-different/page.cue b/content/docs/introduction/cue-is-different/page.cue new file mode 100644 index 0000000000..b0deeb2027 --- /dev/null +++ b/content/docs/introduction/cue-is-different/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "cue-is-different": page: _ diff --git a/content/docs/introduction/cue-is-familiar/en.md b/content/docs/introduction/cue-is-familiar/en.md new file mode 100644 index 0000000000..24e590b76a --- /dev/null +++ b/content/docs/introduction/cue-is-familiar/en.md @@ -0,0 +1,109 @@ +--- +title: A Familiar Look and Feel +weight: 10 +--- + +CUE will probably feel rather familiar if you've spent any time working with +data. CUE shares some syntax with JSON, but *significantly* improves the +experience of managing JSON by hand. + +In its simplest form, CUE looks a lot like JSON. +This is because CUE is defined to be a *superset* of JSON - which means that +all valid JSON is CUE, but not vice versa. +Editing JSON manually can be somewhat awkward, so CUE introduces several +conveniences to make writing and reading data easier: + +- comments are allowed, starting with `//` and extending to the end of the line +- field names without special characters don’t need to be quoted +- the outermost curly braces in a CUE file are optional +- commas after a field are optional (and are usually omitted) +- commas after the final element of a list are allowed +- literal multiline strings are allowed, and don't require newlines to be encoded +- nested structs containing one (or a few) fields have a convenient shorthand + +Here's some data encoded as commented CUE, alongside the equivalent JSON +document. Notice how the CUE lacks curly braces at the top and bottom, and +doesn't have commas after each field's value: + +{{{with code "en" "CUE improves on JSON"}}} +#location left right +exec cue export example.cue --out json +cmp stdout out +-- example.cue -- +strings: { + singleLine: "Double quotes == string literal" + multiLine: """ + Multiline strings start and end with triple + double-quotes - no escaping of newlines! + """ +} + +// Many field names don't need to be quoted +// (but can be, if you want). +foo_Bar: 1 +baz2: 2.2 + +// Some field names do need quotes, such as those +// that start with numbers, or contain spaces, +// hyphens, or other special characters. +"qu ux": "3.33" +"4": "four" + +a: deeply: nested: field: "value" + +a: deeply: nested: struct: { + containing: "multiple" + fields: true +} + +// A list's final element can be followed by +// an optional comma. +aList: [ + "a", + "b", + "c", +] +anotherList: [1, 2, 3, 4, 5] +-- out -- +{ + "strings": { + "singleLine": "Double quotes == string literal", + "multiLine": "Multiline strings start and end with triple\ndouble-quotes - no escaping of newlines!" + }, + "foo_Bar": 1, + "baz2": 2.2, + "qu ux": "3.33", + "4": "four", + "a": { + "deeply": { + "nested": { + "field": "value", + "struct": { + "containing": "multiple", + "fields": true + } + } + } + }, + "aList": [ + "a", + "b", + "c" + ], + "anotherList": [ + 1, + 2, + 3, + 4, + 5 + ] +} +{{{end}}} + +However, most CUE users don't choose CUE simply because it's nicer to handle +than JSON - that's just a pragmatic consequence of the language's design. +Folks adopt CUE because of its revolutionary features, and these stem from its +concepts and syntaxes that *won't* seem quite as familiar. + +We'll take a look at them next, in +[*Some Unique Differences*]({{< relref "cue-is-different" >}}) ... diff --git a/content/docs/introduction/cue-is-familiar/gen_cache.cue b/content/docs/introduction/cue-is-familiar/gen_cache.cue new file mode 100644 index 0000000000..94533389f5 --- /dev/null +++ b/content/docs/introduction/cue-is-familiar/gen_cache.cue @@ -0,0 +1,18 @@ +package site +{ + content: { + docs: { + introduction: { + "cue-is-familiar": { + page: { + cache: { + code: { + "CUE improves on JSON": "ilglrIcCyZ9HAIpLjmMXPpq2nx9T+ljH7K7HpB3d+VE=" + } + } + } + } + } + } + } +} diff --git a/content/docs/introduction/cue-is-familiar/page.cue b/content/docs/introduction/cue-is-familiar/page.cue new file mode 100644 index 0000000000..2f903d1745 --- /dev/null +++ b/content/docs/introduction/cue-is-familiar/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "cue-is-familiar": page: _ diff --git a/content/docs/introduction/effective-cue/en.md b/content/docs/introduction/effective-cue/en.md new file mode 100644 index 0000000000..28e186ccaa --- /dev/null +++ b/content/docs/introduction/effective-cue/en.md @@ -0,0 +1,130 @@ +--- +title: Effective CUE +weight: 40 +--- + +CUE is designed to help users and teams manage data and configuration across +all scales. Here are some strategies that help make your use of CUE more +effective. + +### Start with broad definitions and get progressively more specific + +In general, using CUE, we start with a broad definition of a type that +describes all possible instances of the type. We then narrow down these +definitions, often combining constraints from multiple sources (e.g. different +departments, teams, and users), until a concrete data instance remains. + +Here's a demonstration of this at a small scale. Keep in mind that CUE permits +this approach to be used effectively at far larger scales: + +{{< columns >}} +{{{with upload "en" "schema"}}} +-- schema.cue -- +package geo + +#Municipality: { + name: string + pop: int + capital: bool +} +{{{end}}} +{{< columns-separator >}} +{{{with upload "en" "CUE"}}} +-- native.cue -- +package geo + +#LargeCapital: #Municipality & { + name: string + pop: >5M + capital: true +} +{{{end}}} +{{< columns-separator >}} +{{{with upload "en" "data"}}} +-- data.cue -- +package geo + +kinshasa: #LargeCapital & { + name: "Kinshasa" + pop: 16.315M + capital: true +} +{{{end}}} +{{< /columns >}} +{{{with _script_ "en" "HIDDEN: vet"}}} +cue vet .:geo +{{{end}}} + +### Separate configuration from computation + +Situations often arise where it looks like we'll need to do complex +computations to generate some configuration data. This approach is generally in +conflict with the goal of using a *simple* configuration language that permits +and encourages changes that can be understood in isolation, at speed, possibly +by non-domain experts. + +CUE has a stance that computation and configuration should be separated - and +*unification* makes this easy, because of *order irrelevance*. +Data that needs to be be computed can be generated outside CUE, and mixed into +a configuration using unification, CUE packages, and the `cue` command's +support for a variety of +[different data encodings]({{< relref "/docs/integration" >}}). + +The processes that compute this data can even be orchestrated by a CUE +[workflow command]({{< relref "docs/concept/how-cue-enables-configuration" >}}#tooling-and-automation), +enhancing developer workflows and performing inline computation input/output +validation. + +### Make appropriate choices for your level of scale + +The usefulness of a specific language can depend on the scale of the project +being developed, but adopting the "perfect" language for every new scale +milestone can put a cognitive strain on developers. CUE aims to minimize the +high cost of migrating from one language to another, as scaling requirements +change, by delivering solutions to data- and configuration-related tasks at all +scales: + +- **Small scale**: reducing boilerplate in configurations is not necessarily + the most effective thing to do. However, even at a small scale, repetition + can be tedious and lead to errors. For these situations, where it might be + best to keep small configurations specified and updated in isolation, CUE can + be used to define schemas that validate type-free + [JSON]({{< relref "/docs/howto/validate-json-using-cue" >}}) and + [YAML]({{< relref "/docs/howto/validate-yaml-using-cue" >}}) files. +- **Medium scale**: when the size or complexity of configurations reaches a + certain point, reducing boilerplate may start to become a more attractive + proposition. As you saw, earlier in this introduction, CUE can be used to + isolate such boilerplate, with the `cue trim` command removing it + automatically. It's not uncommon for thousands of lines to be obliterated + using this approach. +- **Large scale**: The + [formalism that underlies CUE](https://github.com/cue-lang/cue/blob/master/doc/ref/impl.md) + was specifically developed for large-scale configuration. CUE's import model + incorporates many best practices for large-scale engineering, and the + language is optimized for automation through advanced tooling. The + mathematical model and + [the logic of CUE]({{< relref "/docs/concept/the-logic-of-cue" >}}) + permits automation that's inaccessible for most other approaches - such as + `cue trim`'s capabilities. Read the next section, *Tooling*, for more on + this. + +### Know when to invest in building specialized tooling + +Automation is key. In contemporary operations, a large amount of code and +configuration gets generated, analyzed, reformatted, and mananged by machines. +The CUE language, its APIs, and its tooling have been designed to allow for +this mechanical manipulation, and you should anticipate taking advantage of +these capabilities as the scale and complexity of your mission grows. + +CUE's automation-friendly design is exemplified by these characteristics: + +- the language is easy to scan and parse +- there are specific restrictions on imports that prevent configuration graphs + of unbounded complexity +- data can be split across files and generated from different sources +- order irrelevance permits values and constraints to be mixed together easily, + without having to specify (or care about) the order in which they're combined +- packages are defined at the directory level +- its first-class value and type model is unparalleled. + +*Next page:* [Installing CUE]({{< relref "installation" >}}) diff --git a/content/docs/introduction/effective-cue/gen_cache.cue b/content/docs/introduction/effective-cue/gen_cache.cue new file mode 100644 index 0000000000..d71653263e --- /dev/null +++ b/content/docs/introduction/effective-cue/gen_cache.cue @@ -0,0 +1,30 @@ +package site +{ + content: { + docs: { + introduction: { + "effective-cue": { + page: { + cache: { + upload: { + schema: "w+C/Nm1ONZaP7j6C/G+0z2pHiY17uBKa5/mk4T0DpZs=" + CUE: "FyEuLRJEKmGzOcfzCzeI44B74JirC4SDfvJi+TJ6cl8=" + data: "1q6//vHnr13FVeLGlbl3yFKswopiKeof74YxZMurLb8=" + } + multi_step: { + hash: "8G2FHCBDJIRM1PGJ845QHSC80TJTS1BC2R2K024NI7USR98LEM6G====" + scriptHash: "6RGDEU0BIFNVMT9ULVRBMEHNALS8VTURLA4K2V446BOEH2OAGOJ0====" + steps: [{ + doc: "" + cmd: "cue vet .:geo" + exitCode: 0 + output: "" + }] + } + } + } + } + } + } + } +} diff --git a/content/docs/introduction/effective-cue/page.cue b/content/docs/introduction/effective-cue/page.cue new file mode 100644 index 0000000000..93f192953f --- /dev/null +++ b/content/docs/introduction/effective-cue/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "effective-cue": page: _ diff --git a/content/docs/introduction/gen_cache.cue b/content/docs/introduction/gen_cache.cue index f9654b63a8..9156e2873d 100644 --- a/content/docs/introduction/gen_cache.cue +++ b/content/docs/introduction/gen_cache.cue @@ -6,14 +6,7 @@ package site page: { cache: { code: { - data: "4Zfi0CluF9D48XBPNGO15jRjrct5uflTY9H1wydh77c=" - schema: "FF1B/AyuZly5hTGBjM0DjLFbs8IbeOhPKI+vUbb8c50=" - CUE: "/w2kdYYs1NphuKX/zUrIULP4H9LIKL0CcaZ38bJyQ0g=" - json: "LTxsCHJq6dVJE5EJMIF5VRVDo58SOGUG70wvb0fmQCo=" - "cue form of json": "N3fanrD6FTObi+gUwQAygoBlpNlse/MP8K9aOW93ZK0=" - nodes: "Q9jCAjAC8qT7cWV7KT07OivXhb7nFAx+IbY1zVHZU3Q=" - "non-dry": "lgc6B7iZxN4q03EIaoFRVJ4GEdwk3u0fJkp7zDUBf/o=" - dry: "r9GVMGnuqrFz4c20lzT5zWEo7R9JtALCfvRhxfQ4cs0=" + example: "Upd/bTuqJ3DDUc172utkfnh80U7Ks83RbeEMb/dwUAI=" } } } diff --git a/content/docs/introduction/history/en.md b/content/docs/introduction/history/en.md index a416962fb5..4bbff195b3 100644 --- a/content/docs/introduction/history/en.md +++ b/content/docs/introduction/history/en.md @@ -1,6 +1,6 @@ --- -title: History -weight: 40 +title: The History of CUE +weight: 60 --- Although it is a very different language, the roots of CUE lie in GCL, diff --git a/content/docs/introduction/installation/en.md b/content/docs/introduction/installation/en.md index 83f2e782be..c8a13916d9 100644 --- a/content/docs/introduction/installation/en.md +++ b/content/docs/introduction/installation/en.md @@ -1,6 +1,6 @@ --- -title: Installation -weight: 20 +title: Installing CUE +weight: 50 aliases: - /docs/install - /download diff --git a/content/docs/introduction/next-steps/en.md b/content/docs/introduction/next-steps/en.md new file mode 100644 index 0000000000..3dd4907aa7 --- /dev/null +++ b/content/docs/introduction/next-steps/en.md @@ -0,0 +1,6 @@ +--- +title: Next Steps +weight: 70 +--- + +FIXME: words diff --git a/content/docs/introduction/next-steps/page.cue b/content/docs/introduction/next-steps/page.cue new file mode 100644 index 0000000000..ceb2c3c560 --- /dev/null +++ b/content/docs/introduction/next-steps/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "next-steps": page: _ diff --git a/content/docs/introduction/why-cue/en.md b/content/docs/introduction/why-cue/en.md new file mode 100644 index 0000000000..08c001bbdc --- /dev/null +++ b/content/docs/introduction/why-cue/en.md @@ -0,0 +1,283 @@ +--- +title: Why CUE? +weight: 30 +--- + +The features that make CUE so effective are the result of its careful design, +which is the product of decades of experience in the data and cofiguration +space. The language's behaviours are formally defined in +[The CUE Language Specification]({{< relref "/docs/reference/spec" >}}), +but you don't need to study the spec in order to understand how CUE can make +life easier. + +If you work frequently with data or configuration, the CUE examples in this +introduction might have already given you ideas about how you could use CUE to +simplify, fortify, or otherwise improve your existing setup. +But before jumping in and getting started, it's worth reading through this +site's concept guides on how CUE enables various practices, such as +[configuration]({{< relref "/docs/concept/how-cue-enables-configuration" >}}) +and +[data validation]({{< relref "/docs/concept/how-cue-enables-data-validation" >}}), +and +[how CUE works with different technologies]({{< relref "/docs/integration" >}}). + +As further inspiration, here a more fully-formed example that demonstrates the +concepts and features that users of CUE often find compelling and powerful ... + +### Combining schema, policy, and data, and reducing boilerplate + +You've already seen how *unification* combines data with multiple, layered +constraints that limit the data's acceptable values. You saw how *schema* ("X +is an integer") can live seamlessly alongside *policy constraints* ("X must be +greater than 10") because of CUE's succinct and clear syntax. The *pattern +constraint* feature gives you a way to push down constraints from above, +instead of relying on the authors of nested data to reference your requirements. + +**Here's a demonstration of these concepts in action, working together to drive down complexity**. +Let's start with a hypothetical schema describing the contents of a +website -- its pages and structure -- along with some separate policy +constraints in `policy.cue`: + +{{{with upload "en" "schema"}}} +-- schema.cue -- +package website + +#Page: { + title!: string // title is a required field, as every page must have a title. + urlPath!: string // urlPath is also required. + file!: string // The page's file contains its content. + date!: string // The page's date of publishing. + isDraft!: bool // Is the page part of the published site? + summary?: string // summary is an optional field, as some pages don't need a short summary. + author?: string | null // An optional author name. +} +-- policy.cue -- +package website + +import ( + "strings" + "time" +) + +#Page: { + title?: strings.MinRunes(1) // title cannot be empty (could also be specified as !=""). + urlPath?: strings.MinRunes(1) // urlPath cannot be empty (equivalent to !=""). + file?: =~".html$" | =~".md$" // Content files can be HTML or Markdown. + date?: time.Time // time.Time validates a RFC3339 date-time. + summary?: strings.MaxRunes(150) // Our site layout requires page summaries to be limited in length. + author?: _ // Policy imposes no additional constraints on the author. + isDraft?: _ +} +{{{end}}} + +Here we've used several new language elements, particularly in `policy.cue`: + +- required (`!:`) and optional (`?:`) fields +- regular expression constraints (`=~`) +- importing built-in packages containing the CUE standard library (`import ( ... )`) +- using the standard library (`strings.MinRunes()`) + + +These features are explored and explained in the +[CUE language tour]({{< relref "/docs/tour" >}}) +but -- just for now! -- continue by reading on with the assumption that they +all function as you might expect from their names. + +Now we've set up some schema and policy constraints, we can start to populate +information about our small website's existing pages, marking them with a +publishing date that's in the past to reflect their "already published" state: + +{{{with upload "en" "data"}}} +#codetab(site.cue) linenos="table" +-- site.cue -- +package website + +// pages is a struct whose fields each adhere to the +// #Page constraint imposed through a pattern constraint. +pages: [_]: #Page +pages: { + home: { + title: "Welcome to Widgets'R'We!" + urlPath: "/" + file: "pages/home.html" + summary: "The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: null + } + about: { + title: "About Widgets'R'We" + urlPath: "/about/" + file: "pages/about.html" + summary: "Information about the Widgets'R'We company." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: null + } + contact: { + title: "Contact Widgets'R'We" + urlPath: "/contact/" + file: "pages/contact.html" + summary: "Contact information for the Widgets'R'We company." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: "Markus Marketing" + } +} +{{{end}}} + +We can run `cue vet` to check that each of our pages' data meets the +requirements of the `#Page` constraint. +Like many command line tools, it stays silent to indicate success: + +{{{with script "en" "vet 1"}}} +cue vet +{{{end}}} + +Whilst our site metadata is correctly specified, it's hard not to feel that it +seems a little *unwieldy*. There's quite a lot of duplicated information that +distracts from the important detail of each page - often called "boilerplate". +Let's use CUE to reduce it ... + +We'll use various CUE features to make our configuration shorter, which will +allow the most important information to stand out. Here's the CUE we'll use: + +{{{with upload "en" "defaults"}}} +-- pageDefaults.cue -- +package website + +pages?: [pageId=_]: { + isDraft: _ | *false + author: _ | *null + date: _ | *"1999-01-01T00:00:00Z" + urlPath: _ | *"/\(pageId)/" + file: _ | *"pages/\(pageId).html" +} +{{{end}}} + +In `pageDefaults.cue`, we: + +- use a [pattern constraint]({{< relref "/docs/tour/basics/folding-structs" >}}) + to unify fields inside each member of `pages`, creating what CUE calls a + [template]({{< relref "/docs/tour/types/templates" >}}), +- use the template to assign five fields a + [default value]({{< relref "/docs/tour/types/defaults" >}}) - a value that + only takes effect if a configuration doesn't specify a value elsewhere, +- bring an [alias]({{< relref "/docs/tour/references/aliases" >}}) into + existence (`pageId`) that contains the value of the identifier of each member + of the `pages` struct, and allows us to refer to each page's identifier + *inside* the template. This lets us derive two string fields' default values + dynamically. + +We again use `cue vet`, confirming that *adding* this CUE file to our `website` +package hasn't caused a problem: + +{{{with script "en" "vet 2"}}} +cue vet +{{{end}}} + +Next we need to *remove* the duplicate data from our `site.cue` metadata file - +a tedious and error-prone task, especially if our set of pages were larger, or +were split across more files. Or it *would* be tedious, if it weren't for CUE! + +The `cue trim` command performs **automatic removal of boilerplate** from +configurations where the data can be inferred from other constraints. +Let's try it out here: + +{{{with script "en" "trim"}}} +cue trim +{{{end}}} + +`cue trim`, just as with `cue vet`, stays silent if it was successful. Let's +take a look at the updated `site.cue` metadata file: + +{{{with _script_ "en" "HIDDEN: move before diff"}}} +mv site.cue .site.cue +{{{end}}} + +{{{with upload "en" "site.cue 2"}}} +#force +#codetab(site.cue) linenos="table" +-- site.cue -- +package website + +// pages is a struct whose fields each adhere to the +// #Page constraint imposed through a pattern constraint. +pages: [_]: #Page +pages: { + home: { + title: "Welcome to Widgets'R'We!" + urlPath: "/" + summary: "The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East." + } + about: { + title: "About Widgets'R'We" + summary: "Information about the Widgets'R'We company." + } + contact: { + title: "Contact Widgets'R'We" + summary: "Contact information for the Widgets'R'We company." + author: "Markus Marketing" + } +} +{{{end}}} + +{{{with _script_ "en" "HIDDEN: diff"}}} +diff site.cue .site.cue +{{{end}}} + +Only the notable, *important* information about each page is left behind, with +the boring-but-necessary boilerplate configuration safely isolated in the +`pageDefaults.cue` file. + +Let's use `cue vet` again to confirm that the `#Page` schema still validates +our site metadata after the boilerplate was removed, and then run `cue export` +to show us the concrete data result of our changes: + +{{{with script "en" "check & export"}}} +cue vet +cue export --out yaml +{{{end}}} + +We can see that we exported the same data that we started with ... but with a +significantly clearer and more succinct representation behind the scenes! +Lastly, we'll introduce a new CUE file to the `website` package that *attempts* +to modify an existing page's title: + +{{{with upload "en" "broken update"}}} +#codetab(pages.cue) linenos="table" +-- updates.cue -- +package website + +pages: { + about: title: "Cyber Widgets 2000" +} +{{{end}}} + +The `cue vet` command complains, because the title already has a concrete value +specified elsewhere and data is immutable in CUE. +Notice that it tells us the locations of the conflicting values (encoded as +`filename:line-number:character-number`) so that we can find and resolve this +kind of mistake faster and easier: + +{{{with script "en" "vet 3"}}} +! cue vet +{{{end}}} + +Here's a reminder of the concepts and features that we just used: + +- *Constraints* were *unified* across the multiple files in our CUE package. +- *Pattern constraints* allowed us to push constraints down from above. +- *Templates* and *default values* gave us a way to centralise repetitive + information without repeating it. +- *Aliases* gave us way to construct a template's field values dynamically. +- `cue vet` checked that data adhered to its constraints. +- `cue export` produced concrete data. +- `cue trim` automatically removed duplicate information from our data source. + +{{< warning >}} +**On the next page**: learn some strategies to make effective use of CUE. +{{< /warning >}} + +*Next page:* [Effective CUE]({{< relref "effective-cue" >}}) diff --git a/content/docs/introduction/why-cue/gen_cache.cue b/content/docs/introduction/why-cue/gen_cache.cue new file mode 100644 index 0000000000..72046c78fa --- /dev/null +++ b/content/docs/introduction/why-cue/gen_cache.cue @@ -0,0 +1,99 @@ +package site +{ + content: { + docs: { + introduction: { + "why-cue": { + page: { + cache: { + upload: { + schema: "bkSu4SfR6p8fnR3i3FqzVMmbbvRaFFo+XQu/0p7eo2s=" + data: "fg/Dobcd1DdHyFGgIkwb/GGO1vl/DRflw9jxiPUn9kA=" + defaults: "FqCQ/RoZxiDTS75kU0MEoC9hmCl+roC3f3647PaKZ4I=" + "site.cue 2": "FRF0Naux/D9DWm9Rv8jZ94OJFaMNWPIYkPmQEdxGDSk=" + "broken update": "QyykkM9yNtNFHibOfAOalun1pD7iQaD0l+N49D6klJA=" + } + multi_step: { + hash: "KAI6J8EMGANTVIM7EF1B28VTGD2A8K2RC178TA9844U02891N2R0====" + scriptHash: "CTCHP9D7H7K2OMJFPBHMK9OP8J609364U36R5VJBDHGVBQBC39P0====" + steps: [{ + doc: "" + cmd: "cue vet" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "cue vet" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "cue trim" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "mv site.cue .site.cue" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "diff site.cue .site.cue" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "cue vet" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "cue export --out yaml" + exitCode: 0 + output: """ + pages: + home: + title: Welcome to Widgets'R'We! + urlPath: / + summary: The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East. + isDraft: false + author: null + date: "1999-01-01T00:00:00Z" + file: pages/home.html + about: + title: About Widgets'R'We + summary: Information about the Widgets'R'We company. + isDraft: false + author: null + date: "1999-01-01T00:00:00Z" + urlPath: /about/ + file: pages/about.html + contact: + title: Contact Widgets'R'We + summary: Contact information for the Widgets'R'We company. + author: Markus Marketing + isDraft: false + date: "1999-01-01T00:00:00Z" + urlPath: /contact/ + file: pages/contact.html + + """ + }, { + doc: "" + cmd: "cue vet" + exitCode: 1 + output: """ + pages.about.title: conflicting values "Cyber Widgets 2000" and "About Widgets'R'We": + ./site.cue:13:12 + ./updates.cue:4:16 + + """ + }] + } + } + } + } + } + } + } +} diff --git a/content/docs/introduction/why-cue/page.cue b/content/docs/introduction/why-cue/page.cue new file mode 100644 index 0000000000..98d68365cf --- /dev/null +++ b/content/docs/introduction/why-cue/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "why-cue": page: _ diff --git a/hugo/content/en/docs/introduction/_index.md b/hugo/content/en/docs/introduction/_index.md index 3c26f358dd..5914fd0b1f 100644 --- a/hugo/content/en/docs/introduction/_index.md +++ b/hugo/content/en/docs/introduction/_index.md @@ -8,280 +8,84 @@ aliases: - /docs/about --- -## Welcome! - -CUE is an open-source data validation language and inference engine -with its roots in logic programming. -Although the language is not a general-purpose programming language, -it has many applications, such as -data validation, data templating, configuration, querying, -code generation and even scripting. -The inference engine can be used to validate -data in code or to include it as part of a code generation pipeline. - -A key thing that sets CUE apart from its peer languages -is that it merges types and values into a single concept. -Whereas in most languages types and values are strictly distinct, -CUE orders them in a single hierarchy (a lattice, to be precise). -This is a very powerful concept that allows CUE to do -many fancy things. -It also simplifies matters. -For instance, there is no need for generics, and enums, sum types -and null coalescing are all the same thing. - - -## Applications - -CUE's design ensures that combining CUE values in any -order always gives the same result -(it is associative, commutative and idempotent). -This makes CUE particularly well-suited for cases where CUE -constraints are combined from different sources: - -- Data validation: different departments or groups can each -define their own constraints to apply to the same set of data. - -- Code extraction and generation: extract CUE definitions from -multiple sources (Go code, Protobuf), combine them into a single -definition, and use that to generate definitions in another -format (e.g. OpenAPI). - -- Configuration: values can be combined from different sources - without one having to import the other. - -The ordering of values also allows set containment analysis of entire -configurations. -Where most validation systems are limited to checking whether a concrete -value matches a schema, CUE can validate whether any instance of -one schema is also an instance of another (is it backwards compatible?), -or compute a new schema that represents all instances that match -two other schema. - -## Philosophy and principles - -### Types are Values - -CUE does not distinguish between values and types. -This is a powerful notion that allows CUE to define ultra-detailed -constraints, but it also simplifies things considerably: -there is no separate schema or data definition language to learn -and related language constructs such as sum types, enums, -and even null coalescing collapse onto a single construct. - -Below is a demonstration of this concept. -On the left one can see a JSON object (in CUE syntax) with some properties -about the city of Moscow. -The middle column shows a possible schema for any municipality. -On the right one sees a mix between data and schema as is exemplary of CUE. - -{{< columns >}} -Data -```cue -moscow: { - name: "Moscow" - pop: 11.92M - capital: true -} -``` -{{< columns-separator >}} -Schema -```cue -municipality: { - name: string - pop: int - capital: bool -} -``` -{{< columns-separator >}} -CUE -```cue -largeCapital: { - name: string - pop: >5M - capital: true -} -``` -{{< /columns >}} - -In general, in CUE one starts with a broad definition of a type, describing -all possible instances. -One then narrows down these definitions, possibly by combining constraints -from different sources (departments, users), until a concrete data instance -remains. - - -### Push, not pull, constraints - -CUE's constraints act as data validators, but also double as -a mechanism to reduce boilerplate. -This is a powerful approach, but requires some different thinking. -With traditional inheritance approaches one specifies the templates that -are to be inherited from at each point they should be used. -In CUE, instead, one selects a set of nodes in the configuration to which -to apply a template. -This selection can be at a different point in the configuration altogether. - -Another way to view this, a JSON configuration, say, can be -defined as a sequence of path-leaf values. -For instance, -```json -{ - "a": 3, - "b": { - "c": "foo" - } -} -``` - -could be represented as - -```cue -"a": 3 -"b": "c": "foo" -``` - -All the information of the original JSON file is retained in this -representation. - -CUE generalizes this notion to the following pattern: -```cue -: -``` - -Each field declaration in CUE defines a set of nodes to which to apply -a specific constraint. -Because order doesn't matter, multiple constraints can be applied to the -same nodes, all of which need to apply simultaneously. -Such constraints may even be in different files. -But they may never contradict each other: -if one declaration says a field is `5`, another may not override it to be `6`. -Declaring a field to be both `>5` and `<10` is valid, though. - -This approach is more restricted than full-blown inheritance; -it may not be possible to reuse existing configurations. -On the other hand, it is also a more powerful boilerplate remover. -For instance, suppose each job in a set needs to use a specific -template. -Instead of having to spell this out at each point, -one can declare this separately in a one blanket statement. - -So instead of - -```cue -jobs: { - foo: acmeMonitoring & {...} - bar: acmeMonitoring & {...} - baz: acmeMonitoring & {...} -} -``` - -one can write - -```cue -jobs: [string]: acmeMonitoring - -jobs: { - foo: {...} - bar: {...} - baz: {...} -} -``` - -There is no need to repeat the reference to the monitoring template for -each job, as the first already states that all jobs _must_ use `acmeMonitoring`. -Such requirements can be specified across files. - -This approach not only reduces the boilerplate contained in `acmeMonitoring` -but also removes the repetitiveness of having to specify -this template for each job in `jobs`. -At the same time, this statement acts as a type enforcement. -This dual function is a key aspect of CUE and -typed feature structure languages in general. - -This approach breaks down, of course, if the restrictions in -`acmeMonitoring` are too stringent and jobs need to override them. -To this extent, CUE provides mechanisms to allow defaults, opt-out, and -soft constraints. - - -### Separate configuration from computation - -There comes a time that one (seemingly) will need do complex -computations to generate some configuration data. -But simplicity of a configuration language can be paramount when one quickly -needs to make changes. -These are obviously conflicting interests. - -CUE takes the stance that computation and configuration should -be separated. -And CUE actually makes this easy. -The data that needs to be computed can be generated outside of CUE -and put in a file that is to be mixed in. -The data can even be generated in CUE's scripting layer and automatically -injected in a configuration pipeline. -Both approaches rely on CUE's property that the order in which this data gets -added is irrelevant. - - - -### Be useful at all scales - -The usefulness of a language may depend on the scale of the project. -Having too many different languages can put a cognitive strain on -developers, though, and migrating from one language to another as -scaling requirements change can be very costly. -CUE aims to minimize these costs -by covering a myriad of data- and configuration-related tasks at all scales. - -**Small scale** -At small scales, reducing boilerplate in configurations is not necessarily -the best thing to do. -Even at a small scale, however, repetition can be error prone. -For such cases, CUE can define schema to validate otherwise -typeless data files. - -**Medium scale** -As soon the desire arises to reduce boilerplate, the `cue` tool can -help to automatically rewrite configurations. -See the Quick and Dirty section of the -[Kubernetes tutorial](https://github.com/cue-labs/cue-by-example/blob/main/003_kubernetes_tutorial/README.md) -for an example using the `import` and `trim` tool. -Thousands of lines can be obliterated automatically using this approach. - -**Large scale** -CUE's underlying formalism was developed for large-scale configuration. -Its import model incorporates best practices for large-scale engineering -and it is optimized for automation. -A key to this is advanced tooling. -The mathematical model underlying CUE's operations allows for -automation that is intractable for most other approaches. -CUE's `trim` command is an example of this. - - -### Tooling - -Automation is key. -Nowadays, a good chunk of code gets generated, analyzed, reformatted, -and so on by machines. -The CUE language, APIs, and tooling have been designed to allow for -machine manipulation. -Aspects of this are: - -- make the language easy to scan and parse, -- restrictions on imports, -- allow any piece of data to be split across files and generated - from different sources, -- define packages at the directory level, -- and of course its value and type model. - -The order independence also plays a key role in this. -It allows combining constraints from various sources without having -to define any order in which they are to be applied to get -predictable results. - - - +### Welcome to CUE! + +CUE is an +open-source +data validation language with its roots in logic programming. +It combines succinct yet clear syntax with powerful, flexible constraints that +enable data, schema, policy, and constraints to coexist seamlessly: + +{{< code-tabs >}} +{{< code-tab name="example.cue" language="cue" area="left" >}} +length: 20 & int +width: 10.1 & >10 // Must be greater than 10 +area: length * width +area: <=100 // Must be less than or equal to 100 +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="right" type="terminal" codetocopy="Y3VlIHZldCBleGFtcGxlLmN1ZQ==" >}} +$ cue vet example.cue +area: invalid value 202.0 (out of bound <=100): + ./example.cue:4:9 + ./example.cue:3:9 +{{< /code-tab >}} +{{< /code-tabs >}} + +In its mission to support people using the language and to promote its +adoption, the CUE project develops and publishes a variety of documentation and +tools, including: + +{{< table >}} +| Resource | Description +| --- | --- +| [The `cue` command]({{< relref "installation" >}}) | A command line tool that evaluates CUE, optionally combining it with structured data and other schema formats to validate, transform, and output data and constraints. +| [`cuelang.org/go` APIs](https://pkg.go.dev/cuelang.org/go/cue#section-documentation) | Go APIs that enable CUE's capabilities to be embedded in Go programs. +| [The CUE Language Specification]({{< ref "docs/reference/spec" >}}) | The formal specification of CUE that defines how implementations of the language should behave. +| [cuelang.org](/) | This website, including a foundational [tour through the language]({{< relref "/docs/tour" >}}), hands-on [tutorials]({{< relref "/docs/tutorial" >}}) and [how-to guides]({{< relref "/docs/howto" >}}), and informative [concept guides]({{< relref "/docs/concept" >}}). +| [The CUE Playground](/play/) | A browser-based tool that lets you try out CUE without installing anything. +{{< /table >}} +
+ +By design, CUE isn't a general-purpose programming language, +but its power and flexibility drive its use across a wide range of +applications. It's often used to define, validate and generate +[configuration]({{< relref "/docs/concept/how-cue-enables-configuration" >}}). +CUE also excels at +[validating data]({{< relref "/docs/concept/how-cue-enables-data-validation" >}}) +(such as JSON and YAML) against CUE schemas and policies, whilst also allowing schemas encoded in a variety of +[other formats]({{< relref "/docs/integration/" >}}) +(such as JSON Schema, Protobuf, and OpenAPI) to be used simultaneously. + +CUE's language features enables you to +[template data]({{< relref "/docs/tour/types/templates" >}}), +reducing boilerplate by specifying fields in bulk and allowing data's +important characteristics to stand out prominently. It's also used for +[code generation]({{< relref "/docs/concept/code-generation-and-extraction-use-case" >}}), +and to leverage existing schemas defined in formats such as +[Protobuf]({{< relref "/docs/concept/how-cue-works-with-protocol-buffers" >}}), +[JSON Schema]({{< relref "/docs/concept/how-cue-works-with-json-schema" >}}), +and +[Go]({{< relref "/docs/concept/how-cue-works-with-go" >}}) types. + +Over the next few pages you'll learn about some unique properties of the CUE +language, including: + +- why the merged concepts of "types" and "values" enable succinct and clear + constraints +- how some of CUE's core design principles combine so that the source of each + specific value is never in doubt -- no more hunting through confusing layers + of "overrides" to figure out which files disagree about a particular value +- the advanced tooling that's made possible by CUE's careful design, including + automated boilerplate removal! + + + +{{< warning >}} +Welcome to +}}" style="font-weight: normal;">the CUE community +-- but be warned ... \ +**Prolonged exposure to CUE can seriously affect how you approach data and configuration - for good!** +{{< /warning >}} + +*Next page:* [A Familiar Look and Feel]({{< relref "cue-is-familiar" >}}) diff --git a/hugo/content/en/docs/introduction/cue-is-different/index.md b/hugo/content/en/docs/introduction/cue-is-different/index.md new file mode 100644 index 0000000000..f19b613fa6 --- /dev/null +++ b/hugo/content/en/docs/introduction/cue-is-different/index.md @@ -0,0 +1,315 @@ +--- +title: Some Unique Differences +weight: 20 +--- + +As you just saw in +[*A Familiar Look and Feel*]({{< relref "cue-is-familiar" >}}), +CUE can be used to encode data using a friendlier, more convenient syntax than +JSON - and some folks do just that. However many teams also rely on the +language's advanced capabilities to validate and secure their data and +configurations, and these features build on some rather unique characteristics. + +Let's take a look at some aspects of CUE that you might not have +experienced in a language before ... + +### Order Doesn't Matter + +Most languages require variables, fields, or data to be declared before they're +used - especially when one data value depends on the value of another. + +**CUE is different:** in CUE, *fields can be defined in any order*. + +This property drives many of CUE's most powerful features, and is referred to +as *order irrelevance*. It applies at all levels of granularity: + +- within the fields of each data *struct* (which is what JSON calls an "object"), +- across the fields and structs defined inside a single `.cue` file, +- when merging multiple `.cue` files that make up a CUE *package*. + +Order irrelevance flows from the rules of CUE's most fundamental operation, +called **unification**. Unification is the process by which CUE determines if +values are compatible with one another. It occurs when you explicitly use +the `&` operator, or implicitly when you mention a field multiple times. + +In formal terms, unification is defined so that the operation is associative, +commutative, idempotent, and recursive. This means that when values are +unified, CUE guarantees that every possible order in which they *might* be +combined produces the same underlying data structure, and therefore it doesn't +matter which *specific* ordering is chosen - even when unifying deeply nested +data. + +In practical terms, unification's rules mean that: + +- Data is immutable: if a field is made concrete by assigning it a specific + value, that value is fixed and cannot be changed. (This might appear + restrictive at first glance, but in reality CUE gives you plenty of options + to cater for the different problematic situations you might be imagining!) +- Data and constraints can be combined from multiple sources predictably and + efficiently, optionally using a convenient shorthand form for specifying + sparsely-populated structs. +- If a field is declared more than once, then all its assigned values must be + compatible with each other. When only specifying concrete data, this + simplifies down: all the assigned values must be *identical*. + +In this example, `A` is specified using implicit unification, and `B` is +specified using explicit unification: + +{{< code-tabs >}} +{{< code-tab name="data.cue" language="cue" area="top-left" >}} +A: 1 +B: 2 & 2 +A: 1 +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="top-right" type="terminal" codetocopy="Y3VlIGV4cG9ydCBkYXRhLmN1ZSAtLW91dCB5YW1s" >}} +$ cue export data.cue --out yaml +A: 1 +B: 2 +{{< /code-tab >}} +{{< /code-tabs >}} + +In this example you might be wondering if these unifications have any point - +surely no-one would *actually* specify\ +`B: 2 & 2`? Wouldn't `B: 2` be sufficient? + +In the case of the value of `B`, you'd be right to wonder - this is simply a +demonstration of explict unification so that you can recognise it, when it's +used later in more interesting situations. +Implicit unification, however (as with field `A`), becomes more interesting when +we learn that, by default, *the `cue` command implicitly unifies the top level +of all the data it's given*. +This means that when we invoke the `cue` command and tell it about different +sources of data, the evaluation result is the unification of all those sources. + +To see what this means in practice, we'll unify some CUE, JSON, and YAML, +simply by mentioning their three data files: + +{{< code-tabs >}} +{{< code-tab name="data.cue" language="cue" area="top-left" >}} +CUE: true +a: b: c: 1 +{{< /code-tab >}} +{{< code-tab name="data.yml" language="yml" area="top-right" >}} +YAML: true +a: + b: + e: 3 +{{< /code-tab >}} +{{< code-tab name="data.json" language="json" area="bottom-left" >}} +{ + "JSON": true, + "a": { + "b": { + "d": 2 + } + } +} +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="bottom-right" type="terminal" codetocopy="Y3VlIGV4cG9ydCBkYXRhLmN1ZSBkYXRhLnltbCBkYXRhLmpzb24gLS1vdXQganNvbg==" >}} +$ cue export data.cue data.yml data.json --out json +{ + "JSON": true, + "YAML": true, + "CUE": true, + "a": { + "b": { + "d": 2, + "e": 3, + "c": 1 + } + } +} +{{< /code-tab >}} +{{< /code-tabs >}} + +But watch what happens when we try and specify a field with two incompatible +values - the `cue` command emits an error message pointing out the +incompatibility and the evaluation fails: + +{{< code-tabs >}} +{{< code-tab name="data.cue" language="cue" area="top-left" >}} +source: "CUE" +{{< /code-tab >}} +{{< code-tab name="data.json" language="json" area="top-right" >}} +{ + "source": "JSON" +} +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="bottom" type="terminal" codetocopy="Y3VlIGV4cG9ydCBkYXRhLmN1ZSBkYXRhLmpzb24=" >}} +$ cue export data.cue data.json +source: conflicting values "CUE" and "JSON": + ./data.cue:1:9 + ./data.json:2:15 +{{< /code-tab >}} +{{< /code-tabs >}} + +CUE's design ensures that combining values in any order always gives the same +result. Later, in *Why CUE?*, we'll see some of the situations and use cases +that have elegant solutions enabled by this property. + +### Types Are Values + +In many languages there's a strong distinction between the concrete "values" +that a variable or field can be assigned (e.g. `"foo"`, `4.2`, or `true`) +and the "types" that describe *sets* of permissible values (e.g. strings, +floating-point numbers, or booleans, respectively). Most languages don't +provide a first-class mechanism to constrain values more precisely, leaving any +nuanced value constraints for the user to implement in code with explicit +runtime checks. + +**CUE is different:** *CUE doesn't differentiate between values and types*. + +This is a powerful concept that, as you'll see shortly, allows you to define +detailed constraints - but it also simplifies the process of learning and using +CUE. Because the language has a single, unified syntax for data and for +constraints, there is no separate schema or data definition language to learn. +The syntax even encompasses concepts such as sum types and enums - even null +coalescing is reduced down to this single construct. + +Whilst CUE does provide a type hierarchy that includes `string`, `float`, +and `bool`, along with `bytes`, `int`, `number`, `null`, `[...]` (list), and +`{...}` (struct), these are simply well-known names for constraints that limit +a field's value to well-defined sets or ranges. +The power of CUE comes from constructing precise constraints using these types +as a starting point, progressively layering and unifying additional constraints +built with CUE's rich set of primitives and built-in functions. + +Here's an example that uses +[bounds]({{< relref "docs/tour/types/bounds" >}}) (`>`, `<=`, etc) +to construct constraints that limit a number's value, unified with additional +constraints from a +[disjunction]({{< relref "/docs/tour/types/disjunctions" >}}) +("`|`") that requires a value to be one option from a prescribed set: + +{{< code-tabs >}} +{{< code-tab name="constraints.cue" language="cue" area="left" >}} +#over10: >10 +#under50: <50 +#from5To40: >=5 & <=40 +#options: 9 | 10 | 11 | 39 | 40 | 41 + +myNumber: int & #over10 & #under50 +myNumber: #from5To40 +myNumber: #options +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="right" type="terminal" codetocopy="Y3VlIGV2YWwgY29uc3RyYWludHMuY3Vl" >}} +$ cue eval constraints.cue +#over10: >10 +#under50: <50 +#from5To40: >=5 & <=40 +#options: 9 | 10 | 11 | 39 | 40 | 41 +myNumber: 11 | 39 | 40 +{{< /code-tab >}} +{{< /code-tabs >}} + +In the `cue eval` output, notice how CUE is able to *simplify* the constraints +that apply to the `myNumber` field by ruling out options that definitely aren't +possible, due to the implicit and explicit unification of the constraints that +we defined. + +Later, in *Why CUE?*, we'll see how CUE's merging of types, values, and +constraints into a single concept enables effective and concise schema and +policy validation. + +### Push Constraints, Don't Pull Them + +CUE's constraints act as data validators: + +{{< code-tabs >}} +{{< code-tab name="schema.cue" language="cue" area="top-left" >}} +A: int +B: float +C: string +{{< /code-tab >}} +{{< code-tab name="policy.cue" language="cue" area="top-left" >}} +A: <=99 +B: >4.2 +C: !="rm -rf /" +{{< /code-tab >}} +{{< code-tab name="data.yml" language="yml" area="top-right" >}} +A: 100 +B: 1.1 +C: "rm -rf /" +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="bottom" type="terminal" codetocopy="Y3VlIHZldCBzY2hlbWEuY3VlIHBvbGljeS5jdWUgZGF0YS55bWw=" >}} +$ cue vet schema.cue policy.cue data.yml +A: invalid value 100 (out of bound <=99): + ./policy.cue:1:4 + ./data.yml:1:4 +B: invalid value 1.1 (out of bound >4.2): + ./policy.cue:2:4 + ./data.yml:2:4 +C: invalid value "rm -rf /" (out of bound !="rm -rf /"): + ./policy.cue:3:4 + ./data.yml:3:4 + ./schema.cue:3:4 +{{< /code-tab >}} +{{< /code-tabs >}} + +*Pattern constraints* impose constraints on all the fields whose names match +their pattern. They're written as\ +`[pattern]: value`, where `pattern` must be either a value +of type string, or the wildcard value `_` (called "top"). Here's an +example of pattern constraints in action: + +{{< code-tabs >}} +{{< code-tab name="constraints.cue" language="cue" area="top-left" >}} +A: { + // Every field must be an int. + [_]: int + // Every field whose name starts with b must + // be 10 or greater. + [=~"^b"]: >=10 +} +{{< /code-tab >}} +{{< code-tab name="data.yml" language="yml" area="top-right" >}} +A: + foo: 4.2 + bar: 100 + baz: 5 +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="bottom" type="terminal" codetocopy="Y3VlIHZldCBjb25zdHJhaW50cy5jdWUgZGF0YS55bWw=" >}} +$ cue vet constraints.cue data.yml +A.foo: conflicting values 4.2 and int (mismatched types float and int): + ./constraints.cue:3:7 + ./data.yml:2:8 +A.baz: invalid value 5 (out of bound >=10): + ./constraints.cue:6:12 + ./data.yml:4:8 +{{< /code-tab >}} +{{< /code-tabs >}} + +Pattern constraints *don't* instantiate every field that their pattern might +match. If they *did*, then the example pattern constraint of `[string]: >10` +would bring every field with a name matching `string` into existence - i.e. +*every possible field*, which would take an unacceptable amount of time to +compute ... no matter how powerful your hardware! +So: a pattern constraint merely acts on fields by unifying its constraints with +*existing* fields that match its pattern. + +Many languages have a form of import, or perhaps inheritance, that allows +lower-level (more deeply nested) components to pull restrictions in from some +other entity by specifying them at the point of use. + +**CUE is different:** *unification and pattern constraints offer a simple way +to impose constraints from above*. + +CUE has a robust +[package]({{< relref "/docs/concept/modules-packages-instances" >}}) system +and a carefully designed +[module]({{< relref "/docs/reference/modules" >}}) system +which support imports and cross-entity data and constraint sharing. +Before reaching for those tools, however, it's worth discovering just how +flexible and powerful pattern constraints and unification can be! + +In *Why CUE?* we'll see how pattern constraints are the backbone of a highly +productive mechanism for reducing boilerplate data and configuration, whilst +also enforcing more centralised policy constraints on nested data. + +{{< warning >}} +**On the next page**: discover how CUE's differences have a real impact on data +handling and validation, and why teams trust it with their configurations at +all scales. +{{< /warning >}} + +*Next page:* [Why CUE?]({{< relref "why-cue" >}}) diff --git a/hugo/content/en/docs/introduction/cue-is-familiar/index.md b/hugo/content/en/docs/introduction/cue-is-familiar/index.md new file mode 100644 index 0000000000..03015f6ab6 --- /dev/null +++ b/hugo/content/en/docs/introduction/cue-is-familiar/index.md @@ -0,0 +1,109 @@ +--- +title: A Familiar Look and Feel +weight: 10 +--- + +CUE will probably feel rather familiar if you've spent any time working with +data. CUE shares some syntax with JSON, but *significantly* improves the +experience of managing JSON by hand. + +In its simplest form, CUE looks a lot like JSON. +This is because CUE is defined to be a *superset* of JSON - which means that +all valid JSON is CUE, but not vice versa. +Editing JSON manually can be somewhat awkward, so CUE introduces several +conveniences to make writing and reading data easier: + +- comments are allowed, starting with `//` and extending to the end of the line +- field names without special characters don’t need to be quoted +- the outermost curly braces in a CUE file are optional +- commas after a field are optional (and are usually omitted) +- commas after the final element of a list are allowed +- literal multiline strings are allowed, and don't require newlines to be encoded +- nested structs containing one (or a few) fields have a convenient shorthand + +Here's some data encoded as commented CUE, alongside the equivalent JSON +document. Notice how the CUE lacks curly braces at the top and bottom, and +doesn't have commas after each field's value: + +{{< code-tabs >}} +{{< code-tab name="example.cue" language="cue" area="left" >}} +strings: { + singleLine: "Double quotes == string literal" + multiLine: """ + Multiline strings start and end with triple + double-quotes - no escaping of newlines! + """ +} + +// Many field names don't need to be quoted +// (but can be, if you want). +foo_Bar: 1 +baz2: 2.2 + +// Some field names do need quotes, such as those +// that start with numbers, or contain spaces, +// hyphens, or other special characters. +"qu ux": "3.33" +"4": "four" + +a: deeply: nested: field: "value" + +a: deeply: nested: struct: { + containing: "multiple" + fields: true +} + +// A list's final element can be followed by +// an optional comma. +aList: [ + "a", + "b", + "c", +] +anotherList: [1, 2, 3, 4, 5] +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="right" type="terminal" codetocopy="Y3VlIGV4cG9ydCBleGFtcGxlLmN1ZSAtLW91dCBqc29u" >}} +$ cue export example.cue --out json +{ + "strings": { + "singleLine": "Double quotes == string literal", + "multiLine": "Multiline strings start and end with triple\ndouble-quotes - no escaping of newlines!" + }, + "foo_Bar": 1, + "baz2": 2.2, + "qu ux": "3.33", + "4": "four", + "a": { + "deeply": { + "nested": { + "field": "value", + "struct": { + "containing": "multiple", + "fields": true + } + } + } + }, + "aList": [ + "a", + "b", + "c" + ], + "anotherList": [ + 1, + 2, + 3, + 4, + 5 + ] +} +{{< /code-tab >}} +{{< /code-tabs >}} + +However, most CUE users don't choose CUE simply because it's nicer to handle +than JSON - that's just a pragmatic consequence of the language's design. +Folks adopt CUE because of its revolutionary features, and these stem from its +concepts and syntaxes that *won't* seem quite as familiar. + +We'll take a look at them next, in +[*Some Unique Differences*]({{< relref "cue-is-different" >}}) ... diff --git a/hugo/content/en/docs/introduction/effective-cue/index.md b/hugo/content/en/docs/introduction/effective-cue/index.md new file mode 100644 index 0000000000..551062aa5b --- /dev/null +++ b/hugo/content/en/docs/introduction/effective-cue/index.md @@ -0,0 +1,126 @@ +--- +title: Effective CUE +weight: 40 +--- + +CUE is designed to help users and teams manage data and configuration across +all scales. Here are some strategies that help make your use of CUE more +effective. + +### Start with broad definitions and get progressively more specific + +In general, using CUE, we start with a broad definition of a type that +describes all possible instances of the type. We then narrow down these +definitions, often combining constraints from multiple sources (e.g. different +departments, teams, and users), until a concrete data instance remains. + +Here's a demonstration of this at a small scale. Keep in mind that CUE permits +this approach to be used effectively at far larger scales: + +{{< columns >}} +{{< code-tabs >}} +{{< code-tab name="schema.cue" language="cue" area="top-left" >}} +package geo + +#Municipality: { + name: string + pop: int + capital: bool +} +{{< /code-tab >}}{{< /code-tabs >}} +{{< columns-separator >}} +{{< code-tabs >}} +{{< code-tab name="native.cue" language="cue" area="top-left" >}} +package geo + +#LargeCapital: #Municipality & { + name: string + pop: >5M + capital: true +} +{{< /code-tab >}}{{< /code-tabs >}} +{{< columns-separator >}} +{{< code-tabs >}} +{{< code-tab name="data.cue" language="cue" area="top-left" >}} +package geo + +kinshasa: #LargeCapital & { + name: "Kinshasa" + pop: 16.315M + capital: true +} +{{< /code-tab >}}{{< /code-tabs >}} +{{< /columns >}} +### Separate configuration from computation + +Situations often arise where it looks like we'll need to do complex +computations to generate some configuration data. This approach is generally in +conflict with the goal of using a *simple* configuration language that permits +and encourages changes that can be understood in isolation, at speed, possibly +by non-domain experts. + +CUE has a stance that computation and configuration should be separated - and +*unification* makes this easy, because of *order irrelevance*. +Data that needs to be be computed can be generated outside CUE, and mixed into +a configuration using unification, CUE packages, and the `cue` command's +support for a variety of +[different data encodings]({{< relref "/docs/integration" >}}). + +The processes that compute this data can even be orchestrated by a CUE +[workflow command]({{< relref "docs/concept/how-cue-enables-configuration" >}}#tooling-and-automation), +enhancing developer workflows and performing inline computation input/output +validation. + +### Make appropriate choices for your level of scale + +The usefulness of a specific language can depend on the scale of the project +being developed, but adopting the "perfect" language for every new scale +milestone can put a cognitive strain on developers. CUE aims to minimize the +high cost of migrating from one language to another, as scaling requirements +change, by delivering solutions to data- and configuration-related tasks at all +scales: + +- **Small scale**: reducing boilerplate in configurations is not necessarily + the most effective thing to do. However, even at a small scale, repetition + can be tedious and lead to errors. For these situations, where it might be + best to keep small configurations specified and updated in isolation, CUE can + be used to define schemas that validate type-free + [JSON]({{< relref "/docs/howto/validate-json-using-cue" >}}) and + [YAML]({{< relref "/docs/howto/validate-yaml-using-cue" >}}) files. +- **Medium scale**: when the size or complexity of configurations reaches a + certain point, reducing boilerplate may start to become a more attractive + proposition. As you saw, earlier in this introduction, CUE can be used to + isolate such boilerplate, with the `cue trim` command removing it + automatically. It's not uncommon for thousands of lines to be obliterated + using this approach. +- **Large scale**: The + [formalism that underlies CUE](https://github.com/cue-lang/cue/blob/master/doc/ref/impl.md) + was specifically developed for large-scale configuration. CUE's import model + incorporates many best practices for large-scale engineering, and the + language is optimized for automation through advanced tooling. The + mathematical model and + [the logic of CUE]({{< relref "/docs/concept/the-logic-of-cue" >}}) + permits automation that's inaccessible for most other approaches - such as + `cue trim`'s capabilities. Read the next section, *Tooling*, for more on + this. + +### Know when to invest in building specialized tooling + +Automation is key. In contemporary operations, a large amount of code and +configuration gets generated, analyzed, reformatted, and mananged by machines. +The CUE language, its APIs, and its tooling have been designed to allow for +this mechanical manipulation, and you should anticipate taking advantage of +these capabilities as the scale and complexity of your mission grows. + +CUE's automation-friendly design is exemplified by these characteristics: + +- the language is easy to scan and parse +- there are specific restrictions on imports that prevent configuration graphs + of unbounded complexity +- data can be split across files and generated from different sources +- order irrelevance permits values and constraints to be mixed together easily, + without having to specify (or care about) the order in which they're combined +- packages are defined at the directory level +- its first-class value and type model is unparalleled. + +*Next page:* [Installing CUE]({{< relref "installation" >}}) diff --git a/hugo/content/en/docs/introduction/history/index.md b/hugo/content/en/docs/introduction/history/index.md index a416962fb5..4bbff195b3 100644 --- a/hugo/content/en/docs/introduction/history/index.md +++ b/hugo/content/en/docs/introduction/history/index.md @@ -1,6 +1,6 @@ --- -title: History -weight: 40 +title: The History of CUE +weight: 60 --- Although it is a very different language, the roots of CUE lie in GCL, diff --git a/hugo/content/en/docs/introduction/installation/index.md b/hugo/content/en/docs/introduction/installation/index.md index a14346548e..1c3516a421 100644 --- a/hugo/content/en/docs/introduction/installation/index.md +++ b/hugo/content/en/docs/introduction/installation/index.md @@ -1,6 +1,6 @@ --- -title: Installation -weight: 20 +title: Installing CUE +weight: 50 aliases: - /docs/install - /download diff --git a/hugo/content/en/docs/introduction/next-steps/index.md b/hugo/content/en/docs/introduction/next-steps/index.md new file mode 100644 index 0000000000..3dd4907aa7 --- /dev/null +++ b/hugo/content/en/docs/introduction/next-steps/index.md @@ -0,0 +1,6 @@ +--- +title: Next Steps +weight: 70 +--- + +FIXME: words diff --git a/hugo/content/en/docs/introduction/why-cue/index.md b/hugo/content/en/docs/introduction/why-cue/index.md new file mode 100644 index 0000000000..7d27558745 --- /dev/null +++ b/hugo/content/en/docs/introduction/why-cue/index.md @@ -0,0 +1,297 @@ +--- +title: Why CUE? +weight: 30 +--- + +The features that make CUE so effective are the result of its careful design, +which is the product of decades of experience in the data and cofiguration +space. The language's behaviours are formally defined in +[The CUE Language Specification]({{< relref "/docs/reference/spec" >}}), +but you don't need to study the spec in order to understand how CUE can make +life easier. + +If you work frequently with data or configuration, the CUE examples in this +introduction might have already given you ideas about how you could use CUE to +simplify, fortify, or otherwise improve your existing setup. +But before jumping in and getting started, it's worth reading through this +site's concept guides on how CUE enables various practices, such as +[configuration]({{< relref "/docs/concept/how-cue-enables-configuration" >}}) +and +[data validation]({{< relref "/docs/concept/how-cue-enables-data-validation" >}}), +and +[how CUE works with different technologies]({{< relref "/docs/integration" >}}). + +As further inspiration, here a more fully-formed example that demonstrates the +concepts and features that users of CUE often find compelling and powerful ... + +### Combining schema, policy, and data, and reducing boilerplate + +You've already seen how *unification* combines data with multiple, layered +constraints that limit the data's acceptable values. You saw how *schema* ("X +is an integer") can live seamlessly alongside *policy constraints* ("X must be +greater than 10") because of CUE's succinct and clear syntax. The *pattern +constraint* feature gives you a way to push down constraints from above, +instead of relying on the authors of nested data to reference your requirements. + +**Here's a demonstration of these concepts in action, working together to drive down complexity**. +Let's start with a hypothetical schema describing the contents of a +website -- its pages and structure -- along with some separate policy +constraints in `policy.cue`: + +{{< code-tabs >}} +{{< code-tab name="schema.cue" language="cue" area="top-left" >}} +package website + +#Page: { + title!: string // title is a required field, as every page must have a title. + urlPath!: string // urlPath is also required. + file!: string // The page's file contains its content. + date!: string // The page's date of publishing. + isDraft!: bool // Is the page part of the published site? + summary?: string // summary is an optional field, as some pages don't need a short summary. + author?: string | null // An optional author name. +} +{{< /code-tab >}}{{< code-tab name="policy.cue" language="cue" area="top-left" >}} +package website + +import ( + "strings" + "time" +) + +#Page: { + title?: strings.MinRunes(1) // title cannot be empty (could also be specified as !=""). + urlPath?: strings.MinRunes(1) // urlPath cannot be empty (equivalent to !=""). + file?: =~".html$" | =~".md$" // Content files can be HTML or Markdown. + date?: time.Time // time.Time validates a RFC3339 date-time. + summary?: strings.MaxRunes(150) // Our site layout requires page summaries to be limited in length. + author?: _ // Policy imposes no additional constraints on the author. + isDraft?: _ +} +{{< /code-tab >}}{{< /code-tabs >}} + +Here we've used several new language elements, particularly in `policy.cue`: + +- required (`!:`) and optional (`?:`) fields +- regular expression constraints (`=~`) +- importing built-in packages containing the CUE standard library (`import ( ... )`) +- using the standard library (`strings.MinRunes()`) + + +These features are explored and explained in the +[CUE language tour]({{< relref "/docs/tour" >}}) +but -- just for now! -- continue by reading on with the assumption that they +all function as you might expect from their names. + +Now we've set up some schema and policy constraints, we can start to populate +information about our small website's existing pages, marking them with a +publishing date that's in the past to reflect their "already published" state: + +{{< code-tabs >}} +{{< code-tab name="site.cue" language="cue" area="top-left" linenos="table" >}} +package website + +// pages is a struct whose fields each adhere to the +// #Page constraint imposed through a pattern constraint. +pages: [_]: #Page +pages: { + home: { + title: "Welcome to Widgets'R'We!" + urlPath: "/" + file: "pages/home.html" + summary: "The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: null + } + about: { + title: "About Widgets'R'We" + urlPath: "/about/" + file: "pages/about.html" + summary: "Information about the Widgets'R'We company." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: null + } + contact: { + title: "Contact Widgets'R'We" + urlPath: "/contact/" + file: "pages/contact.html" + summary: "Contact information for the Widgets'R'We company." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: "Markus Marketing" + } +} +{{< /code-tab >}}{{< /code-tabs >}} + +We can run `cue vet` to check that each of our pages' data meets the +requirements of the `#Page` constraint. +Like many command line tools, it stays silent to indicate success: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHZldA==" } +$ cue vet +``` + +Whilst our site metadata is correctly specified, it's hard not to feel that it +seems a little *unwieldy*. There's quite a lot of duplicated information that +distracts from the important detail of each page - often called "boilerplate". +Let's use CUE to reduce it ... + +We'll use various CUE features to make our configuration shorter, which will +allow the most important information to stand out. Here's the CUE we'll use: + +{{< code-tabs >}} +{{< code-tab name="pageDefaults.cue" language="cue" area="top-left" >}} +package website + +pages?: [pageId=_]: { + isDraft: _ | *false + author: _ | *null + date: _ | *"1999-01-01T00:00:00Z" + urlPath: _ | *"/\(pageId)/" + file: _ | *"pages/\(pageId).html" +} +{{< /code-tab >}}{{< /code-tabs >}} + +In `pageDefaults.cue`, we: + +- use a [pattern constraint]({{< relref "/docs/tour/basics/folding-structs" >}}) + to unify fields inside each member of `pages`, creating what CUE calls a + [template]({{< relref "/docs/tour/types/templates" >}}), +- use the template to assign five fields a + [default value]({{< relref "/docs/tour/types/defaults" >}}) - a value that + only takes effect if a configuration doesn't specify a value elsewhere, +- bring an [alias]({{< relref "/docs/tour/references/aliases" >}}) into + existence (`pageId`) that contains the value of the identifier of each member + of the `pages` struct, and allows us to refer to each page's identifier + *inside* the template. This lets us derive two string fields' default values + dynamically. + +We again use `cue vet`, confirming that *adding* this CUE file to our `website` +package hasn't caused a problem: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHZldA==" } +$ cue vet +``` + +Next we need to *remove* the duplicate data from our `site.cue` metadata file - +a tedious and error-prone task, especially if our set of pages were larger, or +were split across more files. Or it *would* be tedious, if it weren't for CUE! + +The `cue trim` command performs **automatic removal of boilerplate** from +configurations where the data can be inferred from other constraints. +Let's try it out here: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHRyaW0=" } +$ cue trim +``` + +`cue trim`, just as with `cue vet`, stays silent if it was successful. Let's +take a look at the updated `site.cue` metadata file: +{{< code-tabs >}} +{{< code-tab name="site.cue" language="cue" area="top-left" linenos="table" >}} +package website + +// pages is a struct whose fields each adhere to the +// #Page constraint imposed through a pattern constraint. +pages: [_]: #Page +pages: { + home: { + title: "Welcome to Widgets'R'We!" + urlPath: "/" + summary: "The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East." + } + about: { + title: "About Widgets'R'We" + summary: "Information about the Widgets'R'We company." + } + contact: { + title: "Contact Widgets'R'We" + summary: "Contact information for the Widgets'R'We company." + author: "Markus Marketing" + } +} +{{< /code-tab >}}{{< /code-tabs >}} +Only the notable, *important* information about each page is left behind, with +the boring-but-necessary boilerplate configuration safely isolated in the +`pageDefaults.cue` file. + +Let's use `cue vet` again to confirm that the `#Page` schema still validates +our site metadata after the boilerplate was removed, and then run `cue export` +to show us the concrete data result of our changes: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHZldApjdWUgZXhwb3J0IC0tb3V0IHlhbWw=" } +$ cue vet +$ cue export --out yaml +pages: + home: + title: Welcome to Widgets'R'We! + urlPath: / + summary: The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East. + isDraft: false + author: null + date: "1999-01-01T00:00:00Z" + file: pages/home.html + about: + title: About Widgets'R'We + summary: Information about the Widgets'R'We company. + isDraft: false + author: null + date: "1999-01-01T00:00:00Z" + urlPath: /about/ + file: pages/about.html + contact: + title: Contact Widgets'R'We + summary: Contact information for the Widgets'R'We company. + author: Markus Marketing + isDraft: false + date: "1999-01-01T00:00:00Z" + urlPath: /contact/ + file: pages/contact.html +``` + +We can see that we exported the same data that we started with ... but with a +significantly clearer and more succinct representation behind the scenes! +Lastly, we'll introduce a new CUE file to the `website` package that *attempts* +to modify an existing page's title: + +{{< code-tabs >}} +{{< code-tab name="updates.cue" language="cue" area="top-left" >}} +package website + +pages: { + about: title: "Cyber Widgets 2000" +} +{{< /code-tab >}}{{< /code-tabs >}} + +The `cue vet` command complains, because the title already has a concrete value +specified elsewhere and data is immutable in CUE. +Notice that it tells us the locations of the conflicting values (encoded as +`filename:line-number:character-number`) so that we can find and resolve this +kind of mistake faster and easier: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHZldA==" } +$ cue vet +pages.about.title: conflicting values "Cyber Widgets 2000" and "About Widgets'R'We": + ./site.cue:13:12 + ./updates.cue:4:16 +``` + +Here's a reminder of the concepts and features that we just used: + +- *Constraints* were *unified* across the multiple files in our CUE package. +- *Pattern constraints* allowed us to push constraints down from above. +- *Templates* and *default values* gave us a way to centralise repetitive + information without repeating it. +- *Aliases* gave us way to construct a template's field values dynamically. +- `cue vet` checked that data adhered to its constraints. +- `cue export` produced concrete data. +- `cue trim` automatically removed duplicate information from our data source. + +{{< warning >}} +**On the next page**: learn some strategies to make effective use of CUE. +{{< /warning >}} + +*Next page:* [Effective CUE]({{< relref "effective-cue" >}})