diff --git a/content/docs/introduction/_en.md b/content/docs/introduction/_en.md index e89d4a449..72132e01b 100644 --- a/content/docs/introduction/_en.md +++ b/content/docs/introduction/_en.md @@ -8,289 +8,84 @@ aliases: - /docs/about --- -## Welcome! - -CUE is an open-source data validation language and inference engine -with its roots in logic programming. -Although the language is not a general-purpose programming language, -it has many applications, such as -data validation, data templating, configuration, querying, -code generation and even scripting. -The inference engine can be used to validate -data in code or to include it as part of a code generation pipeline. - -A key thing that sets CUE apart from its peer languages -is that it merges types and values into a single concept. -Whereas in most languages types and values are strictly distinct, -CUE orders them in a single hierarchy (a lattice, to be precise). -This is a very powerful concept that allows CUE to do -many fancy things. -It also simplifies matters. -For instance, there is no need for generics, and enums, sum types -and null coalescing are all the same thing. - - -## Applications - -CUE's design ensures that combining CUE values in any -order always gives the same result -(it is associative, commutative and idempotent). -This makes CUE particularly well-suited for cases where CUE -constraints are combined from different sources: - -- Data validation: different departments or groups can each -define their own constraints to apply to the same set of data. - -- Code extraction and generation: extract CUE definitions from -multiple sources (Go code, Protobuf), combine them into a single -definition, and use that to generate definitions in another -format (e.g. OpenAPI). - -- Configuration: values can be combined from different sources - without one having to import the other. - -The ordering of values also allows set containment analysis of entire -configurations. -Where most validation systems are limited to checking whether a concrete -value matches a schema, CUE can validate whether any instance of -one schema is also an instance of another (is it backwards compatible?), -or compute a new schema that represents all instances that match -two other schema. - -## Philosophy and principles - -### Types are Values - -CUE does not distinguish between values and types. -This is a powerful notion that allows CUE to define ultra-detailed -constraints, but it also simplifies things considerably: -there is no separate schema or data definition language to learn -and related language constructs such as sum types, enums, -and even null coalescing collapse onto a single construct. - -Below is a demonstration of this concept. -On the left one can see a JSON object (in CUE syntax) with some properties -about the city of Moscow. -The middle column shows a possible schema for any municipality. -On the right one sees a mix between data and schema as is exemplary of CUE. - -{{< columns >}} -Data -{{{with code "en" "data"}}} --- in.cue -- -moscow: { - name: "Moscow" - pop: 11.92M - capital: true -} -{{{end}}} -{{< columns-separator >}} -Schema -{{{with code "en" "schema"}}} --- in.cue -- -municipality: { - name: string - pop: int - capital: bool -} +### Welcome to CUE! + +CUE is an +open-source +data validation language with its roots in logic programming. +It combines succinct yet clear syntax with powerful, flexible constraints that +enable data, schema, policy, and constraints to coexist seamlessly: + +{{{with code "en" "example"}}} +#location left right +! exec cue vet example.cue +cmp stderr out +-- example.cue -- +length: 20 & int +width: 10.1 & >10 // Must be greater than 10 +area: length * width +area: <=100 // Must be less than or equal to 100 +-- out -- +area: invalid value 202.0 (out of bound <=100): + ./example.cue:4:9 + ./example.cue:3:9 {{{end}}} -{{< columns-separator >}} -CUE -{{{with code "en" "CUE"}}} --- in.cue -- -largeCapital: { - name: string - pop: >5M - capital: true -} -{{{end}}} -{{< /columns >}} - -In general, in CUE one starts with a broad definition of a type, describing -all possible instances. -One then narrows down these definitions, possibly by combining constraints -from different sources (departments, users), until a concrete data instance -remains. - - -### Push, not pull, constraints - -CUE's constraints act as data validators, but also double as -a mechanism to reduce boilerplate. -This is a powerful approach, but requires some different thinking. -With traditional inheritance approaches one specifies the templates that -are to be inherited from at each point they should be used. -In CUE, instead, one selects a set of nodes in the configuration to which -to apply a template. -This selection can be at a different point in the configuration altogether. - -Another way to view this, a JSON configuration, say, can be -defined as a sequence of path-leaf values. -For instance, -{{{with code "en" "json"}}} --- in.json -- -{ - "a": 3, - "b": { - "c": "foo" - } -} -{{{end}}} - -could be represented as - -{{{with code "en" "cue form of json"}}} --- in.cue -- -"a": 3 -"b": "c": "foo" -{{{end}}} - -All the information of the original JSON file is retained in this -representation. - -CUE generalizes this notion to the following pattern: -{{{with code "en" "nodes"}}} -#nofmt --- nodes.cue -- -: -{{{end}}} - -Each field declaration in CUE defines a set of nodes to which to apply -a specific constraint. -Because order doesn't matter, multiple constraints can be applied to the -same nodes, all of which need to apply simultaneously. -Such constraints may even be in different files. -But they may never contradict each other: -if one declaration says a field is `5`, another may not override it to be `6`. -Declaring a field to be both `>5` and `<10` is valid, though. - -This approach is more restricted than full-blown inheritance; -it may not be possible to reuse existing configurations. -On the other hand, it is also a more powerful boilerplate remover. -For instance, suppose each job in a set needs to use a specific -template. -Instead of having to spell this out at each point, -one can declare this separately in a one blanket statement. - -So instead of - -{{{with code "en" "non-dry"}}} --- in.cue -- -jobs: { - foo: acmeMonitoring & {...} - bar: acmeMonitoring & {...} - baz: acmeMonitoring & {...} -} -{{{end}}} - -one can write - -{{{with code "en" "dry"}}} --- in.cue -- -jobs: [string]: acmeMonitoring - -jobs: { - foo: {...} - bar: {...} - baz: {...} -} -{{{end}}} - -There is no need to repeat the reference to the monitoring template for -each job, as the first already states that all jobs _must_ use `acmeMonitoring`. -Such requirements can be specified across files. - -This approach not only reduces the boilerplate contained in `acmeMonitoring` -but also removes the repetitiveness of having to specify -this template for each job in `jobs`. -At the same time, this statement acts as a type enforcement. -This dual function is a key aspect of CUE and -typed feature structure languages in general. - -This approach breaks down, of course, if the restrictions in -`acmeMonitoring` are too stringent and jobs need to override them. -To this extent, CUE provides mechanisms to allow defaults, opt-out, and -soft constraints. - - -### Separate configuration from computation - -There comes a time that one (seemingly) will need do complex -computations to generate some configuration data. -But simplicity of a configuration language can be paramount when one quickly -needs to make changes. -These are obviously conflicting interests. - -CUE takes the stance that computation and configuration should -be separated. -And CUE actually makes this easy. -The data that needs to be computed can be generated outside of CUE -and put in a file that is to be mixed in. -The data can even be generated in CUE's scripting layer and automatically -injected in a configuration pipeline. -Both approaches rely on CUE's property that the order in which this data gets -added is irrelevant. - - - -### Be useful at all scales - -The usefulness of a language may depend on the scale of the project. -Having too many different languages can put a cognitive strain on -developers, though, and migrating from one language to another as -scaling requirements change can be very costly. -CUE aims to minimize these costs -by covering a myriad of data- and configuration-related tasks at all scales. - -**Small scale** -At small scales, reducing boilerplate in configurations is not necessarily -the best thing to do. -Even at a small scale, however, repetition can be error prone. -For such cases, CUE can define schema to validate otherwise -typeless data files. - -**Medium scale** -As soon the desire arises to reduce boilerplate, the `cue` tool can -help to automatically rewrite configurations. -See the Quick and Dirty section of the -[Kubernetes tutorial](https://github.com/cue-labs/cue-by-example/blob/main/003_kubernetes_tutorial/README.md) -for an example using the `import` and `trim` tool. -Thousands of lines can be obliterated automatically using this approach. - -**Large scale** -CUE's underlying formalism was developed for large-scale configuration. -Its import model incorporates best practices for large-scale engineering -and it is optimized for automation. -A key to this is advanced tooling. -The mathematical model underlying CUE's operations allows for -automation that is intractable for most other approaches. -CUE's `trim` command is an example of this. - - -### Tooling - -Automation is key. -Nowadays, a good chunk of code gets generated, analyzed, reformatted, -and so on by machines. -The CUE language, APIs, and tooling have been designed to allow for -machine manipulation. -Aspects of this are: - -- make the language easy to scan and parse, -- restrictions on imports, -- allow any piece of data to be split across files and generated - from different sources, -- define packages at the directory level, -- and of course its value and type model. - -The order independence also plays a key role in this. -It allows combining constraints from various sources without having -to define any order in which they are to be applied to get -predictable results. - - +In its mission to support people using the language and to promote its +adoption, the CUE project develops and publishes a variety of documentation and +tools, including: + +{{< table >}} +| Resource | Description +| --- | --- +| [The `cue` command]({{< relref "installation" >}}) | A command line tool that evaluates CUE, optionally combining it with structured data and other schema formats to validate, transform, and output data and constraints. +| [`cuelang.org/go` APIs](https://pkg.go.dev/cuelang.org/go/cue#section-documentation) | Go APIs that enable CUE's capabilities to be embedded in Go programs. +| [The CUE Language Specification]({{< ref "docs/reference/spec" >}}) | The formal specification of CUE that defines how implementations of the language should behave. +| [cuelang.org](/) | This website, including a foundational [tour through the language]({{< relref "/docs/tour" >}}), hands-on [tutorials]({{< relref "/docs/tutorial" >}}) and [how-to guides]({{< relref "/docs/howto" >}}), and informative [concept guides]({{< relref "/docs/concept" >}}). +| [The CUE Playground](/play/) | A browser-based tool that lets you try out CUE without installing anything. +{{< /table >}} +
+ +By design, CUE isn't a general-purpose programming language, +but its power and flexibility drive its use across a wide range of +applications. It's often used to define, validate and generate +[configuration]({{< relref "/docs/concept/how-cue-enables-configuration" >}}). +CUE also excels at +[validating data]({{< relref "/docs/concept/how-cue-enables-data-validation" >}}) +(such as JSON and YAML) against CUE schemas and policies, whilst also allowing schemas encoded in a variety of +[other formats]({{< relref "/docs/integration/" >}}) +(such as JSON Schema, Protobuf, and OpenAPI) to be used simultaneously. + +CUE's language features enables you to +[template data]({{< relref "/docs/tour/types/templates" >}}), +reducing boilerplate by specifying fields in bulk and allowing data's +important characteristics to stand out prominently. It's also used for +[code generation]({{< relref "/docs/concept/code-generation-and-extraction-use-case" >}}), +and to leverage existing schemas defined in formats such as +[Protobuf]({{< relref "/docs/concept/how-cue-works-with-protocol-buffers" >}}), +[JSON Schema]({{< relref "/docs/concept/how-cue-works-with-json-schema" >}}), +and +[Go]({{< relref "/docs/concept/how-cue-works-with-go" >}}) types. + +Over the next few pages you'll learn about some unique properties of the CUE +language, including: + +- why the merged concepts of "types" and "values" enable succinct and clear + constraints +- how some of CUE's core design principles combine so that the source of each + specific value is never in doubt -- no more hunting through confusing layers + of "overrides" to figure out which files disagree about a particular value +- the advanced tooling that's made possible by CUE's careful design, including + automated boilerplate removal! + + + +{{< warning >}} +Welcome to +}}" style="font-weight: normal;">the CUE community +-- but be warned ... \ +**Prolonged exposure to CUE can seriously affect how you approach data and configuration - for good!** +{{< /warning >}} + +*Next page:* [A Familiar Look and Feel]({{< relref "cue-is-familiar" >}}) diff --git a/content/docs/introduction/cue-is-different/en.md b/content/docs/introduction/cue-is-different/en.md new file mode 100644 index 000000000..328b35f21 --- /dev/null +++ b/content/docs/introduction/cue-is-different/en.md @@ -0,0 +1,307 @@ +--- +title: Some Unique Differences +weight: 20 +--- + +As you just saw in +[*A Familiar Look and Feel*]({{< relref "cue-is-familiar" >}}), +CUE can be used to encode data using a friendlier, more convenient syntax than +JSON - and some folks do just that. However many teams also rely on the +language's advanced capabilities to validate and secure their data and +configurations, and these features build on some rather unique characteristics. + +Let's take a look at some aspects of CUE that you might not have +experienced in a language before ... + +### Order Doesn't Matter + +Most languages require variables, fields, or data to be declared before they're +used - especially when one data value depends on the value of another. + +**CUE is different:** in CUE, *fields can be defined in any order*. + +This property drives many of CUE's most powerful features, and is referred to +as *order irrelevance*. It applies at all levels of granularity: + +- within the fields of each data *struct* (which is what JSON calls an "object"), +- across the fields and structs defined inside a single `.cue` file, +- when merging multiple `.cue` files that make up a CUE *package*. + +Order irrelevance flows from the rules of CUE's most fundamental operation, +called **unification**. Unification is the process by which CUE determines if +values are compatible with one another. It occurs when you explicitly use +the `&` operator, or implicitly when you mention a field multiple times. + +In formal terms, unification is defined so that the operation is associative, +commutative, idempotent, and recursive. This means that when values are +unified, CUE guarantees that every possible order in which they *might* be +combined produces the same underlying data structure, and therefore it doesn't +matter which *specific* ordering is chosen - even when unifying deeply nested +data. + +In practical terms, unification's rules mean that: + +- Data is immutable: if a field is made concrete by assigning it a specific + value, that value is fixed and cannot be changed. (This might appear + restrictive at first glance, but in reality CUE gives you plenty of options + to cater for the different problematic situations you might be imagining!) +- Data and constraints can be combined from multiple sources predictably and + efficiently, optionally using a convenient shorthand form for specifying + sparsely-populated structs. +- If a field is declared more than once, then all its assigned values must be + compatible with each other. When only specifying concrete data, this + simplifies down: all the assigned values must be *identical*. + +In this example, `A` is specified using implicit unification, and `B` is +specified using explicit unification: + +{{{with code "en" "simple unification"}}} +exec cue export data.cue --out yaml +cmp stdout out +-- data.cue -- +A: 1 +B: 2 & 2 +A: 1 +-- out -- +A: 1 +B: 2 +{{{end}}} + +In this example you might be wondering if these unifications have any point - +surely no-one would *actually* specify\ +`B: 2 & 2`? Wouldn't `B: 2` be sufficient? + +In the case of the value of `B`, you'd be right to wonder - this is simply a +demonstration of explict unification so that you can recognise it, when it's +used later in more interesting situations. +Implicit unification, however (as with field `A`), becomes more interesting when +we learn that, by default, *the `cue` command implicitly unifies the top level +of all the data it's given*. +This means that when we invoke the `cue` command and tell it about different +sources of data, the evaluation result is the unification of all those sources. + +To see what this means in practice, we'll unify some CUE, JSON, and YAML, +simply by mentioning their three data files: + +{{{with code "en" "multi-file unification"}}} +#location top-left top-right bottom-left bottom-right +exec cue export data.cue data.yml data.json --out json +cmp stdout out +-- data.cue -- +CUE: true +a: b: c: 1 +-- data.yml -- +YAML: true +a: + b: + e: 3 +-- data.json -- +{ + "JSON": true, + "a": { + "b": { + "d": 2 + } + } +} +-- out -- +{ + "JSON": true, + "YAML": true, + "CUE": true, + "a": { + "b": { + "d": 2, + "e": 3, + "c": 1 + } + } +} +{{{end}}} + +But watch what happens when we try and specify a field with two incompatible +values - the `cue` command emits an error message pointing out the +incompatibility and the evaluation fails: + +{{{with code "en" "unification failure"}}} +! exec cue export data.cue data.json +cmp stderr out +-- data.cue -- +source: "CUE" +-- data.json -- +{ + "source": "JSON" +} +-- out -- +source: conflicting values "CUE" and "JSON": + ./data.cue:1:9 + ./data.json:2:15 +{{{end}}} + +CUE's design ensures that combining values in any order always gives the same +result. Later, in *Why CUE?*, we'll see some of the situations and use cases +that have elegant solutions enabled by this property. + +### Types Are Values + +In many languages there's a strong distinction between the concrete "values" +that a variable or field can be assigned (e.g. `"foo"`, `4.2`, or `true`) +and the "types" that describe *sets* of permissible values (e.g. strings, +floating-point numbers, or booleans, respectively). Most languages don't +provide a first-class mechanism to constrain values more precisely, leaving any +nuanced value constraints for the user to implement in code with explicit +runtime checks. + +**CUE is different:** *CUE doesn't differentiate between values and types*. + +This is a powerful concept that, as you'll see shortly, allows you to define +detailed constraints - but it also simplifies the process of learning and using +CUE. Because the language has a single, unified syntax for data and for +constraints, there is no separate schema or data definition language to learn. +The syntax even encompasses concepts such as sum types and enums - even null +coalescing is reduced down to this single construct. + +Whilst CUE does provide a type hierarchy that includes `string`, `float`, +and `bool`, along with `bytes`, `int`, `number`, `null`, `[...]` (list), and +`{...}` (struct), these are simply well-known names for constraints that limit +a field's value to well-defined sets or ranges. +The power of CUE comes from constructing precise constraints using these types +as a starting point, progressively layering and unifying additional constraints +built with CUE's rich set of primitives and built-in functions. + +Here's an example that uses +[bounds]({{< relref "docs/tour/types/bounds" >}}) (`>`, `<=`, etc) +to construct constraints that limit a number's value, unified with additional +constraints from a +[disjunction]({{< relref "/docs/tour/types/disjunctions" >}}) +("`|`") that requires a value to be one option from a prescribed set: + +{{{with code "en" "constraints are values too"}}} +#location left right +exec cue eval constraints.cue +cmp stdout out +-- constraints.cue -- +#over10: >10 +#under50: <50 +#from5To40: >=5 & <=40 +#options: 9 | 10 | 11 | 39 | 40 | 41 + +myNumber: int & #over10 & #under50 +myNumber: #from5To40 +myNumber: #options +-- out -- +#over10: >10 +#under50: <50 +#from5To40: >=5 & <=40 +#options: 9 | 10 | 11 | 39 | 40 | 41 +myNumber: 11 | 39 | 40 +{{{end}}} + +In the `cue eval` output, notice how CUE is able to *simplify* the constraints +that apply to the `myNumber` field by ruling out options that definitely aren't +possible, due to the implicit and explicit unification of the constraints that +we defined. + +Later, in *Why CUE?*, we'll see how CUE's merging of types, values, and +constraints into a single concept enables effective and concise schema and +policy validation. + +### Push Constraints, Don't Pull Them + +CUE's constraints act as data validators: + +{{{with code "en" "constraints validate data"}}} +#location top-left top-left top-right bottom +! exec cue vet schema.cue policy.cue data.yml +cmp stderr out +-- schema.cue -- +A: int +B: float +C: string +-- policy.cue -- +A: <=99 +B: >4.2 +C: !="rm -rf /" +-- data.yml -- +A: 100 +B: 1.1 +C: "rm -rf /" +-- out -- +A: invalid value 100 (out of bound <=99): + ./policy.cue:1:4 + ./data.yml:1:4 +B: invalid value 1.1 (out of bound >4.2): + ./policy.cue:2:4 + ./data.yml:2:4 +C: invalid value "rm -rf /" (out of bound !="rm -rf /"): + ./policy.cue:3:4 + ./data.yml:3:4 + ./schema.cue:3:4 +{{{end}}} + +*Pattern constraints* impose constraints on all the fields whose names match +their pattern. They're written as\ +`[pattern]: value`, where `pattern` must be either a value +of type string, or the wildcard value `_` (called "top"). Here's an +example of pattern constraints in action: + +{{{with code "en" "pattern constraints select fields"}}} +#location top-left top-right bottom +! exec cue vet constraints.cue data.yml +cmp stderr out +-- constraints.cue -- +A: { + // Every field must be an int. + [_]: int + // Every field whose name starts with b must + // be 10 or greater. + [=~"^b"]: >=10 +} +-- data.yml -- +A: + foo: 4.2 + bar: 100 + baz: 5 +-- out -- +A.foo: conflicting values 4.2 and int (mismatched types float and int): + ./constraints.cue:3:7 + ./data.yml:2:8 +A.baz: invalid value 5 (out of bound >=10): + ./constraints.cue:6:12 + ./data.yml:4:8 +{{{end}}} + +Pattern constraints *don't* instantiate every field that their pattern might +match. If they *did*, then the example pattern constraint of `[string]: >10` +would bring every field with a name matching `string` into existence - i.e. +*every possible field*, which would take an unacceptable amount of time to +compute ... no matter how powerful your hardware! +So: a pattern constraint merely acts on fields by unifying its constraints with +*existing* fields that match its pattern. + +Many languages have a form of import, or perhaps inheritance, that allows +lower-level (more deeply nested) components to pull restrictions in from some +other entity by specifying them at the point of use. + +**CUE is different:** *unification and pattern constraints offer a simple way +to impose constraints from above*. + +CUE has a robust +[package]({{< relref "/docs/concept/modules-packages-instances" >}}) system +and a carefully designed +[module]({{< relref "/docs/reference/modules" >}}) system +which support imports and cross-entity data and constraint sharing. +Before reaching for those tools, however, it's worth discovering just how +flexible and powerful pattern constraints and unification can be! + +In *Why CUE?* we'll see how pattern constraints are the backbone of a highly +productive mechanism for reducing boilerplate data and configuration, whilst +also enforcing more centralised policy constraints on nested data. + +{{< warning >}} +**On the next page**: discover how CUE's differences have a real impact on data +handling and validation, and why teams trust it with their configurations at +all scales. +{{< /warning >}} + +*Next page:* [Why CUE?]({{< relref "why-cue" >}}) diff --git a/content/docs/introduction/cue-is-different/gen_cache.cue b/content/docs/introduction/cue-is-different/gen_cache.cue new file mode 100644 index 000000000..6e2df5aa8 --- /dev/null +++ b/content/docs/introduction/cue-is-different/gen_cache.cue @@ -0,0 +1,23 @@ +package site +{ + content: { + docs: { + introduction: { + "cue-is-different": { + page: { + cache: { + code: { + "simple unification": "EjhfOapC4hhWLY7VLKEAepjU5Csbvnk0DNPHltmZfyU=" + "multi-file unification": "f6c1AjdIbUZBZr/HolB940K9JagBYKxwBI9fD02C1mA=" + "unification failure": "HB+Ff08hUEAPT4fKGaxvJG2JlqyikDOmmIrJvEWo6Zk=" + "constraints are values too": "k5XInymlroMgKSFhrEMoGme/D/2Di+5dVm6jKLBGkEs=" + "constraints validate data": "pzYcgXWe2mTiJcCuCbc55ciQTHKbnHxA8/BjQqD50xg=" + "pattern constraints select fields": "k73fNVTL5Ptc9F90EW+GFBsRGmCNejROGTKm1N5hqjA=" + } + } + } + } + } + } + } +} diff --git a/content/docs/introduction/cue-is-different/page.cue b/content/docs/introduction/cue-is-different/page.cue new file mode 100644 index 000000000..b0deeb202 --- /dev/null +++ b/content/docs/introduction/cue-is-different/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "cue-is-different": page: _ diff --git a/content/docs/introduction/cue-is-familiar/en.md b/content/docs/introduction/cue-is-familiar/en.md new file mode 100644 index 000000000..24e590b76 --- /dev/null +++ b/content/docs/introduction/cue-is-familiar/en.md @@ -0,0 +1,109 @@ +--- +title: A Familiar Look and Feel +weight: 10 +--- + +CUE will probably feel rather familiar if you've spent any time working with +data. CUE shares some syntax with JSON, but *significantly* improves the +experience of managing JSON by hand. + +In its simplest form, CUE looks a lot like JSON. +This is because CUE is defined to be a *superset* of JSON - which means that +all valid JSON is CUE, but not vice versa. +Editing JSON manually can be somewhat awkward, so CUE introduces several +conveniences to make writing and reading data easier: + +- comments are allowed, starting with `//` and extending to the end of the line +- field names without special characters don’t need to be quoted +- the outermost curly braces in a CUE file are optional +- commas after a field are optional (and are usually omitted) +- commas after the final element of a list are allowed +- literal multiline strings are allowed, and don't require newlines to be encoded +- nested structs containing one (or a few) fields have a convenient shorthand + +Here's some data encoded as commented CUE, alongside the equivalent JSON +document. Notice how the CUE lacks curly braces at the top and bottom, and +doesn't have commas after each field's value: + +{{{with code "en" "CUE improves on JSON"}}} +#location left right +exec cue export example.cue --out json +cmp stdout out +-- example.cue -- +strings: { + singleLine: "Double quotes == string literal" + multiLine: """ + Multiline strings start and end with triple + double-quotes - no escaping of newlines! + """ +} + +// Many field names don't need to be quoted +// (but can be, if you want). +foo_Bar: 1 +baz2: 2.2 + +// Some field names do need quotes, such as those +// that start with numbers, or contain spaces, +// hyphens, or other special characters. +"qu ux": "3.33" +"4": "four" + +a: deeply: nested: field: "value" + +a: deeply: nested: struct: { + containing: "multiple" + fields: true +} + +// A list's final element can be followed by +// an optional comma. +aList: [ + "a", + "b", + "c", +] +anotherList: [1, 2, 3, 4, 5] +-- out -- +{ + "strings": { + "singleLine": "Double quotes == string literal", + "multiLine": "Multiline strings start and end with triple\ndouble-quotes - no escaping of newlines!" + }, + "foo_Bar": 1, + "baz2": 2.2, + "qu ux": "3.33", + "4": "four", + "a": { + "deeply": { + "nested": { + "field": "value", + "struct": { + "containing": "multiple", + "fields": true + } + } + } + }, + "aList": [ + "a", + "b", + "c" + ], + "anotherList": [ + 1, + 2, + 3, + 4, + 5 + ] +} +{{{end}}} + +However, most CUE users don't choose CUE simply because it's nicer to handle +than JSON - that's just a pragmatic consequence of the language's design. +Folks adopt CUE because of its revolutionary features, and these stem from its +concepts and syntaxes that *won't* seem quite as familiar. + +We'll take a look at them next, in +[*Some Unique Differences*]({{< relref "cue-is-different" >}}) ... diff --git a/content/docs/introduction/cue-is-familiar/gen_cache.cue b/content/docs/introduction/cue-is-familiar/gen_cache.cue new file mode 100644 index 000000000..fb41abf8d --- /dev/null +++ b/content/docs/introduction/cue-is-familiar/gen_cache.cue @@ -0,0 +1,18 @@ +package site +{ + content: { + docs: { + introduction: { + "cue-is-familiar": { + page: { + cache: { + code: { + "CUE improves on JSON": "xrgZxX4rTBfvz8EMBrVilzA+Lht1lai9KoTwLjHdjps=" + } + } + } + } + } + } + } +} diff --git a/content/docs/introduction/cue-is-familiar/page.cue b/content/docs/introduction/cue-is-familiar/page.cue new file mode 100644 index 000000000..2f903d174 --- /dev/null +++ b/content/docs/introduction/cue-is-familiar/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "cue-is-familiar": page: _ diff --git a/content/docs/introduction/effective-cue/en.md b/content/docs/introduction/effective-cue/en.md new file mode 100644 index 000000000..28e186cca --- /dev/null +++ b/content/docs/introduction/effective-cue/en.md @@ -0,0 +1,130 @@ +--- +title: Effective CUE +weight: 40 +--- + +CUE is designed to help users and teams manage data and configuration across +all scales. Here are some strategies that help make your use of CUE more +effective. + +### Start with broad definitions and get progressively more specific + +In general, using CUE, we start with a broad definition of a type that +describes all possible instances of the type. We then narrow down these +definitions, often combining constraints from multiple sources (e.g. different +departments, teams, and users), until a concrete data instance remains. + +Here's a demonstration of this at a small scale. Keep in mind that CUE permits +this approach to be used effectively at far larger scales: + +{{< columns >}} +{{{with upload "en" "schema"}}} +-- schema.cue -- +package geo + +#Municipality: { + name: string + pop: int + capital: bool +} +{{{end}}} +{{< columns-separator >}} +{{{with upload "en" "CUE"}}} +-- native.cue -- +package geo + +#LargeCapital: #Municipality & { + name: string + pop: >5M + capital: true +} +{{{end}}} +{{< columns-separator >}} +{{{with upload "en" "data"}}} +-- data.cue -- +package geo + +kinshasa: #LargeCapital & { + name: "Kinshasa" + pop: 16.315M + capital: true +} +{{{end}}} +{{< /columns >}} +{{{with _script_ "en" "HIDDEN: vet"}}} +cue vet .:geo +{{{end}}} + +### Separate configuration from computation + +Situations often arise where it looks like we'll need to do complex +computations to generate some configuration data. This approach is generally in +conflict with the goal of using a *simple* configuration language that permits +and encourages changes that can be understood in isolation, at speed, possibly +by non-domain experts. + +CUE has a stance that computation and configuration should be separated - and +*unification* makes this easy, because of *order irrelevance*. +Data that needs to be be computed can be generated outside CUE, and mixed into +a configuration using unification, CUE packages, and the `cue` command's +support for a variety of +[different data encodings]({{< relref "/docs/integration" >}}). + +The processes that compute this data can even be orchestrated by a CUE +[workflow command]({{< relref "docs/concept/how-cue-enables-configuration" >}}#tooling-and-automation), +enhancing developer workflows and performing inline computation input/output +validation. + +### Make appropriate choices for your level of scale + +The usefulness of a specific language can depend on the scale of the project +being developed, but adopting the "perfect" language for every new scale +milestone can put a cognitive strain on developers. CUE aims to minimize the +high cost of migrating from one language to another, as scaling requirements +change, by delivering solutions to data- and configuration-related tasks at all +scales: + +- **Small scale**: reducing boilerplate in configurations is not necessarily + the most effective thing to do. However, even at a small scale, repetition + can be tedious and lead to errors. For these situations, where it might be + best to keep small configurations specified and updated in isolation, CUE can + be used to define schemas that validate type-free + [JSON]({{< relref "/docs/howto/validate-json-using-cue" >}}) and + [YAML]({{< relref "/docs/howto/validate-yaml-using-cue" >}}) files. +- **Medium scale**: when the size or complexity of configurations reaches a + certain point, reducing boilerplate may start to become a more attractive + proposition. As you saw, earlier in this introduction, CUE can be used to + isolate such boilerplate, with the `cue trim` command removing it + automatically. It's not uncommon for thousands of lines to be obliterated + using this approach. +- **Large scale**: The + [formalism that underlies CUE](https://github.com/cue-lang/cue/blob/master/doc/ref/impl.md) + was specifically developed for large-scale configuration. CUE's import model + incorporates many best practices for large-scale engineering, and the + language is optimized for automation through advanced tooling. The + mathematical model and + [the logic of CUE]({{< relref "/docs/concept/the-logic-of-cue" >}}) + permits automation that's inaccessible for most other approaches - such as + `cue trim`'s capabilities. Read the next section, *Tooling*, for more on + this. + +### Know when to invest in building specialized tooling + +Automation is key. In contemporary operations, a large amount of code and +configuration gets generated, analyzed, reformatted, and mananged by machines. +The CUE language, its APIs, and its tooling have been designed to allow for +this mechanical manipulation, and you should anticipate taking advantage of +these capabilities as the scale and complexity of your mission grows. + +CUE's automation-friendly design is exemplified by these characteristics: + +- the language is easy to scan and parse +- there are specific restrictions on imports that prevent configuration graphs + of unbounded complexity +- data can be split across files and generated from different sources +- order irrelevance permits values and constraints to be mixed together easily, + without having to specify (or care about) the order in which they're combined +- packages are defined at the directory level +- its first-class value and type model is unparalleled. + +*Next page:* [Installing CUE]({{< relref "installation" >}}) diff --git a/content/docs/introduction/effective-cue/gen_cache.cue b/content/docs/introduction/effective-cue/gen_cache.cue new file mode 100644 index 000000000..3066d94ee --- /dev/null +++ b/content/docs/introduction/effective-cue/gen_cache.cue @@ -0,0 +1,30 @@ +package site +{ + content: { + docs: { + introduction: { + "effective-cue": { + page: { + cache: { + upload: { + schema: "KFWxfEfJfz3Aedft3BUioIJi24SCnJXos4HkkYxsHCo=" + CUE: "70oRF59Lr0I9suxpu8l8vVG++XSPMTa26PJxt+tYyns=" + data: "/rPsPNBccQcT4t0Vb5qUU1QW/X2ykO7EgtY4qoHuEKI=" + } + multi_step: { + hash: "EMB7JQJ9JN7656VQHS9SP3TR0FSNTSNSE0MCJH5U1RUS24RA9VK0====" + scriptHash: "6RGDEU0BIFNVMT9ULVRBMEHNALS8VTURLA4K2V446BOEH2OAGOJ0====" + steps: [{ + doc: "" + cmd: "cue vet .:geo" + exitCode: 0 + output: "" + }] + } + } + } + } + } + } + } +} diff --git a/content/docs/introduction/effective-cue/page.cue b/content/docs/introduction/effective-cue/page.cue new file mode 100644 index 000000000..93f192953 --- /dev/null +++ b/content/docs/introduction/effective-cue/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "effective-cue": page: _ diff --git a/content/docs/introduction/gen_cache.cue b/content/docs/introduction/gen_cache.cue index 1c1a0b733..4a2e161eb 100644 --- a/content/docs/introduction/gen_cache.cue +++ b/content/docs/introduction/gen_cache.cue @@ -6,14 +6,7 @@ package site page: { cache: { code: { - data: "IOdn7JBp4e3oj8s8e7i+gGv5uVPKozMAsDfHgmbqpj0=" - schema: "O/djTELJiIW12SUVNCZEm4c9g2HqOie3nqZ+k8CmHt0=" - CUE: "CBaZo19SuDfzKEzls7bNpDWtvWMO/+jf/7dvlxu2okY=" - json: "DRfUGLPN4bGRVs+BXizP+e9hKkhVqfmmq3FMOD5vRdE=" - "cue form of json": "tmJ2tfGh0yFaZKOmLY86Arhwse0uaCMlGQUIspI9TyQ=" - nodes: "TFaZW7DD0vgAz66ncu5/oEH7v7DNY4AzYHWW3lr1tfI=" - "non-dry": "pB3o00GPiS17XJnFb2pl0ViGTAZR1DQXgANIe3+Zkcc=" - dry: "XJSOpgLsf00vL3+b+OQ9hxNwA1S/Ngh5AKLn0e5zwnk=" + example: "e619dws7hMTkPaLBTs3mPF/FQff4xgdLKPZCeNbtask=" } } } diff --git a/content/docs/introduction/history/en.md b/content/docs/introduction/history/en.md index a416962fb..4bbff195b 100644 --- a/content/docs/introduction/history/en.md +++ b/content/docs/introduction/history/en.md @@ -1,6 +1,6 @@ --- -title: History -weight: 40 +title: The History of CUE +weight: 60 --- Although it is a very different language, the roots of CUE lie in GCL, diff --git a/content/docs/introduction/installation/en.md b/content/docs/introduction/installation/en.md index 5e3c56c06..56eaa722a 100644 --- a/content/docs/introduction/installation/en.md +++ b/content/docs/introduction/installation/en.md @@ -1,6 +1,6 @@ --- -title: Installation -weight: 20 +title: Installing CUE +weight: 50 aliases: - /docs/install - /download diff --git a/content/docs/introduction/next-steps/en.md b/content/docs/introduction/next-steps/en.md new file mode 100644 index 000000000..3dd4907aa --- /dev/null +++ b/content/docs/introduction/next-steps/en.md @@ -0,0 +1,6 @@ +--- +title: Next Steps +weight: 70 +--- + +FIXME: words diff --git a/content/docs/introduction/next-steps/page.cue b/content/docs/introduction/next-steps/page.cue new file mode 100644 index 000000000..ceb2c3c56 --- /dev/null +++ b/content/docs/introduction/next-steps/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "next-steps": page: _ diff --git a/content/docs/introduction/why-cue/en.md b/content/docs/introduction/why-cue/en.md new file mode 100644 index 000000000..08c001bbd --- /dev/null +++ b/content/docs/introduction/why-cue/en.md @@ -0,0 +1,283 @@ +--- +title: Why CUE? +weight: 30 +--- + +The features that make CUE so effective are the result of its careful design, +which is the product of decades of experience in the data and cofiguration +space. The language's behaviours are formally defined in +[The CUE Language Specification]({{< relref "/docs/reference/spec" >}}), +but you don't need to study the spec in order to understand how CUE can make +life easier. + +If you work frequently with data or configuration, the CUE examples in this +introduction might have already given you ideas about how you could use CUE to +simplify, fortify, or otherwise improve your existing setup. +But before jumping in and getting started, it's worth reading through this +site's concept guides on how CUE enables various practices, such as +[configuration]({{< relref "/docs/concept/how-cue-enables-configuration" >}}) +and +[data validation]({{< relref "/docs/concept/how-cue-enables-data-validation" >}}), +and +[how CUE works with different technologies]({{< relref "/docs/integration" >}}). + +As further inspiration, here a more fully-formed example that demonstrates the +concepts and features that users of CUE often find compelling and powerful ... + +### Combining schema, policy, and data, and reducing boilerplate + +You've already seen how *unification* combines data with multiple, layered +constraints that limit the data's acceptable values. You saw how *schema* ("X +is an integer") can live seamlessly alongside *policy constraints* ("X must be +greater than 10") because of CUE's succinct and clear syntax. The *pattern +constraint* feature gives you a way to push down constraints from above, +instead of relying on the authors of nested data to reference your requirements. + +**Here's a demonstration of these concepts in action, working together to drive down complexity**. +Let's start with a hypothetical schema describing the contents of a +website -- its pages and structure -- along with some separate policy +constraints in `policy.cue`: + +{{{with upload "en" "schema"}}} +-- schema.cue -- +package website + +#Page: { + title!: string // title is a required field, as every page must have a title. + urlPath!: string // urlPath is also required. + file!: string // The page's file contains its content. + date!: string // The page's date of publishing. + isDraft!: bool // Is the page part of the published site? + summary?: string // summary is an optional field, as some pages don't need a short summary. + author?: string | null // An optional author name. +} +-- policy.cue -- +package website + +import ( + "strings" + "time" +) + +#Page: { + title?: strings.MinRunes(1) // title cannot be empty (could also be specified as !=""). + urlPath?: strings.MinRunes(1) // urlPath cannot be empty (equivalent to !=""). + file?: =~".html$" | =~".md$" // Content files can be HTML or Markdown. + date?: time.Time // time.Time validates a RFC3339 date-time. + summary?: strings.MaxRunes(150) // Our site layout requires page summaries to be limited in length. + author?: _ // Policy imposes no additional constraints on the author. + isDraft?: _ +} +{{{end}}} + +Here we've used several new language elements, particularly in `policy.cue`: + +- required (`!:`) and optional (`?:`) fields +- regular expression constraints (`=~`) +- importing built-in packages containing the CUE standard library (`import ( ... )`) +- using the standard library (`strings.MinRunes()`) + + +These features are explored and explained in the +[CUE language tour]({{< relref "/docs/tour" >}}) +but -- just for now! -- continue by reading on with the assumption that they +all function as you might expect from their names. + +Now we've set up some schema and policy constraints, we can start to populate +information about our small website's existing pages, marking them with a +publishing date that's in the past to reflect their "already published" state: + +{{{with upload "en" "data"}}} +#codetab(site.cue) linenos="table" +-- site.cue -- +package website + +// pages is a struct whose fields each adhere to the +// #Page constraint imposed through a pattern constraint. +pages: [_]: #Page +pages: { + home: { + title: "Welcome to Widgets'R'We!" + urlPath: "/" + file: "pages/home.html" + summary: "The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: null + } + about: { + title: "About Widgets'R'We" + urlPath: "/about/" + file: "pages/about.html" + summary: "Information about the Widgets'R'We company." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: null + } + contact: { + title: "Contact Widgets'R'We" + urlPath: "/contact/" + file: "pages/contact.html" + summary: "Contact information for the Widgets'R'We company." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: "Markus Marketing" + } +} +{{{end}}} + +We can run `cue vet` to check that each of our pages' data meets the +requirements of the `#Page` constraint. +Like many command line tools, it stays silent to indicate success: + +{{{with script "en" "vet 1"}}} +cue vet +{{{end}}} + +Whilst our site metadata is correctly specified, it's hard not to feel that it +seems a little *unwieldy*. There's quite a lot of duplicated information that +distracts from the important detail of each page - often called "boilerplate". +Let's use CUE to reduce it ... + +We'll use various CUE features to make our configuration shorter, which will +allow the most important information to stand out. Here's the CUE we'll use: + +{{{with upload "en" "defaults"}}} +-- pageDefaults.cue -- +package website + +pages?: [pageId=_]: { + isDraft: _ | *false + author: _ | *null + date: _ | *"1999-01-01T00:00:00Z" + urlPath: _ | *"/\(pageId)/" + file: _ | *"pages/\(pageId).html" +} +{{{end}}} + +In `pageDefaults.cue`, we: + +- use a [pattern constraint]({{< relref "/docs/tour/basics/folding-structs" >}}) + to unify fields inside each member of `pages`, creating what CUE calls a + [template]({{< relref "/docs/tour/types/templates" >}}), +- use the template to assign five fields a + [default value]({{< relref "/docs/tour/types/defaults" >}}) - a value that + only takes effect if a configuration doesn't specify a value elsewhere, +- bring an [alias]({{< relref "/docs/tour/references/aliases" >}}) into + existence (`pageId`) that contains the value of the identifier of each member + of the `pages` struct, and allows us to refer to each page's identifier + *inside* the template. This lets us derive two string fields' default values + dynamically. + +We again use `cue vet`, confirming that *adding* this CUE file to our `website` +package hasn't caused a problem: + +{{{with script "en" "vet 2"}}} +cue vet +{{{end}}} + +Next we need to *remove* the duplicate data from our `site.cue` metadata file - +a tedious and error-prone task, especially if our set of pages were larger, or +were split across more files. Or it *would* be tedious, if it weren't for CUE! + +The `cue trim` command performs **automatic removal of boilerplate** from +configurations where the data can be inferred from other constraints. +Let's try it out here: + +{{{with script "en" "trim"}}} +cue trim +{{{end}}} + +`cue trim`, just as with `cue vet`, stays silent if it was successful. Let's +take a look at the updated `site.cue` metadata file: + +{{{with _script_ "en" "HIDDEN: move before diff"}}} +mv site.cue .site.cue +{{{end}}} + +{{{with upload "en" "site.cue 2"}}} +#force +#codetab(site.cue) linenos="table" +-- site.cue -- +package website + +// pages is a struct whose fields each adhere to the +// #Page constraint imposed through a pattern constraint. +pages: [_]: #Page +pages: { + home: { + title: "Welcome to Widgets'R'We!" + urlPath: "/" + summary: "The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East." + } + about: { + title: "About Widgets'R'We" + summary: "Information about the Widgets'R'We company." + } + contact: { + title: "Contact Widgets'R'We" + summary: "Contact information for the Widgets'R'We company." + author: "Markus Marketing" + } +} +{{{end}}} + +{{{with _script_ "en" "HIDDEN: diff"}}} +diff site.cue .site.cue +{{{end}}} + +Only the notable, *important* information about each page is left behind, with +the boring-but-necessary boilerplate configuration safely isolated in the +`pageDefaults.cue` file. + +Let's use `cue vet` again to confirm that the `#Page` schema still validates +our site metadata after the boilerplate was removed, and then run `cue export` +to show us the concrete data result of our changes: + +{{{with script "en" "check & export"}}} +cue vet +cue export --out yaml +{{{end}}} + +We can see that we exported the same data that we started with ... but with a +significantly clearer and more succinct representation behind the scenes! +Lastly, we'll introduce a new CUE file to the `website` package that *attempts* +to modify an existing page's title: + +{{{with upload "en" "broken update"}}} +#codetab(pages.cue) linenos="table" +-- updates.cue -- +package website + +pages: { + about: title: "Cyber Widgets 2000" +} +{{{end}}} + +The `cue vet` command complains, because the title already has a concrete value +specified elsewhere and data is immutable in CUE. +Notice that it tells us the locations of the conflicting values (encoded as +`filename:line-number:character-number`) so that we can find and resolve this +kind of mistake faster and easier: + +{{{with script "en" "vet 3"}}} +! cue vet +{{{end}}} + +Here's a reminder of the concepts and features that we just used: + +- *Constraints* were *unified* across the multiple files in our CUE package. +- *Pattern constraints* allowed us to push constraints down from above. +- *Templates* and *default values* gave us a way to centralise repetitive + information without repeating it. +- *Aliases* gave us way to construct a template's field values dynamically. +- `cue vet` checked that data adhered to its constraints. +- `cue export` produced concrete data. +- `cue trim` automatically removed duplicate information from our data source. + +{{< warning >}} +**On the next page**: learn some strategies to make effective use of CUE. +{{< /warning >}} + +*Next page:* [Effective CUE]({{< relref "effective-cue" >}}) diff --git a/content/docs/introduction/why-cue/gen_cache.cue b/content/docs/introduction/why-cue/gen_cache.cue new file mode 100644 index 000000000..c38265ffd --- /dev/null +++ b/content/docs/introduction/why-cue/gen_cache.cue @@ -0,0 +1,99 @@ +package site +{ + content: { + docs: { + introduction: { + "why-cue": { + page: { + cache: { + upload: { + schema: "qB76EEPGu+Qc09Z77taBEbviyzMjIWh4ZTZRn5QETw8=" + data: "2BjKd0x4U6HLLNEJ4PL6AtQNJ/rgBi+MvP78hRvg8/A=" + defaults: "hlVocRQHABOue1CV7hv66A/U+xxFnkveP9Svl9AbHFA=" + "site.cue 2": "WVgPBTsE8zQoSjkmzR5KBPTRKyjSCWPfsbIbaGTQufc=" + "broken update": "iMVnW9ehyM7Yax1bJK9Aloceavppq60w2rbLMPmhxfE=" + } + multi_step: { + hash: "3BJAHRQNIT2VUK6G14QT7I47B4T7UJLIBJLGD38160EE8RCNSDT0====" + scriptHash: "CTCHP9D7H7K2OMJFPBHMK9OP8J609364U36R5VJBDHGVBQBC39P0====" + steps: [{ + doc: "" + cmd: "cue vet" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "cue vet" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "cue trim" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "mv site.cue .site.cue" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "diff site.cue .site.cue" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "cue vet" + exitCode: 0 + output: "" + }, { + doc: "" + cmd: "cue export --out yaml" + exitCode: 0 + output: """ + pages: + home: + title: Welcome to Widgets'R'We! + urlPath: / + summary: The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East. + isDraft: false + author: null + date: "1999-01-01T00:00:00Z" + file: pages/home.html + about: + title: About Widgets'R'We + summary: Information about the Widgets'R'We company. + isDraft: false + author: null + date: "1999-01-01T00:00:00Z" + urlPath: /about/ + file: pages/about.html + contact: + title: Contact Widgets'R'We + summary: Contact information for the Widgets'R'We company. + author: Markus Marketing + isDraft: false + date: "1999-01-01T00:00:00Z" + urlPath: /contact/ + file: pages/contact.html + + """ + }, { + doc: "" + cmd: "cue vet" + exitCode: 1 + output: """ + pages.about.title: conflicting values "Cyber Widgets 2000" and "About Widgets'R'We": + ./site.cue:13:12 + ./updates.cue:4:16 + + """ + }] + } + } + } + } + } + } + } +} diff --git a/content/docs/introduction/why-cue/page.cue b/content/docs/introduction/why-cue/page.cue new file mode 100644 index 000000000..98d68365c --- /dev/null +++ b/content/docs/introduction/why-cue/page.cue @@ -0,0 +1,3 @@ +package site + +content: docs: introduction: "why-cue": page: _ diff --git a/hugo/content/en/docs/introduction/_index.md b/hugo/content/en/docs/introduction/_index.md index 3c26f358d..5914fd0b1 100644 --- a/hugo/content/en/docs/introduction/_index.md +++ b/hugo/content/en/docs/introduction/_index.md @@ -8,280 +8,84 @@ aliases: - /docs/about --- -## Welcome! - -CUE is an open-source data validation language and inference engine -with its roots in logic programming. -Although the language is not a general-purpose programming language, -it has many applications, such as -data validation, data templating, configuration, querying, -code generation and even scripting. -The inference engine can be used to validate -data in code or to include it as part of a code generation pipeline. - -A key thing that sets CUE apart from its peer languages -is that it merges types and values into a single concept. -Whereas in most languages types and values are strictly distinct, -CUE orders them in a single hierarchy (a lattice, to be precise). -This is a very powerful concept that allows CUE to do -many fancy things. -It also simplifies matters. -For instance, there is no need for generics, and enums, sum types -and null coalescing are all the same thing. - - -## Applications - -CUE's design ensures that combining CUE values in any -order always gives the same result -(it is associative, commutative and idempotent). -This makes CUE particularly well-suited for cases where CUE -constraints are combined from different sources: - -- Data validation: different departments or groups can each -define their own constraints to apply to the same set of data. - -- Code extraction and generation: extract CUE definitions from -multiple sources (Go code, Protobuf), combine them into a single -definition, and use that to generate definitions in another -format (e.g. OpenAPI). - -- Configuration: values can be combined from different sources - without one having to import the other. - -The ordering of values also allows set containment analysis of entire -configurations. -Where most validation systems are limited to checking whether a concrete -value matches a schema, CUE can validate whether any instance of -one schema is also an instance of another (is it backwards compatible?), -or compute a new schema that represents all instances that match -two other schema. - -## Philosophy and principles - -### Types are Values - -CUE does not distinguish between values and types. -This is a powerful notion that allows CUE to define ultra-detailed -constraints, but it also simplifies things considerably: -there is no separate schema or data definition language to learn -and related language constructs such as sum types, enums, -and even null coalescing collapse onto a single construct. - -Below is a demonstration of this concept. -On the left one can see a JSON object (in CUE syntax) with some properties -about the city of Moscow. -The middle column shows a possible schema for any municipality. -On the right one sees a mix between data and schema as is exemplary of CUE. - -{{< columns >}} -Data -```cue -moscow: { - name: "Moscow" - pop: 11.92M - capital: true -} -``` -{{< columns-separator >}} -Schema -```cue -municipality: { - name: string - pop: int - capital: bool -} -``` -{{< columns-separator >}} -CUE -```cue -largeCapital: { - name: string - pop: >5M - capital: true -} -``` -{{< /columns >}} - -In general, in CUE one starts with a broad definition of a type, describing -all possible instances. -One then narrows down these definitions, possibly by combining constraints -from different sources (departments, users), until a concrete data instance -remains. - - -### Push, not pull, constraints - -CUE's constraints act as data validators, but also double as -a mechanism to reduce boilerplate. -This is a powerful approach, but requires some different thinking. -With traditional inheritance approaches one specifies the templates that -are to be inherited from at each point they should be used. -In CUE, instead, one selects a set of nodes in the configuration to which -to apply a template. -This selection can be at a different point in the configuration altogether. - -Another way to view this, a JSON configuration, say, can be -defined as a sequence of path-leaf values. -For instance, -```json -{ - "a": 3, - "b": { - "c": "foo" - } -} -``` - -could be represented as - -```cue -"a": 3 -"b": "c": "foo" -``` - -All the information of the original JSON file is retained in this -representation. - -CUE generalizes this notion to the following pattern: -```cue -: -``` - -Each field declaration in CUE defines a set of nodes to which to apply -a specific constraint. -Because order doesn't matter, multiple constraints can be applied to the -same nodes, all of which need to apply simultaneously. -Such constraints may even be in different files. -But they may never contradict each other: -if one declaration says a field is `5`, another may not override it to be `6`. -Declaring a field to be both `>5` and `<10` is valid, though. - -This approach is more restricted than full-blown inheritance; -it may not be possible to reuse existing configurations. -On the other hand, it is also a more powerful boilerplate remover. -For instance, suppose each job in a set needs to use a specific -template. -Instead of having to spell this out at each point, -one can declare this separately in a one blanket statement. - -So instead of - -```cue -jobs: { - foo: acmeMonitoring & {...} - bar: acmeMonitoring & {...} - baz: acmeMonitoring & {...} -} -``` - -one can write - -```cue -jobs: [string]: acmeMonitoring - -jobs: { - foo: {...} - bar: {...} - baz: {...} -} -``` - -There is no need to repeat the reference to the monitoring template for -each job, as the first already states that all jobs _must_ use `acmeMonitoring`. -Such requirements can be specified across files. - -This approach not only reduces the boilerplate contained in `acmeMonitoring` -but also removes the repetitiveness of having to specify -this template for each job in `jobs`. -At the same time, this statement acts as a type enforcement. -This dual function is a key aspect of CUE and -typed feature structure languages in general. - -This approach breaks down, of course, if the restrictions in -`acmeMonitoring` are too stringent and jobs need to override them. -To this extent, CUE provides mechanisms to allow defaults, opt-out, and -soft constraints. - - -### Separate configuration from computation - -There comes a time that one (seemingly) will need do complex -computations to generate some configuration data. -But simplicity of a configuration language can be paramount when one quickly -needs to make changes. -These are obviously conflicting interests. - -CUE takes the stance that computation and configuration should -be separated. -And CUE actually makes this easy. -The data that needs to be computed can be generated outside of CUE -and put in a file that is to be mixed in. -The data can even be generated in CUE's scripting layer and automatically -injected in a configuration pipeline. -Both approaches rely on CUE's property that the order in which this data gets -added is irrelevant. - - - -### Be useful at all scales - -The usefulness of a language may depend on the scale of the project. -Having too many different languages can put a cognitive strain on -developers, though, and migrating from one language to another as -scaling requirements change can be very costly. -CUE aims to minimize these costs -by covering a myriad of data- and configuration-related tasks at all scales. - -**Small scale** -At small scales, reducing boilerplate in configurations is not necessarily -the best thing to do. -Even at a small scale, however, repetition can be error prone. -For such cases, CUE can define schema to validate otherwise -typeless data files. - -**Medium scale** -As soon the desire arises to reduce boilerplate, the `cue` tool can -help to automatically rewrite configurations. -See the Quick and Dirty section of the -[Kubernetes tutorial](https://github.com/cue-labs/cue-by-example/blob/main/003_kubernetes_tutorial/README.md) -for an example using the `import` and `trim` tool. -Thousands of lines can be obliterated automatically using this approach. - -**Large scale** -CUE's underlying formalism was developed for large-scale configuration. -Its import model incorporates best practices for large-scale engineering -and it is optimized for automation. -A key to this is advanced tooling. -The mathematical model underlying CUE's operations allows for -automation that is intractable for most other approaches. -CUE's `trim` command is an example of this. - - -### Tooling - -Automation is key. -Nowadays, a good chunk of code gets generated, analyzed, reformatted, -and so on by machines. -The CUE language, APIs, and tooling have been designed to allow for -machine manipulation. -Aspects of this are: - -- make the language easy to scan and parse, -- restrictions on imports, -- allow any piece of data to be split across files and generated - from different sources, -- define packages at the directory level, -- and of course its value and type model. - -The order independence also plays a key role in this. -It allows combining constraints from various sources without having -to define any order in which they are to be applied to get -predictable results. - - - +### Welcome to CUE! + +CUE is an +open-source +data validation language with its roots in logic programming. +It combines succinct yet clear syntax with powerful, flexible constraints that +enable data, schema, policy, and constraints to coexist seamlessly: + +{{< code-tabs >}} +{{< code-tab name="example.cue" language="cue" area="left" >}} +length: 20 & int +width: 10.1 & >10 // Must be greater than 10 +area: length * width +area: <=100 // Must be less than or equal to 100 +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="right" type="terminal" codetocopy="Y3VlIHZldCBleGFtcGxlLmN1ZQ==" >}} +$ cue vet example.cue +area: invalid value 202.0 (out of bound <=100): + ./example.cue:4:9 + ./example.cue:3:9 +{{< /code-tab >}} +{{< /code-tabs >}} + +In its mission to support people using the language and to promote its +adoption, the CUE project develops and publishes a variety of documentation and +tools, including: + +{{< table >}} +| Resource | Description +| --- | --- +| [The `cue` command]({{< relref "installation" >}}) | A command line tool that evaluates CUE, optionally combining it with structured data and other schema formats to validate, transform, and output data and constraints. +| [`cuelang.org/go` APIs](https://pkg.go.dev/cuelang.org/go/cue#section-documentation) | Go APIs that enable CUE's capabilities to be embedded in Go programs. +| [The CUE Language Specification]({{< ref "docs/reference/spec" >}}) | The formal specification of CUE that defines how implementations of the language should behave. +| [cuelang.org](/) | This website, including a foundational [tour through the language]({{< relref "/docs/tour" >}}), hands-on [tutorials]({{< relref "/docs/tutorial" >}}) and [how-to guides]({{< relref "/docs/howto" >}}), and informative [concept guides]({{< relref "/docs/concept" >}}). +| [The CUE Playground](/play/) | A browser-based tool that lets you try out CUE without installing anything. +{{< /table >}} +
+ +By design, CUE isn't a general-purpose programming language, +but its power and flexibility drive its use across a wide range of +applications. It's often used to define, validate and generate +[configuration]({{< relref "/docs/concept/how-cue-enables-configuration" >}}). +CUE also excels at +[validating data]({{< relref "/docs/concept/how-cue-enables-data-validation" >}}) +(such as JSON and YAML) against CUE schemas and policies, whilst also allowing schemas encoded in a variety of +[other formats]({{< relref "/docs/integration/" >}}) +(such as JSON Schema, Protobuf, and OpenAPI) to be used simultaneously. + +CUE's language features enables you to +[template data]({{< relref "/docs/tour/types/templates" >}}), +reducing boilerplate by specifying fields in bulk and allowing data's +important characteristics to stand out prominently. It's also used for +[code generation]({{< relref "/docs/concept/code-generation-and-extraction-use-case" >}}), +and to leverage existing schemas defined in formats such as +[Protobuf]({{< relref "/docs/concept/how-cue-works-with-protocol-buffers" >}}), +[JSON Schema]({{< relref "/docs/concept/how-cue-works-with-json-schema" >}}), +and +[Go]({{< relref "/docs/concept/how-cue-works-with-go" >}}) types. + +Over the next few pages you'll learn about some unique properties of the CUE +language, including: + +- why the merged concepts of "types" and "values" enable succinct and clear + constraints +- how some of CUE's core design principles combine so that the source of each + specific value is never in doubt -- no more hunting through confusing layers + of "overrides" to figure out which files disagree about a particular value +- the advanced tooling that's made possible by CUE's careful design, including + automated boilerplate removal! + + + +{{< warning >}} +Welcome to +}}" style="font-weight: normal;">the CUE community +-- but be warned ... \ +**Prolonged exposure to CUE can seriously affect how you approach data and configuration - for good!** +{{< /warning >}} + +*Next page:* [A Familiar Look and Feel]({{< relref "cue-is-familiar" >}}) diff --git a/hugo/content/en/docs/introduction/cue-is-different/index.md b/hugo/content/en/docs/introduction/cue-is-different/index.md new file mode 100644 index 000000000..f19b613fa --- /dev/null +++ b/hugo/content/en/docs/introduction/cue-is-different/index.md @@ -0,0 +1,315 @@ +--- +title: Some Unique Differences +weight: 20 +--- + +As you just saw in +[*A Familiar Look and Feel*]({{< relref "cue-is-familiar" >}}), +CUE can be used to encode data using a friendlier, more convenient syntax than +JSON - and some folks do just that. However many teams also rely on the +language's advanced capabilities to validate and secure their data and +configurations, and these features build on some rather unique characteristics. + +Let's take a look at some aspects of CUE that you might not have +experienced in a language before ... + +### Order Doesn't Matter + +Most languages require variables, fields, or data to be declared before they're +used - especially when one data value depends on the value of another. + +**CUE is different:** in CUE, *fields can be defined in any order*. + +This property drives many of CUE's most powerful features, and is referred to +as *order irrelevance*. It applies at all levels of granularity: + +- within the fields of each data *struct* (which is what JSON calls an "object"), +- across the fields and structs defined inside a single `.cue` file, +- when merging multiple `.cue` files that make up a CUE *package*. + +Order irrelevance flows from the rules of CUE's most fundamental operation, +called **unification**. Unification is the process by which CUE determines if +values are compatible with one another. It occurs when you explicitly use +the `&` operator, or implicitly when you mention a field multiple times. + +In formal terms, unification is defined so that the operation is associative, +commutative, idempotent, and recursive. This means that when values are +unified, CUE guarantees that every possible order in which they *might* be +combined produces the same underlying data structure, and therefore it doesn't +matter which *specific* ordering is chosen - even when unifying deeply nested +data. + +In practical terms, unification's rules mean that: + +- Data is immutable: if a field is made concrete by assigning it a specific + value, that value is fixed and cannot be changed. (This might appear + restrictive at first glance, but in reality CUE gives you plenty of options + to cater for the different problematic situations you might be imagining!) +- Data and constraints can be combined from multiple sources predictably and + efficiently, optionally using a convenient shorthand form for specifying + sparsely-populated structs. +- If a field is declared more than once, then all its assigned values must be + compatible with each other. When only specifying concrete data, this + simplifies down: all the assigned values must be *identical*. + +In this example, `A` is specified using implicit unification, and `B` is +specified using explicit unification: + +{{< code-tabs >}} +{{< code-tab name="data.cue" language="cue" area="top-left" >}} +A: 1 +B: 2 & 2 +A: 1 +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="top-right" type="terminal" codetocopy="Y3VlIGV4cG9ydCBkYXRhLmN1ZSAtLW91dCB5YW1s" >}} +$ cue export data.cue --out yaml +A: 1 +B: 2 +{{< /code-tab >}} +{{< /code-tabs >}} + +In this example you might be wondering if these unifications have any point - +surely no-one would *actually* specify\ +`B: 2 & 2`? Wouldn't `B: 2` be sufficient? + +In the case of the value of `B`, you'd be right to wonder - this is simply a +demonstration of explict unification so that you can recognise it, when it's +used later in more interesting situations. +Implicit unification, however (as with field `A`), becomes more interesting when +we learn that, by default, *the `cue` command implicitly unifies the top level +of all the data it's given*. +This means that when we invoke the `cue` command and tell it about different +sources of data, the evaluation result is the unification of all those sources. + +To see what this means in practice, we'll unify some CUE, JSON, and YAML, +simply by mentioning their three data files: + +{{< code-tabs >}} +{{< code-tab name="data.cue" language="cue" area="top-left" >}} +CUE: true +a: b: c: 1 +{{< /code-tab >}} +{{< code-tab name="data.yml" language="yml" area="top-right" >}} +YAML: true +a: + b: + e: 3 +{{< /code-tab >}} +{{< code-tab name="data.json" language="json" area="bottom-left" >}} +{ + "JSON": true, + "a": { + "b": { + "d": 2 + } + } +} +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="bottom-right" type="terminal" codetocopy="Y3VlIGV4cG9ydCBkYXRhLmN1ZSBkYXRhLnltbCBkYXRhLmpzb24gLS1vdXQganNvbg==" >}} +$ cue export data.cue data.yml data.json --out json +{ + "JSON": true, + "YAML": true, + "CUE": true, + "a": { + "b": { + "d": 2, + "e": 3, + "c": 1 + } + } +} +{{< /code-tab >}} +{{< /code-tabs >}} + +But watch what happens when we try and specify a field with two incompatible +values - the `cue` command emits an error message pointing out the +incompatibility and the evaluation fails: + +{{< code-tabs >}} +{{< code-tab name="data.cue" language="cue" area="top-left" >}} +source: "CUE" +{{< /code-tab >}} +{{< code-tab name="data.json" language="json" area="top-right" >}} +{ + "source": "JSON" +} +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="bottom" type="terminal" codetocopy="Y3VlIGV4cG9ydCBkYXRhLmN1ZSBkYXRhLmpzb24=" >}} +$ cue export data.cue data.json +source: conflicting values "CUE" and "JSON": + ./data.cue:1:9 + ./data.json:2:15 +{{< /code-tab >}} +{{< /code-tabs >}} + +CUE's design ensures that combining values in any order always gives the same +result. Later, in *Why CUE?*, we'll see some of the situations and use cases +that have elegant solutions enabled by this property. + +### Types Are Values + +In many languages there's a strong distinction between the concrete "values" +that a variable or field can be assigned (e.g. `"foo"`, `4.2`, or `true`) +and the "types" that describe *sets* of permissible values (e.g. strings, +floating-point numbers, or booleans, respectively). Most languages don't +provide a first-class mechanism to constrain values more precisely, leaving any +nuanced value constraints for the user to implement in code with explicit +runtime checks. + +**CUE is different:** *CUE doesn't differentiate between values and types*. + +This is a powerful concept that, as you'll see shortly, allows you to define +detailed constraints - but it also simplifies the process of learning and using +CUE. Because the language has a single, unified syntax for data and for +constraints, there is no separate schema or data definition language to learn. +The syntax even encompasses concepts such as sum types and enums - even null +coalescing is reduced down to this single construct. + +Whilst CUE does provide a type hierarchy that includes `string`, `float`, +and `bool`, along with `bytes`, `int`, `number`, `null`, `[...]` (list), and +`{...}` (struct), these are simply well-known names for constraints that limit +a field's value to well-defined sets or ranges. +The power of CUE comes from constructing precise constraints using these types +as a starting point, progressively layering and unifying additional constraints +built with CUE's rich set of primitives and built-in functions. + +Here's an example that uses +[bounds]({{< relref "docs/tour/types/bounds" >}}) (`>`, `<=`, etc) +to construct constraints that limit a number's value, unified with additional +constraints from a +[disjunction]({{< relref "/docs/tour/types/disjunctions" >}}) +("`|`") that requires a value to be one option from a prescribed set: + +{{< code-tabs >}} +{{< code-tab name="constraints.cue" language="cue" area="left" >}} +#over10: >10 +#under50: <50 +#from5To40: >=5 & <=40 +#options: 9 | 10 | 11 | 39 | 40 | 41 + +myNumber: int & #over10 & #under50 +myNumber: #from5To40 +myNumber: #options +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="right" type="terminal" codetocopy="Y3VlIGV2YWwgY29uc3RyYWludHMuY3Vl" >}} +$ cue eval constraints.cue +#over10: >10 +#under50: <50 +#from5To40: >=5 & <=40 +#options: 9 | 10 | 11 | 39 | 40 | 41 +myNumber: 11 | 39 | 40 +{{< /code-tab >}} +{{< /code-tabs >}} + +In the `cue eval` output, notice how CUE is able to *simplify* the constraints +that apply to the `myNumber` field by ruling out options that definitely aren't +possible, due to the implicit and explicit unification of the constraints that +we defined. + +Later, in *Why CUE?*, we'll see how CUE's merging of types, values, and +constraints into a single concept enables effective and concise schema and +policy validation. + +### Push Constraints, Don't Pull Them + +CUE's constraints act as data validators: + +{{< code-tabs >}} +{{< code-tab name="schema.cue" language="cue" area="top-left" >}} +A: int +B: float +C: string +{{< /code-tab >}} +{{< code-tab name="policy.cue" language="cue" area="top-left" >}} +A: <=99 +B: >4.2 +C: !="rm -rf /" +{{< /code-tab >}} +{{< code-tab name="data.yml" language="yml" area="top-right" >}} +A: 100 +B: 1.1 +C: "rm -rf /" +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="bottom" type="terminal" codetocopy="Y3VlIHZldCBzY2hlbWEuY3VlIHBvbGljeS5jdWUgZGF0YS55bWw=" >}} +$ cue vet schema.cue policy.cue data.yml +A: invalid value 100 (out of bound <=99): + ./policy.cue:1:4 + ./data.yml:1:4 +B: invalid value 1.1 (out of bound >4.2): + ./policy.cue:2:4 + ./data.yml:2:4 +C: invalid value "rm -rf /" (out of bound !="rm -rf /"): + ./policy.cue:3:4 + ./data.yml:3:4 + ./schema.cue:3:4 +{{< /code-tab >}} +{{< /code-tabs >}} + +*Pattern constraints* impose constraints on all the fields whose names match +their pattern. They're written as\ +`[pattern]: value`, where `pattern` must be either a value +of type string, or the wildcard value `_` (called "top"). Here's an +example of pattern constraints in action: + +{{< code-tabs >}} +{{< code-tab name="constraints.cue" language="cue" area="top-left" >}} +A: { + // Every field must be an int. + [_]: int + // Every field whose name starts with b must + // be 10 or greater. + [=~"^b"]: >=10 +} +{{< /code-tab >}} +{{< code-tab name="data.yml" language="yml" area="top-right" >}} +A: + foo: 4.2 + bar: 100 + baz: 5 +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="bottom" type="terminal" codetocopy="Y3VlIHZldCBjb25zdHJhaW50cy5jdWUgZGF0YS55bWw=" >}} +$ cue vet constraints.cue data.yml +A.foo: conflicting values 4.2 and int (mismatched types float and int): + ./constraints.cue:3:7 + ./data.yml:2:8 +A.baz: invalid value 5 (out of bound >=10): + ./constraints.cue:6:12 + ./data.yml:4:8 +{{< /code-tab >}} +{{< /code-tabs >}} + +Pattern constraints *don't* instantiate every field that their pattern might +match. If they *did*, then the example pattern constraint of `[string]: >10` +would bring every field with a name matching `string` into existence - i.e. +*every possible field*, which would take an unacceptable amount of time to +compute ... no matter how powerful your hardware! +So: a pattern constraint merely acts on fields by unifying its constraints with +*existing* fields that match its pattern. + +Many languages have a form of import, or perhaps inheritance, that allows +lower-level (more deeply nested) components to pull restrictions in from some +other entity by specifying them at the point of use. + +**CUE is different:** *unification and pattern constraints offer a simple way +to impose constraints from above*. + +CUE has a robust +[package]({{< relref "/docs/concept/modules-packages-instances" >}}) system +and a carefully designed +[module]({{< relref "/docs/reference/modules" >}}) system +which support imports and cross-entity data and constraint sharing. +Before reaching for those tools, however, it's worth discovering just how +flexible and powerful pattern constraints and unification can be! + +In *Why CUE?* we'll see how pattern constraints are the backbone of a highly +productive mechanism for reducing boilerplate data and configuration, whilst +also enforcing more centralised policy constraints on nested data. + +{{< warning >}} +**On the next page**: discover how CUE's differences have a real impact on data +handling and validation, and why teams trust it with their configurations at +all scales. +{{< /warning >}} + +*Next page:* [Why CUE?]({{< relref "why-cue" >}}) diff --git a/hugo/content/en/docs/introduction/cue-is-familiar/index.md b/hugo/content/en/docs/introduction/cue-is-familiar/index.md new file mode 100644 index 000000000..03015f6ab --- /dev/null +++ b/hugo/content/en/docs/introduction/cue-is-familiar/index.md @@ -0,0 +1,109 @@ +--- +title: A Familiar Look and Feel +weight: 10 +--- + +CUE will probably feel rather familiar if you've spent any time working with +data. CUE shares some syntax with JSON, but *significantly* improves the +experience of managing JSON by hand. + +In its simplest form, CUE looks a lot like JSON. +This is because CUE is defined to be a *superset* of JSON - which means that +all valid JSON is CUE, but not vice versa. +Editing JSON manually can be somewhat awkward, so CUE introduces several +conveniences to make writing and reading data easier: + +- comments are allowed, starting with `//` and extending to the end of the line +- field names without special characters don’t need to be quoted +- the outermost curly braces in a CUE file are optional +- commas after a field are optional (and are usually omitted) +- commas after the final element of a list are allowed +- literal multiline strings are allowed, and don't require newlines to be encoded +- nested structs containing one (or a few) fields have a convenient shorthand + +Here's some data encoded as commented CUE, alongside the equivalent JSON +document. Notice how the CUE lacks curly braces at the top and bottom, and +doesn't have commas after each field's value: + +{{< code-tabs >}} +{{< code-tab name="example.cue" language="cue" area="left" >}} +strings: { + singleLine: "Double quotes == string literal" + multiLine: """ + Multiline strings start and end with triple + double-quotes - no escaping of newlines! + """ +} + +// Many field names don't need to be quoted +// (but can be, if you want). +foo_Bar: 1 +baz2: 2.2 + +// Some field names do need quotes, such as those +// that start with numbers, or contain spaces, +// hyphens, or other special characters. +"qu ux": "3.33" +"4": "four" + +a: deeply: nested: field: "value" + +a: deeply: nested: struct: { + containing: "multiple" + fields: true +} + +// A list's final element can be followed by +// an optional comma. +aList: [ + "a", + "b", + "c", +] +anotherList: [1, 2, 3, 4, 5] +{{< /code-tab >}} +{{< code-tab name="TERMINAL" language="" area="right" type="terminal" codetocopy="Y3VlIGV4cG9ydCBleGFtcGxlLmN1ZSAtLW91dCBqc29u" >}} +$ cue export example.cue --out json +{ + "strings": { + "singleLine": "Double quotes == string literal", + "multiLine": "Multiline strings start and end with triple\ndouble-quotes - no escaping of newlines!" + }, + "foo_Bar": 1, + "baz2": 2.2, + "qu ux": "3.33", + "4": "four", + "a": { + "deeply": { + "nested": { + "field": "value", + "struct": { + "containing": "multiple", + "fields": true + } + } + } + }, + "aList": [ + "a", + "b", + "c" + ], + "anotherList": [ + 1, + 2, + 3, + 4, + 5 + ] +} +{{< /code-tab >}} +{{< /code-tabs >}} + +However, most CUE users don't choose CUE simply because it's nicer to handle +than JSON - that's just a pragmatic consequence of the language's design. +Folks adopt CUE because of its revolutionary features, and these stem from its +concepts and syntaxes that *won't* seem quite as familiar. + +We'll take a look at them next, in +[*Some Unique Differences*]({{< relref "cue-is-different" >}}) ... diff --git a/hugo/content/en/docs/introduction/effective-cue/index.md b/hugo/content/en/docs/introduction/effective-cue/index.md new file mode 100644 index 000000000..551062aa5 --- /dev/null +++ b/hugo/content/en/docs/introduction/effective-cue/index.md @@ -0,0 +1,126 @@ +--- +title: Effective CUE +weight: 40 +--- + +CUE is designed to help users and teams manage data and configuration across +all scales. Here are some strategies that help make your use of CUE more +effective. + +### Start with broad definitions and get progressively more specific + +In general, using CUE, we start with a broad definition of a type that +describes all possible instances of the type. We then narrow down these +definitions, often combining constraints from multiple sources (e.g. different +departments, teams, and users), until a concrete data instance remains. + +Here's a demonstration of this at a small scale. Keep in mind that CUE permits +this approach to be used effectively at far larger scales: + +{{< columns >}} +{{< code-tabs >}} +{{< code-tab name="schema.cue" language="cue" area="top-left" >}} +package geo + +#Municipality: { + name: string + pop: int + capital: bool +} +{{< /code-tab >}}{{< /code-tabs >}} +{{< columns-separator >}} +{{< code-tabs >}} +{{< code-tab name="native.cue" language="cue" area="top-left" >}} +package geo + +#LargeCapital: #Municipality & { + name: string + pop: >5M + capital: true +} +{{< /code-tab >}}{{< /code-tabs >}} +{{< columns-separator >}} +{{< code-tabs >}} +{{< code-tab name="data.cue" language="cue" area="top-left" >}} +package geo + +kinshasa: #LargeCapital & { + name: "Kinshasa" + pop: 16.315M + capital: true +} +{{< /code-tab >}}{{< /code-tabs >}} +{{< /columns >}} +### Separate configuration from computation + +Situations often arise where it looks like we'll need to do complex +computations to generate some configuration data. This approach is generally in +conflict with the goal of using a *simple* configuration language that permits +and encourages changes that can be understood in isolation, at speed, possibly +by non-domain experts. + +CUE has a stance that computation and configuration should be separated - and +*unification* makes this easy, because of *order irrelevance*. +Data that needs to be be computed can be generated outside CUE, and mixed into +a configuration using unification, CUE packages, and the `cue` command's +support for a variety of +[different data encodings]({{< relref "/docs/integration" >}}). + +The processes that compute this data can even be orchestrated by a CUE +[workflow command]({{< relref "docs/concept/how-cue-enables-configuration" >}}#tooling-and-automation), +enhancing developer workflows and performing inline computation input/output +validation. + +### Make appropriate choices for your level of scale + +The usefulness of a specific language can depend on the scale of the project +being developed, but adopting the "perfect" language for every new scale +milestone can put a cognitive strain on developers. CUE aims to minimize the +high cost of migrating from one language to another, as scaling requirements +change, by delivering solutions to data- and configuration-related tasks at all +scales: + +- **Small scale**: reducing boilerplate in configurations is not necessarily + the most effective thing to do. However, even at a small scale, repetition + can be tedious and lead to errors. For these situations, where it might be + best to keep small configurations specified and updated in isolation, CUE can + be used to define schemas that validate type-free + [JSON]({{< relref "/docs/howto/validate-json-using-cue" >}}) and + [YAML]({{< relref "/docs/howto/validate-yaml-using-cue" >}}) files. +- **Medium scale**: when the size or complexity of configurations reaches a + certain point, reducing boilerplate may start to become a more attractive + proposition. As you saw, earlier in this introduction, CUE can be used to + isolate such boilerplate, with the `cue trim` command removing it + automatically. It's not uncommon for thousands of lines to be obliterated + using this approach. +- **Large scale**: The + [formalism that underlies CUE](https://github.com/cue-lang/cue/blob/master/doc/ref/impl.md) + was specifically developed for large-scale configuration. CUE's import model + incorporates many best practices for large-scale engineering, and the + language is optimized for automation through advanced tooling. The + mathematical model and + [the logic of CUE]({{< relref "/docs/concept/the-logic-of-cue" >}}) + permits automation that's inaccessible for most other approaches - such as + `cue trim`'s capabilities. Read the next section, *Tooling*, for more on + this. + +### Know when to invest in building specialized tooling + +Automation is key. In contemporary operations, a large amount of code and +configuration gets generated, analyzed, reformatted, and mananged by machines. +The CUE language, its APIs, and its tooling have been designed to allow for +this mechanical manipulation, and you should anticipate taking advantage of +these capabilities as the scale and complexity of your mission grows. + +CUE's automation-friendly design is exemplified by these characteristics: + +- the language is easy to scan and parse +- there are specific restrictions on imports that prevent configuration graphs + of unbounded complexity +- data can be split across files and generated from different sources +- order irrelevance permits values and constraints to be mixed together easily, + without having to specify (or care about) the order in which they're combined +- packages are defined at the directory level +- its first-class value and type model is unparalleled. + +*Next page:* [Installing CUE]({{< relref "installation" >}}) diff --git a/hugo/content/en/docs/introduction/history/index.md b/hugo/content/en/docs/introduction/history/index.md index a416962fb..4bbff195b 100644 --- a/hugo/content/en/docs/introduction/history/index.md +++ b/hugo/content/en/docs/introduction/history/index.md @@ -1,6 +1,6 @@ --- -title: History -weight: 40 +title: The History of CUE +weight: 60 --- Although it is a very different language, the roots of CUE lie in GCL, diff --git a/hugo/content/en/docs/introduction/installation/index.md b/hugo/content/en/docs/introduction/installation/index.md index 903832809..d4a1543d0 100644 --- a/hugo/content/en/docs/introduction/installation/index.md +++ b/hugo/content/en/docs/introduction/installation/index.md @@ -1,6 +1,6 @@ --- -title: Installation -weight: 20 +title: Installing CUE +weight: 50 aliases: - /docs/install - /download diff --git a/hugo/content/en/docs/introduction/next-steps/index.md b/hugo/content/en/docs/introduction/next-steps/index.md new file mode 100644 index 000000000..3dd4907aa --- /dev/null +++ b/hugo/content/en/docs/introduction/next-steps/index.md @@ -0,0 +1,6 @@ +--- +title: Next Steps +weight: 70 +--- + +FIXME: words diff --git a/hugo/content/en/docs/introduction/why-cue/index.md b/hugo/content/en/docs/introduction/why-cue/index.md new file mode 100644 index 000000000..7d2755874 --- /dev/null +++ b/hugo/content/en/docs/introduction/why-cue/index.md @@ -0,0 +1,297 @@ +--- +title: Why CUE? +weight: 30 +--- + +The features that make CUE so effective are the result of its careful design, +which is the product of decades of experience in the data and cofiguration +space. The language's behaviours are formally defined in +[The CUE Language Specification]({{< relref "/docs/reference/spec" >}}), +but you don't need to study the spec in order to understand how CUE can make +life easier. + +If you work frequently with data or configuration, the CUE examples in this +introduction might have already given you ideas about how you could use CUE to +simplify, fortify, or otherwise improve your existing setup. +But before jumping in and getting started, it's worth reading through this +site's concept guides on how CUE enables various practices, such as +[configuration]({{< relref "/docs/concept/how-cue-enables-configuration" >}}) +and +[data validation]({{< relref "/docs/concept/how-cue-enables-data-validation" >}}), +and +[how CUE works with different technologies]({{< relref "/docs/integration" >}}). + +As further inspiration, here a more fully-formed example that demonstrates the +concepts and features that users of CUE often find compelling and powerful ... + +### Combining schema, policy, and data, and reducing boilerplate + +You've already seen how *unification* combines data with multiple, layered +constraints that limit the data's acceptable values. You saw how *schema* ("X +is an integer") can live seamlessly alongside *policy constraints* ("X must be +greater than 10") because of CUE's succinct and clear syntax. The *pattern +constraint* feature gives you a way to push down constraints from above, +instead of relying on the authors of nested data to reference your requirements. + +**Here's a demonstration of these concepts in action, working together to drive down complexity**. +Let's start with a hypothetical schema describing the contents of a +website -- its pages and structure -- along with some separate policy +constraints in `policy.cue`: + +{{< code-tabs >}} +{{< code-tab name="schema.cue" language="cue" area="top-left" >}} +package website + +#Page: { + title!: string // title is a required field, as every page must have a title. + urlPath!: string // urlPath is also required. + file!: string // The page's file contains its content. + date!: string // The page's date of publishing. + isDraft!: bool // Is the page part of the published site? + summary?: string // summary is an optional field, as some pages don't need a short summary. + author?: string | null // An optional author name. +} +{{< /code-tab >}}{{< code-tab name="policy.cue" language="cue" area="top-left" >}} +package website + +import ( + "strings" + "time" +) + +#Page: { + title?: strings.MinRunes(1) // title cannot be empty (could also be specified as !=""). + urlPath?: strings.MinRunes(1) // urlPath cannot be empty (equivalent to !=""). + file?: =~".html$" | =~".md$" // Content files can be HTML or Markdown. + date?: time.Time // time.Time validates a RFC3339 date-time. + summary?: strings.MaxRunes(150) // Our site layout requires page summaries to be limited in length. + author?: _ // Policy imposes no additional constraints on the author. + isDraft?: _ +} +{{< /code-tab >}}{{< /code-tabs >}} + +Here we've used several new language elements, particularly in `policy.cue`: + +- required (`!:`) and optional (`?:`) fields +- regular expression constraints (`=~`) +- importing built-in packages containing the CUE standard library (`import ( ... )`) +- using the standard library (`strings.MinRunes()`) + + +These features are explored and explained in the +[CUE language tour]({{< relref "/docs/tour" >}}) +but -- just for now! -- continue by reading on with the assumption that they +all function as you might expect from their names. + +Now we've set up some schema and policy constraints, we can start to populate +information about our small website's existing pages, marking them with a +publishing date that's in the past to reflect their "already published" state: + +{{< code-tabs >}} +{{< code-tab name="site.cue" language="cue" area="top-left" linenos="table" >}} +package website + +// pages is a struct whose fields each adhere to the +// #Page constraint imposed through a pattern constraint. +pages: [_]: #Page +pages: { + home: { + title: "Welcome to Widgets'R'We!" + urlPath: "/" + file: "pages/home.html" + summary: "The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: null + } + about: { + title: "About Widgets'R'We" + urlPath: "/about/" + file: "pages/about.html" + summary: "Information about the Widgets'R'We company." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: null + } + contact: { + title: "Contact Widgets'R'We" + urlPath: "/contact/" + file: "pages/contact.html" + summary: "Contact information for the Widgets'R'We company." + date: "1999-01-01T00:00:00Z" + isDraft: false + author: "Markus Marketing" + } +} +{{< /code-tab >}}{{< /code-tabs >}} + +We can run `cue vet` to check that each of our pages' data meets the +requirements of the `#Page` constraint. +Like many command line tools, it stays silent to indicate success: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHZldA==" } +$ cue vet +``` + +Whilst our site metadata is correctly specified, it's hard not to feel that it +seems a little *unwieldy*. There's quite a lot of duplicated information that +distracts from the important detail of each page - often called "boilerplate". +Let's use CUE to reduce it ... + +We'll use various CUE features to make our configuration shorter, which will +allow the most important information to stand out. Here's the CUE we'll use: + +{{< code-tabs >}} +{{< code-tab name="pageDefaults.cue" language="cue" area="top-left" >}} +package website + +pages?: [pageId=_]: { + isDraft: _ | *false + author: _ | *null + date: _ | *"1999-01-01T00:00:00Z" + urlPath: _ | *"/\(pageId)/" + file: _ | *"pages/\(pageId).html" +} +{{< /code-tab >}}{{< /code-tabs >}} + +In `pageDefaults.cue`, we: + +- use a [pattern constraint]({{< relref "/docs/tour/basics/folding-structs" >}}) + to unify fields inside each member of `pages`, creating what CUE calls a + [template]({{< relref "/docs/tour/types/templates" >}}), +- use the template to assign five fields a + [default value]({{< relref "/docs/tour/types/defaults" >}}) - a value that + only takes effect if a configuration doesn't specify a value elsewhere, +- bring an [alias]({{< relref "/docs/tour/references/aliases" >}}) into + existence (`pageId`) that contains the value of the identifier of each member + of the `pages` struct, and allows us to refer to each page's identifier + *inside* the template. This lets us derive two string fields' default values + dynamically. + +We again use `cue vet`, confirming that *adding* this CUE file to our `website` +package hasn't caused a problem: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHZldA==" } +$ cue vet +``` + +Next we need to *remove* the duplicate data from our `site.cue` metadata file - +a tedious and error-prone task, especially if our set of pages were larger, or +were split across more files. Or it *would* be tedious, if it weren't for CUE! + +The `cue trim` command performs **automatic removal of boilerplate** from +configurations where the data can be inferred from other constraints. +Let's try it out here: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHRyaW0=" } +$ cue trim +``` + +`cue trim`, just as with `cue vet`, stays silent if it was successful. Let's +take a look at the updated `site.cue` metadata file: +{{< code-tabs >}} +{{< code-tab name="site.cue" language="cue" area="top-left" linenos="table" >}} +package website + +// pages is a struct whose fields each adhere to the +// #Page constraint imposed through a pattern constraint. +pages: [_]: #Page +pages: { + home: { + title: "Welcome to Widgets'R'We!" + urlPath: "/" + summary: "The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East." + } + about: { + title: "About Widgets'R'We" + summary: "Information about the Widgets'R'We company." + } + contact: { + title: "Contact Widgets'R'We" + summary: "Contact information for the Widgets'R'We company." + author: "Markus Marketing" + } +} +{{< /code-tab >}}{{< /code-tabs >}} +Only the notable, *important* information about each page is left behind, with +the boring-but-necessary boilerplate configuration safely isolated in the +`pageDefaults.cue` file. + +Let's use `cue vet` again to confirm that the `#Page` schema still validates +our site metadata after the boilerplate was removed, and then run `cue export` +to show us the concrete data result of our changes: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHZldApjdWUgZXhwb3J0IC0tb3V0IHlhbWw=" } +$ cue vet +$ cue export --out yaml +pages: + home: + title: Welcome to Widgets'R'We! + urlPath: / + summary: The homepage of Widgets'R'We - manufacturer and purveyor of the finest left-angled reverse-clockwise widgets in the North East. + isDraft: false + author: null + date: "1999-01-01T00:00:00Z" + file: pages/home.html + about: + title: About Widgets'R'We + summary: Information about the Widgets'R'We company. + isDraft: false + author: null + date: "1999-01-01T00:00:00Z" + urlPath: /about/ + file: pages/about.html + contact: + title: Contact Widgets'R'We + summary: Contact information for the Widgets'R'We company. + author: Markus Marketing + isDraft: false + date: "1999-01-01T00:00:00Z" + urlPath: /contact/ + file: pages/contact.html +``` + +We can see that we exported the same data that we started with ... but with a +significantly clearer and more succinct representation behind the scenes! +Lastly, we'll introduce a new CUE file to the `website` package that *attempts* +to modify an existing page's title: + +{{< code-tabs >}} +{{< code-tab name="updates.cue" language="cue" area="top-left" >}} +package website + +pages: { + about: title: "Cyber Widgets 2000" +} +{{< /code-tab >}}{{< /code-tabs >}} + +The `cue vet` command complains, because the title already has a concrete value +specified elsewhere and data is immutable in CUE. +Notice that it tells us the locations of the conflicting values (encoded as +`filename:line-number:character-number`) so that we can find and resolve this +kind of mistake faster and easier: + +```text { title="TERMINAL" type="terminal" codeToCopy="Y3VlIHZldA==" } +$ cue vet +pages.about.title: conflicting values "Cyber Widgets 2000" and "About Widgets'R'We": + ./site.cue:13:12 + ./updates.cue:4:16 +``` + +Here's a reminder of the concepts and features that we just used: + +- *Constraints* were *unified* across the multiple files in our CUE package. +- *Pattern constraints* allowed us to push constraints down from above. +- *Templates* and *default values* gave us a way to centralise repetitive + information without repeating it. +- *Aliases* gave us way to construct a template's field values dynamically. +- `cue vet` checked that data adhered to its constraints. +- `cue export` produced concrete data. +- `cue trim` automatically removed duplicate information from our data source. + +{{< warning >}} +**On the next page**: learn some strategies to make effective use of CUE. +{{< /warning >}} + +*Next page:* [Effective CUE]({{< relref "effective-cue" >}})