Skip to content

Commit

Permalink
Initial commit
Browse files Browse the repository at this point in the history
  • Loading branch information
rob-metalinkage authored Jun 26, 2024
0 parents commit c11e0ba
Show file tree
Hide file tree
Showing 29 changed files with 873 additions and 0 deletions.
20 changes: 20 additions & 0 deletions .github/workflows/process-bblocks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: Validate and process Building Blocks
on:
workflow_dispatch:
push:
branches:
- master
- main

permissions:
contents: write
pages: write
id-token: write

jobs:
validate-and-process:
name: Validate and process
uses: opengeospatial/bblocks-postprocess/.github/workflows/validate-and-process.yml@master
# with:
# sparql_username: sparql
# sparql_password: Secr3T
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
build-local/
.idea/
21 changes: 21 additions & 0 deletions JSONLD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# JSON-LD (Semantic Uplift)

Building Blocks defining JSON schemas can be annotated with JSON-LD contexts using either:
- including a file (_context.jsonld_) in the building block directory
- using the property `x-jsonld-context` in the _schema.(yaml/json) for the building block - e.g. `x-jsonld-context: ../../../sosa-ssn.jsonld`

The JSON LD context:

1. Maps JSON elements to URIs (which can be URIs of a richer semantic model)
2. Allows validation of complex logical constraints using SHACL Rules to [validate examples](TESTING.md#SHACL)
3. [Perform transforms](TXFORMS.md) to any other RDF model and validate results

## Modularity support

JSON-LD contexts are very complex and hard to debug if the schema is at all complex.

The Building Blocks design allows automatic combination of contexts based on the schema re-use patterns.

## Context design

_TBD: document local contexts and use of @base mappings. _
65 changes: 65 additions & 0 deletions JSONSCHEMA-PROFILING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Profiling JSON Schemas

Profiling JSON schemas is a complex subject. This document is a placeholder for a description of best practices and support tooling.

## Why is this challenging?

The JSON Schema specification and tooling landscape is complex. Versions of the OpenAPI Specification (OAS) use different versions of JSON-Schema and tool support varies.

In particular, reuse mechanisms such as $dynamicRef may not be available.

## Version-agnostic Building Blocks

The Building Block post-processing tooling automatically generates OAS 3.0 and OAS 3.1 compatible schemas. It is recommended to develop new building blocks using improved modularity and reuse support in modern schema versions, and allow the Building Block to create a "down-compiled version".

## OAS 3.0 Compatibility

OGC APIs are currently bound to OAS v3.0 which limits JSON schema patterns that can be supported.

The implication is that currently it is typical necessary to recreate complex structural hierarchies and compose into "allOf[]" structures in order to place constraints into a location.

e.g. to make the relatively simple constraints that a "SurveyObservationCollection" is a collection "SurveyObservation" objects, and must declare that some schema for describing "SensorType" is used, a significant portion of the structure must be detailed.

```json

"SurveyVectorObsCollection": {
"allOf": [
{
"$ref": "https://opengeospatial.github.io/ogcapi-sosa/build/annotated/unstable/sosa/features/observationCollection/schema.json"
},
{
"properties": {
"features": {
"type": "array",
"items": {
"$ref": "#/$defs/SurveyVectorObsFeature"
}
},
"properties": {
"properties": {
"madeBySensor": {
"$ref": "#/$defs/SensorType"
}
},
"required": [
"madeBySensor"
]
}
}
}
]
}
}
```

## Future options

A number of strategies are being considered to simplify this. At this stage alternative approaches:
- define a new constraint language that can be directly compiled from simple statements into the target schema constraint
- use the most compact form in more recent JSON schema - potentially requiring extension hooks in the base schemas - and compile to legacy forms with full structure replication
- work with JSON schema community to define new capabilities designed for easier profiling in a future version of JSON
- create a "wizard" tool to write constraints

All these have significant impact and effort implications.

Follow updates at [this issue](https://github.com/opengeospatial/bblock-template/issues/2)
62 changes: 62 additions & 0 deletions PROFILES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Profiles and BuildingBlocks

BuildingBlocks support two modes:
- composition
- profiling

[USAGE](USAGE.md) describes common aspects. This document described profiling mechanisms.

**Profiles** allow all the underlying details of base standards to be automatically included in testing and validation - this _encapsulates_ the underlying complexity of base specifications.

This dramatically **simplifies** profiles in terms of both development and usage, and ensures **consistency** and conformance of profiles with base specifications.

In particular, if base specifications use the OGC BuildingBlocks then profiles can _leverage_ all the effort in design, testing and validation capabilities.

Thus profiles also use the same structures, so they can be profiled in turn.

## What is a profile?

A profile defines a set of constraints on a base specification. Implementations of profiles conform to the base specification.

Because many technologies like JSON and RDF are permissive (by default) about additional information being present, definition of an *extension* is effectively defining a *constraint* on how additional information should be represented.

## Profiles of profiles...

Profiles can be designed as separate re-usable sets of constraints that can be reused - for example a time-series of water-quality monitoring observations could be specified as a profile of both a time-series profile of Observations and a water-quality profile for the results of such observations.
In turn the time-series profile could defined as data structure using GeoJSON, or Coverage JSON. The water-quality content requirements could be described using constraints independent of the data structure.

## What forms of constraints are possible?

The **OGC BuildingBlock** model supports a range of possible constraint approaches. The goal is to make such constraints **_machine-readable_** to the extent possible.

Constraints SHOULD be defined in a form that allows for **_validation_** of test cases and examples.

Built-in support is provided for automatic validation of the following forms:
- project metadata (description)
- well-formed example encoding (JSON, TTL)
- JSON schema (for JSON examples) for **structure**
- SHACL (Shapes Constraint Language for RDF) for **content** and **logical consistency**

In addition [custom validators](VALIDATORS) can be added to the validation workflow.

Using a JSON-LD context "semantic uplift" of JSON to RDF supports use of SHACL and other forms of validators to

## Testing

Test cases should be provided for each component part of a specification. This requires a minimal conformant **base example** that the specific test case can be added to.

(Note consideration is being given to making such a baseline example resuable by reference instead of duplication, and potentially derived automatically from declared schema)

Testing should start by validating the **base example** passes all declared constraints, then for each profile constraint:
- identifying a set of valid cases that should conform to the constraint, testing each aspect
- creating a copy of the base under the **/tests/** folder with a name indicating which constraint and case is being tested - e.g. **my-building-block/tests/mything-property-b-number.json**
- adding the specific example to the example
- creating one or more failure tests with **-fail** file name endings - e.g. **my-building-block/tests/mything-property-b-number-fail.json**








33 changes: 33 additions & 0 deletions RDF.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# RDF-only building blocks

Building Blocks can be defined that use RDF only,

A RDF building block can

1. Define RDF (TTL) examples how to use the Semantic model
2. Apply SHACL Rules to [validate examples](TESTING.md#SHACL)
3. [Perform transforms](TXFORMS.md) and validate results

Test cases and examples as either TTL or JSONLD will undergo syntax and SHACL validation.

`examples.yaml` can have embedded TTL - eg.

```
- title: Example of SOSA ObservationCollection
comment:
This class is a target for the SOSA v 1.1 update.
snippets:
- language: turtle
code: |-
@prefix sosa: <http://www.w3.org/ns/sosa/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix eg: <http://example.org/my-feature/> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
eg:c1 a sosa:ObservationCollection ;
sosa:hasMember eg:pop1999, eg:pop2000 ;
sosa:observedProperty <http://dbpedia.org/ontology/population> ;
.
```
17 changes: 17 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# OGC Building Block template

This template provides a working example of an [OGC Building Block](https:blocks.ogc.org). For more info see [the Documentation](https://ogcincubator.github.io/bblocks-docs/).

This template is a working automation recipe to define, test and document a set of Building Blocks.

The automation-generated documentation for this example is here: [https://opengeospatial.github.io/bblock-template/](https://opengeospatial.github.io/bblock-template/)

[Examples of using this with typical applications of OGC standards](https://github.com/ogcincubator/bblocks-examples)

Please replace the contents of this README with information about your Building Block(s).

# How to use this template

[More information on design and usage](https://github.com/opengeospatial/bblock-template/blob/master/USAGE.md)


37 changes: 37 additions & 0 deletions SCHEMAS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Reusing existing JSON Schemas

Building Blocks can be defined to reuse existing JSON schemas in a more sophisticated way.

A re-used schema can be:

1. [profiled (by extension or constraints)](JSONSCHEMA-PROFILING.md)
2. Mapped to a [semantic (RDF)](RDF.md) model allowing richer specification of constraints.
3. [Tested](TESTING.md) with examples and test cases.

## How to reuse

This is simply a matter of referencing the reused schema in the building block schema(.json|.yaml):

```json
{ "$ref": "http://somestablelocation.org/schema.json" }
```

## How to reuse a building block with its added components...

this is done in a two-step process:

1. in the (_bblocks-config.yaml)[_bblocks-config.yaml] file tell the processing where to find building block registers:
```yaml
schema-mapping:
default: https://opengeospatial.github.io/bblocks/annotated-schemas/

imports:
- default
- https://opengeospatial.github.io/ogcapi-sosa/build/register.json
```
The default is the OGC master register of building blocks.
2. use the _bblocks:{id}_ syntax as href in schema $ref elements.
This means your building block will inherit all json-ld contexts and SHACL rules from the referenced building block automatically and apply during [testing](TESTING.md).
62 changes: 62 additions & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Testing

Building Blocks have powerful automated testing capabilities using the built-in github actions.

These can be supplemented with additional custom validation and transformation processes as required.


## Examples

Examples defined in the `examples.yaml` (inline or by file reference) get validated and included in generated documentation.

Test cases defines in the `tests/` subdirectory of each building block get validated. Additional or external tests
can be added in `tests.yaml` as a list of objects with a `ref` property pointing to the test resource's location,
and optionally defining the `output-filename` and/or `require-fail` properties (for more information, see the
example `tests.yaml` file provided in the template and
[the JSON Schema for `tests.yaml`](https://github.com/opengeospatial/bblocks-postprocess/blob/master/ogc/bblocks/extra-tests-schema.yaml).

In each case, the `/build/tests/` directory contains a set of validation outputs.

Validation includes the following steps:

1. (if JSON and context supplied) JSON-LD uplift ( {testcase}.ttl generated)
2. (if JSON schema supplied) JSON schema validation
3. (if SHACL rules defined) SHACL validation

A summary report is produced at `/build/tests/report.html`.

This is linked from the generated building block index.

## SHACL Validation

SHACL rules can be defined in a ```rules.shacl``` file or any other files or URLs in the bblocks.json:

```
"shaclRules": [
"vocabs-shacl.ttl"
]
"shaclClosures": [
"../../vocabularies/terms.ttl",
```

"shaclClosures" refers to additional files with RDF data required to perform validation - such as checking the types of related objects.

this is particularly useful for relatively small, static vocabularies (e.g. "codelists") that form part of the specification realised by the building block

## Tools

In addition to built-in testing capabilities the following online tools can be helpful in developing and debugging different layers of the design:

* [JSON Schema validator](https://www.jsonschemavalidator.net/)
* [JSON-LD Playground](https://json-ld.org/playground/)
* [SHACL Validator](https://shacl-play.sparna.fr/play/validate)

Hint - to use the JSON Schema validator with a published schema you can create a wrapper such as

```json
{
"$ref": "https://my.org/schema.json?version"
}
```

(updating ```version``` forces the validator to pick up any changes in the published schema.)
Loading

0 comments on commit c11e0ba

Please sign in to comment.