generated from opengeospatial/bblock-template
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit c11e0ba
Showing
29 changed files
with
873 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
name: Validate and process Building Blocks | ||
on: | ||
workflow_dispatch: | ||
push: | ||
branches: | ||
- master | ||
- main | ||
|
||
permissions: | ||
contents: write | ||
pages: write | ||
id-token: write | ||
|
||
jobs: | ||
validate-and-process: | ||
name: Validate and process | ||
uses: opengeospatial/bblocks-postprocess/.github/workflows/validate-and-process.yml@master | ||
# with: | ||
# sparql_username: sparql | ||
# sparql_password: Secr3T |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
build-local/ | ||
.idea/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
# JSON-LD (Semantic Uplift) | ||
|
||
Building Blocks defining JSON schemas can be annotated with JSON-LD contexts using either: | ||
- including a file (_context.jsonld_) in the building block directory | ||
- using the property `x-jsonld-context` in the _schema.(yaml/json) for the building block - e.g. `x-jsonld-context: ../../../sosa-ssn.jsonld` | ||
|
||
The JSON LD context: | ||
|
||
1. Maps JSON elements to URIs (which can be URIs of a richer semantic model) | ||
2. Allows validation of complex logical constraints using SHACL Rules to [validate examples](TESTING.md#SHACL) | ||
3. [Perform transforms](TXFORMS.md) to any other RDF model and validate results | ||
|
||
## Modularity support | ||
|
||
JSON-LD contexts are very complex and hard to debug if the schema is at all complex. | ||
|
||
The Building Blocks design allows automatic combination of contexts based on the schema re-use patterns. | ||
|
||
## Context design | ||
|
||
_TBD: document local contexts and use of @base mappings. _ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
# Profiling JSON Schemas | ||
|
||
Profiling JSON schemas is a complex subject. This document is a placeholder for a description of best practices and support tooling. | ||
|
||
## Why is this challenging? | ||
|
||
The JSON Schema specification and tooling landscape is complex. Versions of the OpenAPI Specification (OAS) use different versions of JSON-Schema and tool support varies. | ||
|
||
In particular, reuse mechanisms such as $dynamicRef may not be available. | ||
|
||
## Version-agnostic Building Blocks | ||
|
||
The Building Block post-processing tooling automatically generates OAS 3.0 and OAS 3.1 compatible schemas. It is recommended to develop new building blocks using improved modularity and reuse support in modern schema versions, and allow the Building Block to create a "down-compiled version". | ||
|
||
## OAS 3.0 Compatibility | ||
|
||
OGC APIs are currently bound to OAS v3.0 which limits JSON schema patterns that can be supported. | ||
|
||
The implication is that currently it is typical necessary to recreate complex structural hierarchies and compose into "allOf[]" structures in order to place constraints into a location. | ||
|
||
e.g. to make the relatively simple constraints that a "SurveyObservationCollection" is a collection "SurveyObservation" objects, and must declare that some schema for describing "SensorType" is used, a significant portion of the structure must be detailed. | ||
|
||
```json | ||
|
||
"SurveyVectorObsCollection": { | ||
"allOf": [ | ||
{ | ||
"$ref": "https://opengeospatial.github.io/ogcapi-sosa/build/annotated/unstable/sosa/features/observationCollection/schema.json" | ||
}, | ||
{ | ||
"properties": { | ||
"features": { | ||
"type": "array", | ||
"items": { | ||
"$ref": "#/$defs/SurveyVectorObsFeature" | ||
} | ||
}, | ||
"properties": { | ||
"properties": { | ||
"madeBySensor": { | ||
"$ref": "#/$defs/SensorType" | ||
} | ||
}, | ||
"required": [ | ||
"madeBySensor" | ||
] | ||
} | ||
} | ||
} | ||
] | ||
} | ||
} | ||
``` | ||
|
||
## Future options | ||
|
||
A number of strategies are being considered to simplify this. At this stage alternative approaches: | ||
- define a new constraint language that can be directly compiled from simple statements into the target schema constraint | ||
- use the most compact form in more recent JSON schema - potentially requiring extension hooks in the base schemas - and compile to legacy forms with full structure replication | ||
- work with JSON schema community to define new capabilities designed for easier profiling in a future version of JSON | ||
- create a "wizard" tool to write constraints | ||
|
||
All these have significant impact and effort implications. | ||
|
||
Follow updates at [this issue](https://github.com/opengeospatial/bblock-template/issues/2) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
# Profiles and BuildingBlocks | ||
|
||
BuildingBlocks support two modes: | ||
- composition | ||
- profiling | ||
|
||
[USAGE](USAGE.md) describes common aspects. This document described profiling mechanisms. | ||
|
||
**Profiles** allow all the underlying details of base standards to be automatically included in testing and validation - this _encapsulates_ the underlying complexity of base specifications. | ||
|
||
This dramatically **simplifies** profiles in terms of both development and usage, and ensures **consistency** and conformance of profiles with base specifications. | ||
|
||
In particular, if base specifications use the OGC BuildingBlocks then profiles can _leverage_ all the effort in design, testing and validation capabilities. | ||
|
||
Thus profiles also use the same structures, so they can be profiled in turn. | ||
|
||
## What is a profile? | ||
|
||
A profile defines a set of constraints on a base specification. Implementations of profiles conform to the base specification. | ||
|
||
Because many technologies like JSON and RDF are permissive (by default) about additional information being present, definition of an *extension* is effectively defining a *constraint* on how additional information should be represented. | ||
|
||
## Profiles of profiles... | ||
|
||
Profiles can be designed as separate re-usable sets of constraints that can be reused - for example a time-series of water-quality monitoring observations could be specified as a profile of both a time-series profile of Observations and a water-quality profile for the results of such observations. | ||
In turn the time-series profile could defined as data structure using GeoJSON, or Coverage JSON. The water-quality content requirements could be described using constraints independent of the data structure. | ||
|
||
## What forms of constraints are possible? | ||
|
||
The **OGC BuildingBlock** model supports a range of possible constraint approaches. The goal is to make such constraints **_machine-readable_** to the extent possible. | ||
|
||
Constraints SHOULD be defined in a form that allows for **_validation_** of test cases and examples. | ||
|
||
Built-in support is provided for automatic validation of the following forms: | ||
- project metadata (description) | ||
- well-formed example encoding (JSON, TTL) | ||
- JSON schema (for JSON examples) for **structure** | ||
- SHACL (Shapes Constraint Language for RDF) for **content** and **logical consistency** | ||
|
||
In addition [custom validators](VALIDATORS) can be added to the validation workflow. | ||
|
||
Using a JSON-LD context "semantic uplift" of JSON to RDF supports use of SHACL and other forms of validators to | ||
|
||
## Testing | ||
|
||
Test cases should be provided for each component part of a specification. This requires a minimal conformant **base example** that the specific test case can be added to. | ||
|
||
(Note consideration is being given to making such a baseline example resuable by reference instead of duplication, and potentially derived automatically from declared schema) | ||
|
||
Testing should start by validating the **base example** passes all declared constraints, then for each profile constraint: | ||
- identifying a set of valid cases that should conform to the constraint, testing each aspect | ||
- creating a copy of the base under the **/tests/** folder with a name indicating which constraint and case is being tested - e.g. **my-building-block/tests/mything-property-b-number.json** | ||
- adding the specific example to the example | ||
- creating one or more failure tests with **-fail** file name endings - e.g. **my-building-block/tests/mything-property-b-number-fail.json** | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,33 @@ | ||
# RDF-only building blocks | ||
|
||
Building Blocks can be defined that use RDF only, | ||
|
||
A RDF building block can | ||
|
||
1. Define RDF (TTL) examples how to use the Semantic model | ||
2. Apply SHACL Rules to [validate examples](TESTING.md#SHACL) | ||
3. [Perform transforms](TXFORMS.md) and validate results | ||
|
||
Test cases and examples as either TTL or JSONLD will undergo syntax and SHACL validation. | ||
|
||
`examples.yaml` can have embedded TTL - eg. | ||
|
||
``` | ||
- title: Example of SOSA ObservationCollection | ||
comment: | ||
This class is a target for the SOSA v 1.1 update. | ||
snippets: | ||
- language: turtle | ||
code: |- | ||
@prefix sosa: <http://www.w3.org/ns/sosa/> . | ||
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> . | ||
@prefix eg: <http://example.org/my-feature/> . | ||
@prefix skos: <http://www.w3.org/2004/02/skos/core#> . | ||
eg:c1 a sosa:ObservationCollection ; | ||
sosa:hasMember eg:pop1999, eg:pop2000 ; | ||
sosa:observedProperty <http://dbpedia.org/ontology/population> ; | ||
. | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
# OGC Building Block template | ||
|
||
This template provides a working example of an [OGC Building Block](https:blocks.ogc.org). For more info see [the Documentation](https://ogcincubator.github.io/bblocks-docs/). | ||
|
||
This template is a working automation recipe to define, test and document a set of Building Blocks. | ||
|
||
The automation-generated documentation for this example is here: [https://opengeospatial.github.io/bblock-template/](https://opengeospatial.github.io/bblock-template/) | ||
|
||
[Examples of using this with typical applications of OGC standards](https://github.com/ogcincubator/bblocks-examples) | ||
|
||
Please replace the contents of this README with information about your Building Block(s). | ||
|
||
# How to use this template | ||
|
||
[More information on design and usage](https://github.com/opengeospatial/bblock-template/blob/master/USAGE.md) | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
# Reusing existing JSON Schemas | ||
|
||
Building Blocks can be defined to reuse existing JSON schemas in a more sophisticated way. | ||
|
||
A re-used schema can be: | ||
|
||
1. [profiled (by extension or constraints)](JSONSCHEMA-PROFILING.md) | ||
2. Mapped to a [semantic (RDF)](RDF.md) model allowing richer specification of constraints. | ||
3. [Tested](TESTING.md) with examples and test cases. | ||
|
||
## How to reuse | ||
|
||
This is simply a matter of referencing the reused schema in the building block schema(.json|.yaml): | ||
|
||
```json | ||
{ "$ref": "http://somestablelocation.org/schema.json" } | ||
``` | ||
|
||
## How to reuse a building block with its added components... | ||
|
||
this is done in a two-step process: | ||
|
||
1. in the (_bblocks-config.yaml)[_bblocks-config.yaml] file tell the processing where to find building block registers: | ||
```yaml | ||
schema-mapping: | ||
default: https://opengeospatial.github.io/bblocks/annotated-schemas/ | ||
|
||
imports: | ||
- default | ||
- https://opengeospatial.github.io/ogcapi-sosa/build/register.json | ||
``` | ||
The default is the OGC master register of building blocks. | ||
2. use the _bblocks:{id}_ syntax as href in schema $ref elements. | ||
This means your building block will inherit all json-ld contexts and SHACL rules from the referenced building block automatically and apply during [testing](TESTING.md). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
# Testing | ||
|
||
Building Blocks have powerful automated testing capabilities using the built-in github actions. | ||
|
||
These can be supplemented with additional custom validation and transformation processes as required. | ||
|
||
|
||
## Examples | ||
|
||
Examples defined in the `examples.yaml` (inline or by file reference) get validated and included in generated documentation. | ||
|
||
Test cases defines in the `tests/` subdirectory of each building block get validated. Additional or external tests | ||
can be added in `tests.yaml` as a list of objects with a `ref` property pointing to the test resource's location, | ||
and optionally defining the `output-filename` and/or `require-fail` properties (for more information, see the | ||
example `tests.yaml` file provided in the template and | ||
[the JSON Schema for `tests.yaml`](https://github.com/opengeospatial/bblocks-postprocess/blob/master/ogc/bblocks/extra-tests-schema.yaml). | ||
|
||
In each case, the `/build/tests/` directory contains a set of validation outputs. | ||
|
||
Validation includes the following steps: | ||
|
||
1. (if JSON and context supplied) JSON-LD uplift ( {testcase}.ttl generated) | ||
2. (if JSON schema supplied) JSON schema validation | ||
3. (if SHACL rules defined) SHACL validation | ||
|
||
A summary report is produced at `/build/tests/report.html`. | ||
|
||
This is linked from the generated building block index. | ||
|
||
## SHACL Validation | ||
|
||
SHACL rules can be defined in a ```rules.shacl``` file or any other files or URLs in the bblocks.json: | ||
|
||
``` | ||
"shaclRules": [ | ||
"vocabs-shacl.ttl" | ||
] | ||
"shaclClosures": [ | ||
"../../vocabularies/terms.ttl", | ||
``` | ||
|
||
"shaclClosures" refers to additional files with RDF data required to perform validation - such as checking the types of related objects. | ||
|
||
this is particularly useful for relatively small, static vocabularies (e.g. "codelists") that form part of the specification realised by the building block | ||
|
||
## Tools | ||
|
||
In addition to built-in testing capabilities the following online tools can be helpful in developing and debugging different layers of the design: | ||
|
||
* [JSON Schema validator](https://www.jsonschemavalidator.net/) | ||
* [JSON-LD Playground](https://json-ld.org/playground/) | ||
* [SHACL Validator](https://shacl-play.sparna.fr/play/validate) | ||
|
||
Hint - to use the JSON Schema validator with a published schema you can create a wrapper such as | ||
|
||
```json | ||
{ | ||
"$ref": "https://my.org/schema.json?version" | ||
} | ||
``` | ||
|
||
(updating ```version``` forces the validator to pick up any changes in the published schema.) |
Oops, something went wrong.