diff --git a/.github/workflows/check.yml b/.github/workflows/check.yml new file mode 100644 index 0000000..8446f70 --- /dev/null +++ b/.github/workflows/check.yml @@ -0,0 +1,14 @@ +name: Check Markdown links + +on: + push: + workflow_dispatch: + schedule: + - cron: "0 3 1 * *" + +jobs: + markdown-link-check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@main + - uses: michaelkreil/check-markdown-links@main diff --git a/readme.md b/readme.md index 043a406..d00f2cf 100644 --- a/readme.md +++ b/readme.md @@ -3,7 +3,7 @@ ## Versions - [**v2.0**](v02/readme.md) <- current version -- [v1.0](v01) +- [v1.0](v01/container/readme.md) ## Implementations diff --git a/v01/container/readme.md b/v01/container/readme.md index 5f84211..0387aff 100644 --- a/v01/container/readme.md +++ b/v01/container/readme.md @@ -11,7 +11,7 @@ The file is composed of several parts: - **index** of these tiles 4. **index** of all blocks -

+

## file @@ -79,7 +79,7 @@ The file is composed of several parts: - Each `block` contains data of up to 256x256 (= 65536) `tile`s. - Levels 0-8 can be stored with one `block` each. level 9 might contain 512x512 `tile`s so 4 `block`s are necessary. -

+

- Each `block` contains the concatenated `tile` blobs and ends with a `tile_index`. - Neither the order of `block`s in the `file` nor the order of `tile`s in a `block` matters as long as their indexes are correct. @@ -96,7 +96,7 @@ The file is composed of several parts: - `tile`s are read horizontally then vertically - `j = (row - row_min)*(col_max - col_min + 1) + (col - col_min)` -

+

- identical `tile`s can be stored once and referenced multiple times to save storage space - if a `tile` does not exist, the length of `tile` is zero diff --git a/v02/container/readme.md b/v02/container/readme.md index b9492e3..3e33adc 100644 --- a/v02/container/readme.md +++ b/v02/container/readme.md @@ -1,140 +1,181 @@ -# Versatiles container format specification v02 +# Versatiles Container Format Specification v2.0 -## Overview +## Contents +- [Versatiles Container Format Specification v2.0](#versatiles-container-format-specification-v20) + - [Contents](#contents) + - [1. Introduction](#1-introduction) + - [2. General Guidelines](#2-general-guidelines) + - [3. File structure](#3-file-structure) + - [3.1. File Header](#31-file-header) + - [3.1.1. Value `tile_format`](#311-value-tile_format) + - [3.1.2. Value `precompression`](#312-value-precompression) + - [3.2. Metadata Chunk](#32-metadata-chunk) + - [3.3. Blocks](#33-blocks) + - [3.3.1. Tile Blobs](#331-tile-blobs) + - [3.3.2. Tile Index](#332-tile-index) + - [3.4. Block Index](#34-block-index) + - [4. Glossary](#4-glossary) -- All numbers are stored in big endian byte order. -- Tiles are organised in the XYZ scheme, and not the TMS scheme. So tiles with x=0,y=0 are in the top left corner. +## 1. Introduction -The file is composed of four parts: -1. starting with a [**`file_header`**](#file_header) -2. followed by compressed [**`metadata`**](#metadata) -3. followed by several [**`block`s**](#block), where each block consists of: - - concatenated [**`tile_blobs`**](#tile_blobs) - - followed by [**`tile_index`**](#tile_index) as an index of these tiles -4. followed by [**`block_index`**](#block_index) as an index of all blocks +This document defines the Versatiles Container Format v2.0, which describes the structure and encoding mechanisms for efficiently storing large numbers of map tiles. -| File Format | +## 2. General Guidelines + +- **Byte order:** All numeric values are encoded in big-endian byte order. +- **Tile organization:** Tiles are organised according to the XYZ scheme, with the origin (x=0, y=0) located at the top-left (northwest) corner. + +## 3. File structure + +The Versatiles Container format consists of four main components: + +1. **[File Header](#31-file-header):** Introduces the file, details global properties, and indicates the locations of [Metadata](#32-metadata-chunk) and [Block Index](#34-block-index). +2. **[Metadata](#32-metadata-chunk):** Provides detailed information about the tileset, including attribution and layer definitions. +3. **[Blocks](#33-blocks):** Aggregates tiles into larger units (Blocks) for efficient storage and access, each containing [**Tile Blobs**](#331-tile-blobs) and [**Tile Index**](#332-tile-index). +4. **[Block Index](#34-block-index):** Acts as a parent directory for all blocks within the file. + +| File Format | |:-------------------------------:| | ![File Format](file_format.svg) | -## `file_header` -- has a length of 66 bytes -- all offsets are relative to start of the file -| offset | length | type | description | -|--------|--------|--------|-------------------------| -| 0 | 14 | string | `"versatiles_v02"` | -| 14 | 1 | u8 | `tile_format` | -| 15 | 1 | u8 | `tile_precompression` | -| 16 | 1 | u8 | min zoom level | -| 17 | 1 | u8 | max zoom level | -| 18 | 4 | i32 | bbox min x (10⁷ × lon) | -| 22 | 4 | i32 | bbox min y (10⁷ × lat) | -| 26 | 4 | i32 | bbox max x (10⁷ × lon) | -| 30 | 4 | i32 | bbox max y (10⁷ × lat) | -| 34 | 8 | u64 | offset of `metadata` | -| 42 | 8 | u64 | length of `metadata` | -| 50 | 8 | u64 | offset of `block_index` | -| 58 | 8 | u64 | length of `block_index` | +### 3.1. File Header +- **Length:** 66 bytes. +- **Location:** At the start of the file. +- **Purpose:** Outlines essential file properties and indicates subsequent section locations. +- all offsets are relative to start of the file -### `tile_format` values: +| Offset | Length | Type | Description | +|-------:|-------:|:------:|------------------------------------------| +| 0 | 14 | string | File identifier (`"versatiles_v02"`) | +| 14 | 1 | u8 | `tile_format` value | +| 15 | 1 | u8 | `precompression` value | +| 16 | 1 | u8 | minimum zoom level | +| 17 | 1 | u8 | maximum zoom level | +| 18 | 4 | i32 | bbox min x (10⁷ × lon) | +| 22 | 4 | i32 | bbox min y (10⁷ × lat) | +| 26 | 4 | i32 | bbox max x (10⁷ × lon) | +| 30 | 4 | i32 | bbox max y (10⁷ × lat) | +| 34 | 8 | u64 | offset of [Metadata](#32-metadata-chunk) | +| 42 | 8 | u64 | length of [Metadata](#32-metadata-chunk) | +| 50 | 8 | u64 | offset of [Block Index](#34-block-index) | +| 58 | 8 | u64 | length of [Block Index](#34-block-index) | -| value | type | mime | -|--------|----------|----------------------------| -| `0x00` | bin | *application/octet-stream* | -| `0x10` | png | *image/png* | -| `0x11` | jpg | *image/jpeg* | -| `0x12` | webp | *image/webp* | -| `0x13` | avif | *image/avif* | -| `0x14` | svg | *image/svg+xml* | -| `0x20` | pbf | *application/x-protobuf* | -| `0x21` | geojson | *application/geo+json* | -| `0x22` | topojson | *application/topo+json* | -| `0x23` | json | *application/json* | -### `tile_precompression` +#### 3.1.1. Value `tile_format` -allowed values: -- `0`: uncompressed -- `1`: gzip compressed -- `2`: brotli compressed +| Value hex | Value dec | Type | Mime | +|----------:|----------:|:--------:|----------------------------| +| `0x00` | `0` | bin | *application/octet-stream* | +| `0x10` | `8` | png | *image/png* | +| `0x11` | `9` | jpg | *image/jpeg* | +| `0x12` | `10` | webp | *image/webp* | +| `0x13` | `11` | avif | *image/avif* | +| `0x14` | `12` | svg | *image/svg+xml* | +| `0x20` | `16` | pbf | *application/x-protobuf* | +| `0x21` | `17` | geojson | *application/geo+json* | +| `0x22` | `18` | topojson | *application/topo+json* | +| `0x23` | `19` | json | *application/json* | -## `metadata` -Content of `tiles.json`. Encoded in UTF-8, compressed with `$tile_precompression`. -If no metadata is specified, offset and length must be `0`. +#### 3.1.2. Value `precompression` +[Metadata](#32-metadata-chunk) and all [Tile Blobs](#331-tile-blobs) are pre-compressed with: -## Blocks +| Value | Method | +|-------|--------------| +| `0` | Uncompressed | +| `1` | gzip | +| `2` | Brotli | -### `block_index` -- Brotli compressed data structure -- Empty `block`s are not stored -- For each block, `block_index` contains a 33 bytes long record: -| offset | length | type | description | -|-----------|--------|------|---------------------------| -| 0 + 33*i | 1 | u8 | `level` | -| 1 + 33*i | 4 | u32 | `column`/256 | -| 5 + 33*i | 4 | u32 | `row`/256 | -| 9 + 33*i | 1 | u8 | `col_min` (0..255) | -| 10 + 33*i | 1 | u8 | `row_min` (0..255) | -| 11 + 33*i | 1 | u8 | `col_max` (0..255) | -| 12 + 33*i | 1 | u8 | `row_max` (0..255) | -| 13 + 33*i | 8 | u64 | offset of `block` in file | -| 21 + 33*i | 8 | u64 | length of `tile_blobs` | -| 29 + 33*i | 4 | u32 | length of `tile_index` | +### 3.2. Metadata Chunk -- Since a `block` only consists of `tile_blobs` appended by `tile_index`, the length of `block` must be the sum of the lengths of `tile_blobs` and `tile_index`. -- Note: To efficiently find the `block` that contains the `tile` you are looking for, use a data structure such as a "map", "dictionary" or "associative array" and fill it with the data from the `block_index`. +- **Content:** Encapsulates `tiles.json`, detailing tileset metadata. +- **Encoding:** UTF-8. +- **Compression:** Defined by the [`precompression`](#312-value-precompression) flag in the [File Header](#31-file-header). +- **Note:** The absence of Metadata is indicated by zero offsets and lengths in the [File Header](#31-file-header). -### `block` -- Each `block` is like a "super tile" and contains data of up to 256×256 (= 65536) `tile`s. -- Levels 0-8 can be stored with one `block` each. Level 9 can contain up to 512×512 `tile`s so up to 4 `block`s are necessary. -- Number of Blocks: `max(1, pow(2, (level-7))` +### 3.3. Blocks +- **Structure:** Blocks act as aggregators for up to 256×256 tiles. +- **Zoom Levels:** Single Blocks can span entire zoom levels (0-8). Higher zoom levels (>8) may require multiple Blocks. +- Maximum number of Blocks per zoom level: `pow(4, max(0, level - 8))`. -| `block`s per level | +| Blocks per level | |:---------------------------------:| | ![Level Blocks](level_blocks.svg) | +- Each Block contains concatenated [Tile Blobs](#331-tile-blobs) and ends with a [Tile Index](#332-tile-index). +- Neither the [Tile Blobs](#331-tile-blobs) in a Block nor Blocks in the file need to follow any particular order. + -- Each `block` contains the concatenated `tile` blobs and ends with a `tile_index`. -- Neither `tile`s in a `block` nor `block`s in a `file` have to be sorted in any kind of order, as long as their indexes are correct. -- But it is recommended to sort the `tile`s in a `block` in an ascending order (e.g. to improve caching efficiency when reading/converting the whole file). +#### 3.3.1. Tile Blobs -## Tiles +- Tile Blobs are concatenated binary data, each containing one tile. All tiles have the same format and are pre-compressed. +- **Format:** Each Tile Blob has the same file format, determined by the [`tile_format`](#311-value-tile_format) code in the [File Header](#31-file-header). +- **Compression:** Each Tile Blob is compressed as specified by the [`precompression`](#312-value-precompression) flag in the [File Header](#31-file-header). -### `tile_index` -- Brotli compressed data structure -- `tile`s are read horizontally then vertically -- `index = (row - row_min)*(col_max - col_min + 1) + (col - col_min)` -- (`col_min`, `row_min`, `col_max`, `row_max` are specified in `block_index`) -- identical `tile`s can be stored once and referenced multiple times to save storage space -- if a `tile` does not exist, the length of `tile` is `0` -- offsets of `tile_blob`s are relative to the beginning of the `block`. So the offset of the first `tile_blob` should always be `0`. -| offset | length | type | description | -|--------|--------|------|----------------------------------| -| 12*i | 8 | u64 | offset of `tile_blob` in `block` | -| 12*i+8 | 4 | u32 | length of `tile_blob` | +#### 3.3.2. Tile Index +- **Compression:** Brotli. +- **Purpose:** Maps coordinates of tiles within a block to their respective binary position and length. +- Tiles are ordered horizontally then vertically +- `index = (row - row_min) * (col_max - col_min + 1) + (col - col_min)` +- (`col_min`, `row_min`, `col_max`, `row_max` are specified in [Block Index](#34-block-index)) +- identical [Tile Blobs](#331-tile-blobs) can be stored once and referenced multiple times to save storage space +- if a tile does not exist, the length of Tile Blob is `0` +- offsets of [Tile Blobs](#331-tile-blobs) are relative to the beginning of the Block. So the offset of the first Tile Blob should always be `0`. -| index of `tile_blob`s | +| Offset | Length | Type | Description | +|--------|-------:|:----:|------------------------------| +| 12*i | 8 | u64 | offset of Tile Blob in Block | +| 12*i+8 | 4 | u32 | length of Tile Blob | + +| index of Tile Blobs | |:-------------------------------:| | ![Block Tiles](block_tiles.svg) | -### `tile_blob` -- each tile is a PNG/PBF/... file as data blob -- compressed with `$tile_precompression` +### 3.4. Block Index + +- **Compression:** Brotli. +- **Function:** Provides a directory for locating [Blocks](#33-blocks) within the container file. +- Empty Blocks are not stored. +- Each 33-byte entry within the Block Index is structured as follows: + +| Offset | Length | Type | Description | +|----------:|-------:|:----:|-----------------------------------------| +| 0 + 33*i | 1 | u8 | `level` | +| 1 + 33*i | 4 | u32 | `column`/256 | +| 5 + 33*i | 4 | u32 | `row`/256 | +| 9 + 33*i | 1 | u8 | `col_min` (0..255) | +| 10 + 33*i | 1 | u8 | `row_min` (0..255) | +| 11 + 33*i | 1 | u8 | `col_max` (0..255) | +| 12 + 33*i | 1 | u8 | `row_max` (0..255) | +| 13 + 33*i | 8 | u64 | offset of Block in file | +| 21 + 33*i | 8 | u64 | length of [Tile Blobs](#331-tile-blobs) | +| 29 + 33*i | 4 | u32 | length of [Tile Index](#332-tile-index) | + +- Since a Block consists only of [Tile Blobs](#331-tile-blobs) appended by the [Tile Index](#332-tile-index), the length of Block must be the sum of the lengths of the [Tile Blobs](#331-tile-blobs) and the [Tile Index](#332-tile-index). +- Note: To efficiently find the Block containing the tile you are looking for, use a data structure such as a "map", "dictionary" or "associative array" and fill it with the data from the Block Index. + + + +## 4. Glossary + +- **Blob:** A chunk of binary data. [Object storage on Wikipedia](https://en.wikipedia.org/wiki/Object_storage) +- **Block:** A composite unit containing up to 256×256 tiles. +- **Brotli:** A compression algorithm known for its efficiency and performance. It offers better compression than gzip. [Brotli on Wikipedia](https://en.wikipedia.org/wiki/Brotli) +- **Tile:** A square geographic area at a specified zoom level, containing map information as an image or as vector data. diff --git a/v02/readme.md b/v02/readme.md new file mode 100644 index 0000000..7fa3f01 --- /dev/null +++ b/v02/readme.md @@ -0,0 +1,156 @@ +# VersaTiles Stack Specification v2.1 + +## Foreword + +This document provides detailed specifications for the VersaTiles stack, a modular framework for creating, storing, serving and displaying map tiles. The stack is structured into four distinct layers, each dedicated to a specific aspect of tile creation, management and presentation. +The specification uses the terminology "SHALL", "SHOULD" and "MAY" to denote mandatory requirements, strong recommendations and optional practices respectively. + +## 1. Stack Layers Overview + +- **Generator:** Creates map tiles from geographic data sources such as OpenStreetMap (OSM). +- **Server:** Manages the storage and distribution of map tiles to clients. +- **Network:** Enhances security and optimizes performance through load balancing, TLS and caching. +- **Frontend:** Provides the user interface for map interaction, leveraging libraries like MapLibre GL JS. + +### 2. Generator Layer + +The Generator creates map tiles from geographic data sources such as OpenStreetMap (OSM). + +[^req]: Requirement 2.1 The Generator layer SHALL produce map tiles based on input from geographic data sources, such as OpenStreetMap (OSM). + +2.1.2 The Generator SHALL output tiles in formats compliant with the specifications detailed in later sections of this document. + +**Requirements:** +- Tiles must be packaged in a [*.versatiles containers](container/readme.md). +- Vector tiles must conform to the [Shortbread schema](https://shortbread-tiles.org/). +- Containers must include detailed metadata compliant with [TileJSON 3.0.0](https://github.com/mapbox/tilejson-spec/tree/master/3.0.0), specifically: + - `attribution` detailing source data copyrights. + - `vector_layers` describing the vector tiles' layered composition. + +**Recommendations:** +- Use optimal compression techniques to efficiently reduce tile size without compromising data integrity. Recommended methods include: + - Brotli compression for vector tiles. + - WebP format for raster tiles to improve loading efficiency and reduce bandwidth. + +### 3. Server + +2.2.1 The Server layer SHALL manage the storage of generated map tiles. + +2.2.2 The Server SHALL distribute map tiles to clients upon request, adhering to the HTTP/HTTPS protocols. + +2.2.3 The Server SHOULD implement mechanisms to determine its public URL dynamically to facilitate proper resource referencing. + +### 2.3 Network + +2.3.1 The Network layer SHALL implement security measures, including, but not limited to, Transport Layer Security (TLS). + +2.3.2 The Network layer SHOULD provide load balancing functionalities to distribute incoming traffic across multiple server instances efficiently. + +2.3.3 The Network layer MAY implement caching strategies to optimize the performance and scalability of tile delivery. + +### 2.4 Frontend + +2.4.1 The Frontend layer SHALL provide an interactive interface for map visualization and interaction. + +2.4.2 The Frontend SHOULD leverage established libraries such as MapLibre GL JS to ensure compatibility and performance. + +2.4.3 The Frontend MAY support additional functionalities, such as layer toggling and dynamic data visualization, to enhance user experience. + + + + +--- + + + +# VersaTiles Stack Specification v2.0 + +This document provides detailed specifications for the VersaTiles stack, a modular framework designed for the creation, hosting, and display of map tiles. The stack is structured into four distinct layers, each dedicated to a specific aspect of tile management and presentation. + +## Stack Layers Overview + +- **Generator:** Creates map tiles from geographic data sources such as OpenStreetMap (OSM). +- **Server:** Manages the storage and distribution of map tiles to clients. +- **Network:** Enhances security and optimizes performance through load balancing, TLS and caching. +- **Frontend:** Provides the user interface for map interaction, leveraging libraries like MapLibre GL JS. + +## Generator Layer Specifications + +Responsible for tile generation, adhering to defined standards and formats. + +**Requirements:** +- Tiles must be packaged in a [*.versatiles containers](container/readme.md). +- Vector tiles must conform to the [Shortbread schema](https://shortbread-tiles.org/). +- Containers must include detailed metadata compliant with [TileJSON 3.0.0](https://github.com/mapbox/tilejson-spec/tree/master/3.0.0), specifically: + - `attribution` detailing source data copyrights. + - `vector_layers` describing the vector tiles' layered composition. + +**Recommendations:** +- Use optimal compression techniques to efficiently reduce tile size without compromising data integrity. Recommended methods include: + - Brotli compression for vector tiles. + - WebP format for raster tiles to improve loading efficiency and reduce bandwidth. + +## Server Layer Specifications + +The Server layer is responsible for serving map tiles via HTTP. + +**Requirements:** +- Must recognize and process [*VersaTiles containers*](container/readme.md). +- Requires knowledge of its public URL for resource referencing. +- Mandatory HTTP service provision. +- Adopts a structured folder hierarchy for organized tile and metadata access: + - `/tiles/`: Central directory for tile retrieval. + - `/tiles/sources.json`: Comprehensive index of available tile sources. + - `/tiles/{name}/{z}/{x}/{y}`: Standardized tile access endpoints. + - `/tiles/{name}/tiles.json`: Incorporates a legitimate [TileJSON 3.0.0](https://github.com/mapbox/tilejson-spec/tree/master/3.0.0) document. + - `/assets/`: Storage for additional resources like sprites, glyphs, styles, and MapLibre GL JS files. + - `/assets/sprites/` + - `/assets/glyphs/` + - `/assets/glyphs/{name}`: Font names must only contain letters, numbers and underscore + - `/assets/glyphs/fonts.json`: Index of available fonts. + - `/assets/styles/` + - `/assets/maplibre/maplibre.*`: JavaScript and CSS of newest MapLibre GL JS + +**Recommendations:** +- Implementation of Cross-Origin Resource Sharing (CORS) to facilitate resource sharing across different domains. +- Configurability via `config.yaml` for custom server setup, including domain configuration, IP/port listening preferences, operational modes (e.g., development vs. production), tile source definition, and static content management: + ```yaml + # public URL + domain: 'https://example.org' # required + + # listen to + listen_ip: '0.0.0.0' # default: 0.0.0.0 + listen_port: 3000 # default: 8080 + + # set to true use only minimal recompression and only if necessary, e.g. for development + fast: true # default: false + + # set tile sources, required + tile_sources: + - { name: 'osm', source: './planet.versatiles' } + - { name: 'landsat', source: 'https://example.org/landsat.versatiles' } + + # set static content, optional + static_content: + - { source: './styles', prefix: 'assets/styles' } + - { source: './frontend.tar' } + ``` + +## Network Layer Specifications + +The Network layer ensures security and performance. + +**Requirements:** +- Efficient request routing to the Server layer. +- Minimal routing of duplicated requests. + +**Recommendations:** +- Implementation of Transport Layer Security (TLS) to encrypt data in transit. +- Effective caching strategies to reduce load times and server strain for frequently accessed tiles. + +## Frontend Layer Specifications + +The Frontend layer is the user interface that renders and interacts with the tiles. + +**Requirements:** +- Must ensure compatibility with both vector and raster tiles.