feat(copc): Now module for COPC format (#3030)

docs/modules/copc/README.md

@@ -0,0 +1,25 @@
+# Overview
+
+![copc-logo](../../images/logos/copc-logo-80.png)
+
+<p class="badges">
+  <img src="https://img.shields.io/badge/From-v4.1-blue.svg?style=flat-square" alt="From-v4.1" />
+</p>
+
+The `@loaders.gl/copc` module provides support for the [COPC](/docs/modules/copc/formats/copc) format. 
+
+## Installation
+
+```bash
+npm install @loaders.gl/core @loaders.gl/copc
+```
+
+## APIs
+
+| Loader                                                       |
+| ------------------------------------------------------------ |
+| [`COPCSource`](/docs/modules/copc/api-reference/copc-source) |
+
+## Attribution
+
+This module is a fork of Connor Manning's awesome [copc.js](https://github.com/connormanning/copc.js/) module, under MIT license.

docs/modules/copc/api-reference/copc-source.md

@@ -0,0 +1,3 @@
+# COPCSource 🚧
+
+TBA

docs/modules/copc/formats/copc.md

@@ -0,0 +1,16 @@
+# COPC - Cloud-Optimized Point Cloud
+
+- *[Specification at COPC.io](https://copc.io/)*
+- *[Video Overview](https://www.youtube.com/watch?v=rWkKKZYN86A)*
+
+A COPC file is a LAZ 1.4 file that stores point data organized in a clustered octree. It contains a VLR (LAS Variable Length Record) that describe the octree organization of data that are stored in LAZ 1.4 chunks.
+
+Data organization of COPC is modeled after the [EPT data format](https://entwine.io/en/latest/entwine-point-tile.html), but COPC clusters the storage of the octree as variably-chunked LAZ data in a single file. This allows the data to be consumed sequentially by any reader than can handle variably-chunked LAZ 1.4 (LASzip, for example), or as a spatial subset for readers that interpret the COPC hierarchy. 
+
+## Implementation
+
+Key aspects distinguish an organized COPC LAZ file from an LAZ 1.4 that is unorganized:
+
+- It MUST contain ONLY LAS PDRFs 6, 7, or 8 formatted data
+- It MUST contain a COPC info VLR
+- It MUST contain a COPC hierarchy VLR

docs/specifications/cloud-native-geospatial.md

@@ -0,0 +1,37 @@
+# Cloud Native Geospatial Formats
+
+Author: Ib Green
+
+There is a lot of excitement in the geospatial community about “cloud native geospatial formats”.
+
+## The Formats
+
+Notable characteristics of these formats are found below
+
+| Format                    | Description                                                                                                                                                                                              |
+| ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Big Data:                 | CNGFs are designed for big geospatial data                                                                                                                                                               |
+| Serverless                | All CNGFs can be loaded by a client directly from files on e.g. s3 or a CDN without an intermediary server.                                                                                              |
+| Chunked/Tiled/Pre-indexed | Data in CNGFs is structured in a way that allows clients (backend and front-end clients) to do partial reads from big files, loading just the data that is required for the geospatial region they need. |
+| HTTP range requests       | A core technology that is being exploited is the ability to do a standard REST HTTP GET call but reading just the required range of bytes from a large, potentially too-large-to-load file.              |
+
+In some cases, the files are really collections of smaller files (e.g. “tiles”) that can be read using range requests and clients are expected to load a small subset of the tiles at a time rather than the full file.
+
+Key contenders in cloud native geospatial format category are:
+
+| Format | Description |
+| --- | --- |
+| flatgeobuf (spec) | A project of passion from Björn Hartell (Norway) that started as a compact binary geojson alternative. 
+The format initially gained interest because of its beautiful streaming capabilities (demo). Bjorn has kept working on it and added spatial indexing, making a good case for counting it as a cloud native geospatial format. |
+| Geoparquet | Parquet is a binary columnar data format optimized for storage. Files can be chunked so that it is possible to read a range of rows without reading the whole file. Geoparquet defines metadata fields specifying which buffers contain WKB-encoded geometry. 
+| Geoarrow |  Arrow is a binary columnar data format optimized for in-memory usage. Files can be chunked so that it is possible to read a range of rows without reading the whole file. Like geoparquet, geoarrow defines almost identical metadata fields specifying which buffers contain WKB-encoded geometry. |
+| COG (Cloud Optimized Geotiff) |
+| pmtiles | Stores of a large number of tiles in a single very big file, indexed for partial (HTTP range request) reads.  Can be cleaner than having directories of 10 - 100K tile files. |
+| COPC | Store massive point clouds in a single file with additive subclouds being available for range HTTP requests. |
+| STAC | STAC is not a file format but a catalog format, that complements the CNGFs above. It is generally used to describe collections of cloud optimized geotiff files, however it is a general geospatial data catalog format that is increasingly being used for more general geospatial data archives. E.g. both Amazon and Microsoft offer petabyte sized archives of satellite data indexed by STAC. |
+
+## References
+
+- [Cloud Native Geospatial Foundation]9https://cloudnativegeo.org/) foundation.
+- [Radiant Earth](https://radiant.earth/) - Non-profit foundation, CEO Jed Sundvall (worked on Cloud-Optimized GeoTiff standard), 
+

modules/copc/README.md

@@ -0,0 +1,7 @@
+# @loaders.gl/copc
+
+[loaders.gl](https://loaders.gl/docs) is a collection of framework-independent 3D and geospatial parsers and encoders.
+
+This module contains loaders for the COPC format.
+
+For documentation please visit the [website](https://loaders.gl).

modules/copc/package.json

@@ -0,0 +1,51 @@
+{
+  "private": true,
+  "name": "@loaders.gl/copc",
+  "version": "4.3.0-alpha.3",
+  "description": "Framework-independent loader for the COPC format",
+  "license": "MIT",
+  "type": "module",
+  "publishConfig": {
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/visgl/loaders.gl"
+  },
+  "keywords": [
+    "webgl",
+    "loader",
+    "3d",
+    "mesh",
+    "point cloud",
+    "PCD"
+  ],
+  "types": "dist/index.d.ts",
+  "main": "dist/index.cjs",
+  "module": "dist/index.js",
+  "exports": {
+    ".": {
+      "import": "./dist/index.js",
+      "require": "./dist/index.cjs",
+      "types": "./dist/index.d.ts"
+    }
+  },
+  "sideEffects": false,
+  "files": [
+    "src",
+    "dist",
+    "README.md"
+  ],
+  "scripts": {
+    "pre-build": "npm run build-bundle && npm run build-bundle -- --env=dev",
+    "build-bundle": "# ocular-bundle ./src/index.ts"
+  },
+  "dependencies": {
+    "@loaders.gl/images": "4.3.0-alpha.3",
+    "@loaders.gl/loader-utils": "4.3.0-alpha.3",
+    "@loaders.gl/mvt": "4.3.0-alpha.3",
+    "@loaders.gl/schema": "4.3.0-alpha.3",
+    "copc": "0.0.6"
+  },
+  "gitHead": "c95a4ff72512668a93d9041ce8636bac09333fd5"
+}

modules/copc/src/README.md

@@ -0,0 +1,41 @@
+# copc.js fork notes
+
+copc.js was forked after some consideration:
+
+- Split out LAZ functionality (use `@loaders.gl/las` module).
+- Adapt to loaders.gl dynamic source system (replace `Getter`).
+- Avoid bundling issues (e.g. dynamic import of fs) as loaders.gl already handles these.
+- Modernize / simplify code (see below)
+
+If some of the loaders.gl changes could be upstreamed we might consider unforking.
+
+## Code "simplifications"
+
+- Remove dated typescript constructs (e.g. ES2015 module syntax is preferred over namespaces)
+- Multiple exports of same name (Hierarchy etc)
+- Avoid importing Node.js dependencies
+- Consolidate large number of small source code files.
+
+## Original License
+
+MIT License
+
+Copyright (c) 2021 Connor Manning
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

modules/copc/src/copc-source.ts

@@ -0,0 +1,205 @@
+// loaders.gl
+// SPDX-License-Identifier: MIT
+// Copyright (c) vis.gl contributors
+
+import type {Schema, Field, DataType} from '@loaders.gl/schema';
+import type {
+  Source,
+  LoaderOptions,
+  GetTileParameters,
+  GetTileDataParameters
+} from '@loaders.gl/loader-utils';
+import type {TileSource} from '@loaders.gl/loader-utils';
+// import {getArrayTypeFromDataType} from '@loaders.gl/schema';
+import {DataSource, DataSourceProps, resolvePath} from '@loaders.gl/loader-utils';
+
+import {Copc, Hierarchy, Dimension, Getter} from 'copc';
+
+type COPCMetadata = Record<string, unknown>;
+
+type GetNodeParameters = {
+  nodeIndex: {
+    x: number;
+    y: number;
+    z: number;
+    d: number;
+  };
+  columns?: string[];
+  offset?: number;
+  limit?: number;
+};
+
+const VERSION = '1.0.0';
+
+/**
+ * Creates vector tile data sources for COPC urls or blobs
+ */
+export const COPCSource = {
+  name: 'COPC',
+  id: 'copc',
+  module: 'copc',
+  version: VERSION,
+  extensions: ['laz'],
+  mimeTypes: ['application/octet-stream'],
+  options: {url: undefined!, copc: {}},
+  type: 'copc',
+  fromUrl: true,
+  fromBlob: true,
+
+  testURL: (url: string) => url.endsWith('.pmtiles'),
+  createDataSource: (url: string | Blob, props: COPCTileSourceProps) =>
+    new COPCTileSource(url, props)
+} as const satisfies Source<COPCTileSource, COPCTileSourceProps>;
+
+export type COPCTileSourceProps = DataSourceProps & {
+  attributions?: string[];
+  copc?: {
+    loadOptions?: LoaderOptions; // COPCLoaderOptions;
+    // TODO - add options here
+  };
+};
+
+/**
+ * A COPC data source
+ * @note Can be either a raster or vector tile source depending on the contents of the COPC file.
+ */
+export class COPCTileSource extends DataSource implements TileSource {
+  data: string | Blob;
+  props: COPCTileSourceProps;
+  mimeType: string | null = null;
+  metadata: Promise<COPCMetadata>;
+
+  protected _initPromise: Promise<{
+    copc: Copc;
+    hierarchy: Hierarchy.Subtree;
+    rootNode: Hierarchy.Node;
+  }>;
+  protected _urlOrGetter: string | Getter;
+
+  constructor(data: string | Blob, props: COPCTileSourceProps) {
+    super(props);
+    this.props = props;
+    const url = typeof data === 'string' ? resolvePath(data) : '';
+    this.data = data;
+    this._urlOrGetter = url;
+    this._initPromise = this._initCopc(url);
+    this.metadata = this.getMetadata();
+  }
+
+  async getSchema(): Promise<Schema> {
+    const {copc, rootNode} = await this._initPromise;
+    const view = await Copc.loadPointDataView(this._urlOrGetter, copc, rootNode);
+
+    const fields: Field[] = [];
+    for (const [name, dimension] of Object.entries(view.dimensions)) {
+      if (dimension) {
+        const type = getDataTypeFromDimension(dimension);
+        fields.push({name, type, nullable: false});
+      }
+    }
+
+    return {fields, metadata: {}};
+  }
+
+  async getMetadata(): Promise<COPCMetadata> {
+    const {copc} = await this._initPromise;
+    const metadata: COPCMetadata = {
+      formatSpecificMetadata: copc
+    };
+    return metadata;
+  }
+
+  async getTile(tileParams: GetTileParameters): Promise<number[] | null> {
+    const nodeIndex = {x: tileParams.x, y: tileParams.y, z: tileParams.z, d: 0};
+    return this.getPoints({nodeIndex});
+  }
+
+  async getTileData(parameters: GetTileDataParameters): Promise<unknown | null> {
+    throw new Error('Not implemented');
+  }
+
+  async getPoints(parameters: GetNodeParameters) {
+    const {copc} = await this._initPromise;
+    const node = await this.getNode(parameters);
+    const view = node && (await Copc.loadPointDataView(this._urlOrGetter, copc, node));
+    if (!view) {
+      return null;
+    }
+
+    // console.log('Dimensions:', view.dimensions);
+
+    const schema = await this.getSchema();
+    const columnNames = schema.fields.map((field) => field.name);
+    const columnGetters = columnNames.map((name) => view.getter(name));
+
+    // const offset = parameters.offset || 0;
+    // const limit = Math.min(parameters.limit ?? view.pointCount, view.pointCount - offset);
+    // const ArrayType = getArrayTypeFromDataType(limit);
+
+    function getXyzi(index: number): number[] {
+      return columnGetters.map((get) => get(index));
+    }
+    const point = getXyzi(0);
+    // console.log('Point:', point);
+    return point;
+  }
+
+  async getNode(parameters: GetNodeParameters): Promise<Hierarchy.Node | undefined> {
+    const {hierarchy} = await this._initPromise;
+    const {x, y, z, d} = parameters.nodeIndex;
+    const key = `${x}-${y}-${z}-${d}`;
+    const {[key]: node} = hierarchy.nodes;
+    return node;
+  }
+
+  async _initCopc(url: string) {
+    const copc = await Copc.create(this._urlOrGetter);
+    const hierarchy = await Copc.loadHierarchyPage(this._urlOrGetter, copc.info.rootHierarchyPage);
+    const {['0-0-0-0']: rootNode} = hierarchy.nodes;
+    if (!rootNode) {
+      throw new Error(`Failed to load COPC hierarchy root node ${url}`);
+    }
+    return {copc, hierarchy, rootNode};
+  }
+
+  /*
+  async getTile(tileParams: GetTileParameters): Promise<ArrayBuffer | null> {
+    const {x, y, z} = tileParams;
+    const rangeResponse = await this.pmtiles.getZxy(z, x, y);
+    const arrayBuffer = rangeResponse?.data;
+    if (!arrayBuffer) {
+      // console.error('No arrayBuffer', tileParams);
+      return null;
+    }
+    return arrayBuffer;
+  }
+
+  // Tile Source interface implementation: deck.gl compatible API
+  // TODO - currently only handles image tiles, not vector tiles
+
+  async getTileData(tileParams: GetTileDataParameters): Promise<any> {
+    const {x, y, z} = tileParams.index;
+    const metadata = await this.metadata;
+    switch (metadata.tileMIMEType) {
+      case 'application/vnd.mapbox-vector-tile':
+        return await this.getVectorTile({x, y, z, layers: []});
+      default:
+        return await this.getImageTile({x, y, z, layers: []});
+    }
+  }
+  */
+}
+
+function getDataTypeFromDimension(dimension: Dimension): DataType {
+  const {type, size} = dimension;
+  switch (type) {
+    case 'unsigned':
+      return size === 1 ? 'uint8' : size === 2 ? 'uint16' : size === 4 ? 'uint32' : 'uint64';
+    case 'signed':
+      return size === 1 ? 'int8' : size === 2 ? 'int16' : size === 4 ? 'int32' : 'int64';
+    case 'float':
+      return size === 4 ? 'float32' : 'float64';
+    default:
+      return 'null';
+  }
+}