data_formats

Data formats

• http://www.sdrl.uc.edu/sdrl/referenceinfo/universalfileformats/file-format-storehouse/universal-file-datasets-summary
  • http://www.sdrl.uc.edu/sdrl/referenceinfo/universalfileformats/file-format-storehouse/universal-dataset-number-58
• https://datacite.org/

Standards

DOI

The Digital Object Identifier (DOI) System is for identifying content objects in the digital environment.

Metadata Encoding and Transmission Standard (METS)

The METS schema is a standard for encoding descriptive, administrative, and structural metadata regarding objects within a digital library.

Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH)

OAI-PMH provides a standard way for a data repository to expose metadata to third party indexing engines.

Open Data Protocol (OData)

The OData Protocol is an application-level protocol for interacting with data via RESTful web services.

SKOS Core

The Simple Knowledge Organisation System (SKOS) Core provides a model for expressing the basic structure and content of concept schemes such as thesauri, classification schemes, subject heading lists, taxonomies, folksonomies, and other types of controlled vocabulary.

Standard Archive Format for Europe (SAFE)

SAFE has been designed to act as a common format for archiving and conveying data within European Space Agency (ESA) Earth observation archiving facilities.

XML Formatted Data Unit (XFDU)

XFDU defines the CCSDS Recommended Standard for the packaging of data and metadata, including software, into a single package (e.g., file or message) to facilitate information transfer and archiving.

XMP

Adobe's Extensible Metadata Platform (XMP) is a labeling technology that metadata about a file to be embedded into the file itself.

Metadata Working Group

The Metadata Working Group (MWG) is a consortium of leading companies in the digital media industry, focused on the following goals:

• Preservation and seamless interoperability of digital image metadata
• Interoperability and availability to all applications, devices, and services

Table Schema

• http://specs.frictionlessdata.io/table-schema/
• https://github.com/frictionlessdata/tableschema-py

etc

• This module implements ISO 8601 date, time and duration parsing. https://github.com/gweis/isodate/

• Linear TSV is a line-oriented, portable tabular data format. https://github.com/solidsnack/tsv

• A library for reading and writing tabular data (csv/xls/json/etc). https://github.com/frictionlessdata/tabulator-py

• A Python implementation of RFC 3986 including validation and authority parsing. https://pypi.org/project/rfc3986/

• https://pages.nist.gov/material-schema/Dataset/

• https://github.com/blaze/datashape

• https://fastparquet.readthedocs.io/en/latest/

• cChardet : to detect non-standard unicode encodings

• chardet : to detect non-standard unicode encodings