A Permanent IOTA Message Storage Solution

About ◈ Prerequisites ◈ API Reference ◈ Config Reference ◈ Supporting the project ◈ Joining the discussion ◈ Future work ◈ LICENSE

---

About

Chronicle provides tools for managing and accessing permanode solutions using the IOTA actor framework backstage. With Chronicle, you can:

• Store IOTA messages in real time, using one or more Scylla clusters
• Explore stored messages using an HTTP API
• Store the data you want by modifying incoming messages
• Filter data to store it how and where you want (work in progress)

Chronicle includes the following crates:

• Chronicle The entry point for the Chronicle application
• API: API that allows you to access stored messages
• Broker: Allows subscribing to incoming messages from IOTA nodes
• Storage: Implements storage related functionality from scylla.rs
• Filter: A simple container for user modified code

Note: This is alpha software, so there may be performance and stability issues. Please report any issues in our issue tracker.

Prerequisites

To run Chronicle, you need the following:

• A Linux LTS operating system such as Ubuntu

• 4 GB RAM

• At least 32 GB of disk space

• 64-bit processor

• Preferred a 10 Gbps network connection

• At least 2 CPU cores (recommended)

• Rust

• At least one Scylla node (version 4 or greater) running on a different device in the same private network as Chronicle. See the Scylla documentation for a tutorial on setting one up. For information about securing your Scylla nodes, see the Scylla security documentation.

• The build-essentials packages

  You can install these packages for Debian based distros, using the following command:

  sudo apt-get install build-essential gcc make cmake cmake-gui cmake-curses-gui pkg-config openssl libssl-dev

  For other Linux distros, please refer to your package manager to install the build-essential pkgs

• (Optional) An IDE that supports Rust autocompletion. We recommend Visual Studio Code with the rust-analyzer extension

• If you want to load historical transactions into your permanode, you can download the files from the IOTA Foundation's archive.

We also recommend updating Rust to the latest stable version:

rustup update stable

Installation

Either download the provided executable (you should only do this if you do not wish to use the filtering functionality), or build it yourself.

Building Chronicle

Clone this repository:

git clone https://github.com/iotaledger/chronicle.rs

cargo build --release

If you wish to use the filter functionality, enable the filter feature in chronicle

cargo build --release --features filter

Configuring Chronicle

Chronicle uses a RON file to store configuration parameters, called config.ron. An example is provided as config.example.ron with default values. See Config Reference for more details about the config file.

API Reference

For an API reference, see the documentation portal.

Config Reference

storage_config

keyspaces: Vec<KeyspaceConfig>

See KeyspaceConfig

Multiple keyspaces can be configured in order to filter incoming messages. If the filter feature is not used, only the first configured keyspace will be considered or the default (chronicle) if none is provided.

In addition to the keyspace name, each requires a map of datacenters (name -> replication factor). See here for more information about datacenters in ScyllaDB.

listen_address: String

The scylla.rs dashboard listen address, where it accepts requests to manage the Scylla cluster.

thread_count: Enum

The number of threads Scylla will use. Can be one of Count(usize) (a simple scalar count) or CoreMultiple(usize) (a multiple of the number of cores the system has).

reporter_count: u8

The number of reporters Scylla will spawn.

local_datacenter: String

The Scylla local datacenter.

partition_config

See PartitionConfig

Specifies the number of partitions to use in the database, as well as the number of milestones to use as chunks.

NOTICE: You can't change partition_config in future without migration.

api_config

Nothing at the moment, please refer to .env.

broker_config

websocket_address: String

The Broker dashboard listen address, where it accepts requests to manage the broker topology.

mqtt_brokers: Vec<Url>

• Messages: mqtt topic used to receive incoming IOTA messages;
• MessagesReferenced: mqtt topic used to receive incoming metadata;

NOTICE: You should at least have one of each.

api_endpoints: Vec<Url>

IOTA node-endpoints used by chronicle to fill gaps.

retries_per_endpoint: u8

Max number of retries to retrieve something from api_endpoints.

retries_per_query: usize

Max number of retries to fetch/insert something from/toscylla, before declaring an outage, which will force the broker application to pause and await for scylla cluster to recover.

collector_count: u8

The number of concurrent collectors which collect data from feed sources also it's used as solidifier_count.

requester_count: u8

The number of concurrent requesters per collector.

NOTE: requesters are used by collector to fetch missing data from api_endpoint

request_timeout_secs: u64

The api_endpoint request timeout.

parallelism: u8

The max number of concurrent solidify requests.

sync_range: Option<SyncRange>

Identiy the milestone data sync range from/to.

complete_gaps_interval_secs: u64

Interval used by syncer to check if there are some gaps to fill/complete.

logs_dir: Option<String>

If provided, it will archive the milestone data in ordered fashion.

max_log_size: Option<u64>

The upper limit of the log_file_size.

NOTE: Ensure to use a limit within your filesystem range.

Running Chronicle

See Building Chronicle.

cd target/release && cp /path/to/your/config.ron ./

cargo run --release

Supporting the project

If you want to contribute to Chronicle, consider posting a bug report, feature request or a pull request.

Please read the following before contributing:

• Contributing guidelines

Joining the discussion

If you want to get involved in the community, need help with getting set up, have any issues related to Chronicle, or just want to discuss IOTA, Distributed Registry Technology (DRT) and IoT with other people, feel free to join our Discord.

Future work

• Add more examples and documentation
• Add more test cases
• Implement selective permanode
• Add dashboard web app

LICENSE

(c) 2021 - IOTA Stiftung

IOTA Chronicle is distributed under the Apache License (Version 2.0).