TSN Adapters

This repository contains examples of using the TSN SDK to ingest data from various sources into TSN. It leverages truflation/truflation connectors to get data from sources, powered by Prefect features such as resiliency, concurrency, caching, observability, UI controls and more.

Reusable Tasks

We have a few reusable tasks that can be used to ingest data from various sources into TSN. They can serve as an example of using the TSN SDK to achieve certain tasks, or may also be reused in your flows (you may want to install this package to use them).

Non-exhaustive list:

• Read a CSV file from a GitHub repository
• Read a Google Sheet
• Insert records into TSN
• Get all records from TSN
• Deploy a primitive
• Reconcile data

Examples

Each example is a Prefect flow that can be run locally or in a remote environment.

Simple GSheets

We have two versions of the flow:

1. Direct - In this version, we directly specify the sheet ID and the source ID to filter by.
2. Dynamic - In this version, we fetch the sheet ID and the source ID from a CSV file in a GitHub repository.

Local Development

This repository includes a docker-compose.yaml file to run a local development environment with a Postgres database and a Prefect server.

Requirements:

• Docker
• TSN Node to be targeted (might be local or remote)

To run the development environment, run the following command:

docker compose up -d

This will start the services and keep them running in the background.

Deploying Prefect or using its Cloud infrastructure is extensively documented here.

You may try the examples just by running their python files, after installing the package in a virtual environment. Make sure to copy .env.example to .env and set the correct environment variables.

Python Environment

We recommend using a virtual environment to manage dependencies. You can create one using the following commands:

python -m venv .venv
source .venv/bin/activate
pip install -e .