Modern data pipeline for exploring github data

Explore analytics data for a github repo using a best-practice modern data pipeline:

1. Meltano/singer to Extract raw data from the github API
2. Meltano/postgres to Load raw github data into a postgres database
3. dbt to Transform raw github data into explorable tables
4. Lightdash to visualise and explore the transformed tables

Requirements

• docker

Usage

1. Configure your credentials

export TAP_GITHUB_ACCESS_TOKEN=<your personal access token for github>
export TAP_GITHUB_REPOSITORY="<your github repo e.g. lightdash/lightdash>"
export TAP_GITHUB_START_DATE="<start date e.g. 2020-10-01T00:00:00Z>"

2. Extract github data, load it to postgres, transform it

docker compose run meltano

3. Visualise and explore data

docker compose up lightdash

Open http://localhost:8080