Skip to content

GitHub archives, git, GitHub API -> Postgres databases -> Grafana dashboards

License

Notifications You must be signed in to change notification settings

onerepo/devstats

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Build Status CII Best Practices

GitHub archives and git Grafana visualization dashboards

Author: Łukasz Gryglicki [email protected]

This is a toolset to visualize GitHub archives using Grafana dashboards.

GHA2DB stands for GitHub Archives to DashBoards.

More information about Kubernetes dashboards here.

Presentations

  • Presentations are available here.
  • Direct link.

Deploying on your own project(s)

See the simple DevStats example repository for single project deployment (Homebrew), follow instructions to deploy for your own project.

Goal

We want to create a toolset for visualizing various metrics for the Kubernetes community (and also for all CNCF projects).

Everything is open source so that it can be used by other CNCF and non-CNCF open source projects.

The only requirement is that project must be hosted on a public GitHub repository/repositories.

Data hiding

If you want to hide your data (replace with anon-#) please follow instructions here.

Forking and installing locally

This toolset uses only Open Source tools: GitHub archives, GitHub API, git, Postgres databases and multiple Grafana instances. It is written in Go, and can be forked and installed by anyone.

Contributions and PRs are welcome. If you see a bug or want to add a new metric please create an issue and/or PR.

To work on this project locally please fork the original repository, and:

Please see Development for local development guide.

For more detailed description of all environment variables, tools, switches etc, please see Usage.

Metrics

We want to support all kind of metrics, including historical ones. Please see requested metrics to see what kind of metrics are needed. Many of them cannot be computed based on the data sources currently used.

Repository groups

There are some groups of repositories that are grouped together as a repository groups. They are defined in scripts/kubernetes/repo_groups.sql.

To setup default repository groups:

  • PG_PASS=pwd ./kubernetes/setup_repo_groups.sh.

This is a part of kubernetes/psql.sh script and kubernetes psql dump already has groups configured.

In an All CNCF project repository groups are mapped to individual CNCF projects scripts/all/repo_groups.sql:

Company Affiliations

We also want to have per company statistics. To implement such metrics we need a mapping of developers and their employers.

There is a project that attempts to create such mapping cncf/gitdm.

DevStats has an import tool that fetches company affiliations from cncf/gitdm and allows to create per company metrics/statistics.

If you see errors in the company affiliations, please open a pull request on cncf/gitdm and the updates will be reflected on https://k8s.devstats.cncf.io a couple days after the PR has been accepted. Note that gitdm supports mapping based on dates, to account for developers moving between companies.

Architecture

For architecture details please see architecture file.

Detailed usage is here

Adding new metrics

Please see metrics to see how to add new metrics.

Adding new projects

To add new project follow adding new project instructions.

Grafana dashboards

Please see dashboards to see list of already defined Grafana dashboards.

Exporting data

Please see exporting.

Detailed Usage instructions

Servers

The servers to run devstats are generously provided by Packet bare metal hosting as part of CNCF's Community Infrastructure Lab.

One line run all projects

  • Use GHA2DB_PROJECTS_OVERRIDE="+cncf" PG_PASS=pwd devstats.
  • Or add this command using crontab -e to run every hour HH:08.

Checking projects activity

  • Use: PG_PASS=... PG_DB=allprj ./devel/activity.sh '1 month,,' > all.txt.
  • Example results here - all CNCF project activity during January 2018, excluding bots.

About

GitHub archives, git, GitHub API -> Postgres databases -> Grafana dashboards

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 54.5%
  • Shell 28.8%
  • SQLPL 8.6%
  • HTML 6.0%
  • Ruby 1.4%
  • Makefile 0.7%