Pull-Requests KPIs

[Amzani]

Problem

At AsyncAPI, we prioritize the prompt acknowledgment of new contributors who submit pull requests, offering timely code reviews, and ensuring efficient code merging. We view this quick turnaround time for reviews as a vital indicator on how successful we are at engaging our community of contributors.

Solution

To solve the problem we first need to get visibility around our key engineering metrics that are split into 3 main categories:

1. Pull-Requests KPIs
2. Release KPIs
3. Issues Interaction KPIs

In this Bet we will focus on Pull-requests KPIs

Pull request flow

Metrics we want to track

We would like to track theses metrics in the repos we monitor

Feature lead time

merge (date) - first commit (date)

PR lead time

merged (date) – opened at (date)

Time to first review

first review created at (date)  - last commit created at (date)

Review time

last review created at (date)  – first review created at (date)

Last review to merge time

merged (date) – last review created at (date) 

List of repositories we should monitor

• asyncapi/asyncapi-react
• asyncapi/bindings
• asyncapi/cli
• asyncapi/glee
• asyncapi/modelina
• asyncapi/parser-api
• asyncapi/parser-js
• asyncapi/spec
• asyncapi/spec-json-schemas
• asyncapi/studio

Solution design

1. Store the metrics in a database // or analytic service (preference for solutions that supports open source)
2. Users must see the metrics in a dashboard to track them over time
3. The system must offer alerting capability (slack integration)

Rabbit holes

Not known for now

Scope

• Define the technical solution // POC
• Dashboard
• Alert system

Out of bounds

The alert system should not be hard coded, it must offer the team the possibility to define the KPIs as part of the working agreement (PRs should have first time review < 24h)

Success criteria

We are able to get visibility on the metrics and reduce the review time.

[github-actions]

Welcome to AsyncAPI. Thanks a lot for reporting your first issue. Please check out our contributors guide and the instructions about a basic recommended setup useful for opening a pull request.
Keep in mind there are also other channels you can use to interact with AsyncAPI community. For more details check out this issue.

[KhudaDad414]

Are we going to collect the metrics in each stage or after a PR is merged? I guess it makes sense to collect it on PR merge since it takes less juice out of our precious runtime of GitHub actions.
But collecting it in different stages would give us fresh data. 🤔

[derberg]

if this is Postman only initiative and related to multiple repos, why do we have it here in spec repo?

My 2 cents:

• I suggest you first focus on one repo, to decrease complexity. You will also see how complex issue you want to tackle, and then you can multiply it by N repos you want to track
• don't try to reinvent but use ready solutions like https://ossinsight.io/analyze/asyncapi/asyncapi-react#issues

reduce the review time

metrics will not help reducing review time. Might be better to just have a dedicated channel in Slack for you folks, where you will be getting notifications (something like we have now with channel for new issues and PRs) about your repos, and only info about "new issues or PRs that do not fulfil KPI" - your team would have to actively react, and improvement-of-KPI for you would be basically less alerts to the channel. This is better solution as you do not need a database, you do not need to handle github events storage complexity and any dashboards. You would just have a GH workflow that runs in a repo, checks time and drops a message to slack.

[Amzani]

@derberg you're right. I'll move this issue to asyncapi/studio as a first repo we start with.
Already explored ossinsight it's not enough.

[Amzani]

Moved to asyncapi/studio#802