Ubiquity DAO
Replies: 12 comments 8 replies
-
Direct Incentives for Reviewers |
Beta Was this translation helpful? Give feedback.
-
|
More thought is needed for the following, but I realized that as we grow our review team, we start to bog down the merge speed by finding all sorts of tiny issues. I think we should strive to change our culture to turn review tasks into new bounties IF the current version achieves the spec minimally. Code quality is subjective here but it will at least "solve the fix" or "make the new feature" faster by allowing the bounty hunter to focus specifically on what was specified for the associated issue, minimizing scope creep and letting us merge faster. The direct incentive for reviewers to break off tasks into new bounties could be that the comment incentives on the pull request are small, but the issue creator incentive is large. They can effectively convert their comment into a new issue/task, which is what I do on a daily basis; hence why almost all of the issues I create reference a previous comment elsewhere. Also code cleanup/refactoring/details could be handled by a reviewer to just jump in and clean up before merging. We should figure out a direct incentive for this behavior. This could lead to apparent speed gains. |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, yeah right, its the urgent issue for now. Tbh, I can't say But can direct incentives for pull request reviews will solve this problem? What we really need to do is to reduce the duration between PR created time and merged time. My opinion is: Then, if something goes wrong about the merged PR, the reviewers should work on hotfixes about it. |
Beta Was this translation helpful? Give feedback.
-
Like I alluded to above, if the incentive comes in the form of creating a new issue instead of directly from reviewer comments then only the "good" or approved changes will receive issue creator credit (priority and time are associated with them, thereby funding the tasks) It will also abstract the new task away from the existing pull request which will help with closing out the existing pull request faster.
I'm actively looking to remove the dependence on me on GitHub because soon I need to be focusing my attention on Series A fundraising and setting up our fiat banking infrastructure.
I've seen some GitHub repository setting/feature which "round robin" requests reviewers to load balance the work which seems interesting to explore. I think this might make more sense when we have a larger team (hopefully soon) because right now I feel that I have a pretty good idea of who is best to review what type of work.
We can make an incentive for the reviewers to bet on the "correct side" this requires more thought but for example, approving pull requests that meet the spec and receiving a delayed payout to verify that the pull request isn't reverted. Requesting changes can add a small direct (paid on merge) financial bonus possibly. |
Beta Was this translation helpful? Give feedback.
-
|
The main concern would be to determine how much we should directly pay to reviewers. we can set it up for a small percentage of the original amount. In this case, we need 2 percent values at least. for both code reviewers and QA. ex; code review: 10%, qa: 5% Once we roll up the incentive feature, we can engage the reviewers. but should consider we don't pay much for a lot of conversation history. |
Beta Was this translation helpful? Give feedback.
-
Same idea different expression. It's what we are already doing but is not limited to what is implement so far.
If incentives are capped to bounty price, we could do comment incentives and then a bigger incentive for approve/request changes (a conclusion to the review) e.g. $10-$25 For a $100 bounty, only a few rounds of reviews per reviewer would be incentivized. At that point the bot can automatically close the pull request with a comment saying that the task was failed because no more reviewer incentives can be provided. However if the reviewers are optimistic on the pull request, they can simply reopen it and complete the review process. It can just check every reviewer requested. Reviewers can also technically dismiss/delegate (which makes sense in a team that knows what type of code each other team member specializes in) Once all reviewers run out for incentives it can then do the automatic close. |
Beta Was this translation helpful? Give feedback.
-
Direct Incentives for Bounty Hunters |
Beta Was this translation helpful? Give feedback.
-
|
@pavlovcik I would suggest automations:
https://github.com/marketplace?type=&verification=&query=Pull+request
https://github.com/sturdy-dev/codereview.gpt https://www.cncf.io/blog/2023/06/06/a-chatgpt-powered-code-reviewer-bot-for-open-source-projects/ |
Beta Was this translation helpful? Give feedback.
-
|
I think we get the initial review quite fast but then it kinda slows down. I think it's also because we have a lot of new contributors that don't know how to do QA, don't know the project and its codebase. Because of the async nature and bounty hunters leaving and joining the project, we can't ship new features/bugs as fast as most of the startups that have a strong and stable team. I think what can really improve our velocity is having really good tests, end-to-end including unit tests covering all the major use cases and as bugs arise they can be added to the tests. This way bounty hunters can be more confident in their work as they can test their code and see if anything breaks meaning they can fix it before we even review it, so it saves time for unnecessary reviews and QA. After the bounty hunter submits the PR, the tests will run in CI and if they succeed, reviewers can be sure that other features are working and focus on reviewing the code quality and QA can focus on only testing the feature/bug that PR is solving and not other features. It does however increase the burden on the bounty hunter because they have to additionally write tests, but I think it's still better in the end. |
Beta Was this translation helpful? Give feedback.
-
High performing tech startups have dedicated teams while we rely on outside collaborators without deep project understanding. Overall I agree with whilefoo that "unit testing" culture (when unit testing is a reqiement for any feature) should help because this way there are less chances to break something (for a bounty hunter who sees the project for the 1st time). This "unit testing" culture also eliminates technical debt and in the end saves money for payouts (should we have set unit testing initially we wouldn't have had such issues). |
Beta Was this translation helpful? Give feedback.
-
|
If unit tests are considered as a suitable solution to increase the PR review velocity, it might me helpful to research automation for unit tests as well. It would be interesting to know real experience with tools that are positioned as "Automated unit test writing, Auto generated tests". The examples
|
Beta Was this translation helpful? Give feedback.
-
|
From the conversations, all of us seem to agree that both unit tests and end-to-end tests can speed up the PR review. |
Beta Was this translation helpful? Give feedback.
-
|
I believe high-quality automated tests will act as the first and strongest line of defense against bugs. This will improve velocity for sure but it will also help hunters, reviewers, and users sleep in peace at night, again indirectly increasing velocity. The nature of ubiquibot is that it is a little extra painful to QA. These are no buttons like a website where you fill username and passed and click login and see if it works or not.
Code quality guidelines can be created by discussion in the team. Placed in readme, these guidelines will save everyone's time. Plus, soon we will hear about not just code quality, but also test quality.
Most hunters will come back and make the same mistakes again. Free up the team's time to review more. Total amount of work contains more testing than actual coding. Usually, I see testing is already being moved to team. Many hunters don't do QA, some provide absolutely useless QA. And now team also will have to do their clean up. First time hunters will have to find hard time in review to deliver good quality in second try. And as they provide good quality in second try, velocity will go up. For this, I have some other ideas but they will probably become obsolete when automated tests kick in. Because there wouldn't be a QA anymore. Hunters will write tests with feature/bug fix.
|
Beta Was this translation helpful? Give feedback.
-
In the future we can easily gate off the high priority items based on experience points. This will allow only proven talent to handle the important stuff. This will also expose us to a perpetual drip of new talent to potentially recruit. But only on "low priority" tasks.
Cool idea let's implement
Are you sure they run in CI from the pull request? I assumed they needed to be merged in to run or something
We can simply add to the base rate of the repository to attempt to compensate for this. I also need to do more research into this idea but in theory I think it's possible to have the following flow for software development:
Seems very rigid but leveraging ChatGPT for test writing I think is a missed opportunity currently |
Beta Was this translation helpful? Give feedback.
-
I think we can lean into collaborators who stick around for awhile. We can just maintain the open bounties on low priority items for recruiting purposes.
Can you start implementing this culture during the next couple of weeks? We can have a brief discussion around increasing base rate but my expectation is probably at least a 2x which seems reasonable. |
Beta Was this translation helpful? Give feedback.
-
|
@rndquu following up on this |
Beta Was this translation helpful? Give feedback.
-
|
I will create a "Technical debt" draft issue (to discuss with the team) for the bot's repo with:
|
Beta Was this translation helpful? Give feedback.
-
Although a bit unreliable, I do like the concept of: in the ubiquity dollar repo it checks if the code coverage has decreased, which implies that the newly pushed code does not have associated unit tests. |
Beta Was this translation helpful? Give feedback.
-
My vision is to let the best talent rise to the top (quantitatively) with our leveling system, and churn out the poor contributors. For example, certain tasks of higher priority can only be handled by certain levels of contributors. With the leveling system, it's easy to implement "dedicated teams" as a continuum, not as a boolean. |
Beta Was this translation helpful? Give feedback.
-
|
Given the complexity of our current architecture, with many microservices and plugins, implementing unit testing and end-to-end (E2E) testing would have a very positive impact on productivity. I'm also familiar with CI processes and am keen to contribute significantly to this area. :) |
Beta Was this translation helpful? Give feedback.
-
A major strategic priority for us is to maximize our velocity for merging in pull requests (with high accuracy, meaning that they achieve the specification without bugs). This is because high performing tech startups ship new features and bug fixes lightning fast ⚡️
This is useful to stay competitive (features) and to retain users (keep them happy with fast bug fixes)
I would like to hear everybody's ideas on how we can improve this because right now it feels that average time is about a couple of weeks per pull request (lots of room for improvement)
Some relevant systems that come to mind:
Beta Was this translation helpful? Give feedback.
All reactions