Skip to content

Commit

Permalink
Release/5.3.0 (#1440)
Browse files Browse the repository at this point in the history
* (un)Staking multiple avoid tx limit (#1244)

* add tx rate limit

* wait for tx limit if not done multi stake/unstake

* dont "decrypt" hotkey

* Grab delegates details from GitHub (#1245)

* add url to init

* add dataclass and util functions

* use in cli

* remove delegates json

---------

Co-authored-by: joeylegere <[email protected]>

* Fix list_delegates on non-archive nodes (#1232)

* Change how pull of archival data is handled

* fix for list_delegates too

* .

* use empty dict

* fix spacing

* specify exception

* log out

* add space in log message

* use warning instead

* Improve development workflow documentation

* Update DEVELOPMENT_WORKFLOW.md

* (un)Staking multiple avoid tx limit (#1244)

* add tx rate limit

* wait for tx limit if not done multi stake/unstake

* dont "decrypt" hotkey

* Grab delegates details from GitHub (#1245)

* add url to init

* add dataclass and util functions

* use in cli

* remove delegates json

---------

Co-authored-by: joeylegere <[email protected]>

* Fix list_delegates on non-archive nodes (#1232)

* Change how pull of archival data is handled

* fix for list_delegates too

* .

* use empty dict

* fix spacing

* specify exception

* log out

* add space in log message

* use warning instead

* Improve development workflow documentation

* Update DEVELOPMENT_WORKFLOW.md

* progress on fix mock subtensor

rename test files for discovery

catch nonce errors

skip long tests

test durations

group cli tests to same runner

try group again

try to group tests using arg

patch start server

dont use groups and wait longer to teardown

dont kill

loadgroup

.

update durations

update durations

delay reruns

progress on new mock

more progress

add back mock init

create axons and prometheus

fix imports

add reset state

fix mock calls

fix setup

move some extr send to subtensor impl

move stake ext call

move unstake ext

add extr outline to mock sub

add ext bodies

fix type ann

return empty map on None

fix global state

* [BIT-351] Ask for wallet name on btcli unstake (#1387)

add not

* fix query map mock

* inc subnetN on mock reg

* add active to neuron on mock reg

* dont store balance in chainstate

* use dict comprehension

* don't print __is_set when displaying config

* use axon info from axon module

* set is set for Config in tests

* passes overview

* remove neurons, take out multiforward Synapse

* take out _neuron, these are in openminers & openvalidators now

* cleanup dendrite multi_forward

* missed MultiForward class in dendrite

* fix tests

* dont skip

* return data free of balance

* return proper types

* no nonces

* use same format for balances

* allow block for is delegate

* fix nominate

* fix nominate again :)

* fix typehint and return value==None

* fix total stake tracking

* nominate passes

* fix state for ismember and totalstake

* not returning last part of tuple

* oops, dont pass self twice

* return null neuron

* get mock block hash

* add set diff mock

* fix total issuance and balance

* add rate limit storage

* fix loop over nom

* fix some unit tests

* fix pow test

* fix get delegates mock

* fix typo

* typo again

* remove assert on err

* typo on storage

* spacing

* BlocksSinceLastStep instead

* fix list subnets

* dont skip

* add is set map

* fix patch

* setup mock net 3

* add a do_prometheus serve func

* add to mock

* remove unused examples

* update bittensor version into to current (510)

* fix prometheus test

* fix some staking tests

* fix transfer tests

* add do_set_weights

* add do_set_weights to mock

* use dsw in tests

* use == true

* use == True

* add do_serve_axon

* use typeddict

* add do serve axon mock

* add _ to subtensor.do_*  methods

* add _subtensor.types

* fix circular import

* fix mock get_balances

* return uid in mock

* run setup for tests with mock sub

* use mock sub for tests

* all subtensor tests pass

* fix mock with underscore

* nominators mock should be list tuple not dict

* fix duplicate setup

* Update README.md

fix typo

* Fix integration test

* remove multiforward from synapse (missed before)

* remove hotkey from proto and dendrite

* Weight Utils fix (#1372)

When Torch sets zeros or ones, it does this on the CPU. The metagraph is loaded on GPU. This results in errors—proposed fix.

* Extract config to new package (#1401)

* integrate openconfig progess

* add type or suppress for argparse

* use for problematic types

* fix type hints

* move test under testcase class

* move tests under testcase class

* add test for btcli with empty arguments on every command

* remove unneeded setup from test

* remove type or suppress from int

* remove to defaults test

* use dot get for prefix check

* move tests to config repo

* fix test by passing argv=[]

* use new package name

* fix name of openconfig package

* Extract wallet (#1403)

* integrate openconfig progess

* add type or suppress for argparse

* use for problematic types

* fix type hints

* move test under testcase class

* move tests under testcase class

* add test for btcli with empty arguments on every command

* remove unneeded setup from test

* remove type or suppress from int

* remove to defaults test

* use dot get for prefix check

* move test to package

* move wallet mock

* move some utils to package

* move wallet to package

* move reregister to registration utils

* add mocks

* move keyfile to package

* move test to utils tests

* use different naming

* fix import

* change function name and use kwargs

* modify naming

* add is_hotkey_registered

* use new function

* fix patch

* arg is cuda (typo)

* specify reregister true

* use generate wallet

* get mock wallet during tests

* extract get ss58 format util

* remove _mock arg

* extract tests

* import from base

* use subtensor for get_balance

* use new hotkey during test

* fix mocking

* add new packages to reqs file

* use new package naming

* bump wallet dep

* BTCli integration with new governance protocol (#1398)

* Begin senate cli

* Add helper functions to subtensor object for query_module, query_module_map

* Remove unused senate helpers

* Rename SenateCommand -> ProposalsCommand

* Remove unused proposals info datatype

* Add proposals data in rich table

* Clarify + beautify calldata column

* Add SenateRegisterCommand

* Add helper function wallet.is_senate_member

* Add subtensor *_senate extrinsics, is_senate_member, get_vote_data impl

* Use helper function to check senate membership

* Add command to view senate members

* Use get_vote_data helper and refactor call data formatting for recursion

* Add senate_vote and senate_leave cmd classes

* Add membership check in senate_register command

* Add senate_leave, senate_vote extrinsic functions

* Import senate_leave, senate_vote extrinsic functions in subtensor_impl

* Add senate, proposal_votes, senate_leave, senate_vote cmds to cli

* Move closure helper funcs to main scope, add display_votes helper

* Add senate size and active proposals metric, vote overview and nice names

* Add delegate nice-name support to proposal_votes

* Use coldkey for senate actions instead of hotkey

* Reverting unnecessary commits for next release. (#1415)

* Reverting unnecessary commits for next release.

* fix missed conflict

---------

Co-authored-by: ifrit98 <[email protected]>

* fix merge conflict

* fix dockerignore

* Use delegate nice-names in senate view

* use new regex for test

* make test assertion more verbose

* add to config for test

* bump version

* create api calls for senate info

* add block optional to is senate member

* prefer using subtensor api

* use new api calls

* dont sync metagraph on neuron init

* remove multiforwad

* fix error calling metagraph from subtensor without passing (#1426)

* fix error calling metagraph from subtensor without passing

* Update bittensor/_subtensor/subtensor_impl.py

Co-authored-by: Cameron Fairchild <[email protected]>

---------

Co-authored-by: Cameron Fairchild <[email protected]>

* Fix docker and pypi upload commands (#1375)

Co-authored-by: philanthrope <[email protected]>

* Fix cli ask for wallet hotkey name (#1430)

* check if NOT hotkey set

* add test for stake and unstake

* remove unneeded import

* add tests for delegate and undelegate

* add tests for delegate ss58 arg

* always sync metagraph during subtensor.metagraph

* derp

* remove unused HelpCommand class for miners/validators

* update changelog from running add_notes_changelog.sh

* fix script for the changelog

* add changelog

* Update scripts/release/github_utils.sh

* add weights call to subtensor (#1436)

* add weights call to subtensor

* add bonds call

* use bonds and weights calls for sync

* move into neurons

* fix type hints

* move func to chain data

* use new funcs in meta init

* fix kwarg name

* Revert "use new funcs in meta init"

This reverts commit 955e671.

* update dockerignore file (#1428)

* update dockerignore file

* add newline

* fix function name in script

* add new changelog additions

* update changelog again

* make minor release

* remove wallet impl

* use reregister from utils

* add arg for base miner reregister

* fix mock return values

* fix return format to use Tuple

* raise -> return

* Revert "fix return format to use Tuple"

This reverts commit 0252fcc.

* Revert "fix mock return values"

This reverts commit 2df31eb.

* raise correct errors in mock

* return tuples for do server

---------

Co-authored-by: joeylegere <[email protected]>
Co-authored-by: “quac88” <“[email protected]”>
Co-authored-by: philanthrope <[email protected]>
Co-authored-by: ifrit98 <[email protected]>
Co-authored-by: Mostima <[email protected]>
Co-authored-by: Julius ter Pelkwijk <[email protected]>
Co-authored-by: Ayden Brewer <[email protected]>
Co-authored-by: Eduardo García <[email protected]>
  • Loading branch information
9 people authored Jul 4, 2023
1 parent 761a327 commit f60c2ae
Show file tree
Hide file tree
Showing 90 changed files with 4,605 additions and 7,041 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ jobs:
command: |
. env/bin/activate
export PYTHONUNBUFFERED=1
pytest -n2 --reruns 3 --durations=0 --verbose --junitxml=test-results/integration_tests.xml \
pytest -n2 --reruns 3 --reruns-delay 15 --durations=0 --verbose --junitxml=test-results/integration_tests.xml \
--cov=. --cov-append --cov-config .coveragerc \
--splits $CIRCLE_NODE_TOTAL --group $((CIRCLE_NODE_INDEX + 1)) \
--splitting-algorithm duration_based_chunks --store-durations --durations-path .test_durations \
Expand Down
19 changes: 13 additions & 6 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,14 +1,21 @@
**/data/
*.log
*.png
*.pstats
**/*.log
**/*.png
**/*.pstats
**/*.ipynb
**/bittensor.egg-info/*
**/lib/*
**/build/*
**/dist/*
**/runs/*
**/env/*
**/venv/*
./circleci/*
./github/*
.ipynb
**/tmp/*
**/test_results/*
**/__pycache__/*
**/.circleci
**/.git
**/.github
**/.hypothesis
**/.vscode
**/.gitignore
147 changes: 97 additions & 50 deletions .test_durations

Large diffs are not rendered by default.

24 changes: 24 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,29 @@
# Changelog

## 5.3.0 / 2023-07-04

## What's Changed
* [BIT-351] Ask for wallet name on btcli unstake by @camfairchild in https://github.com/opentensor/bittensor/pull/1387
* Fix tests using pure-Python MockSubtensor by @camfairchild in https://github.com/opentensor/bittensor/pull/1349
* Update README.md by @mostimasblunderbuss in https://github.com/opentensor/bittensor/pull/1397
* Update authint version by @ifrit98 in https://github.com/opentensor/bittensor/pull/1395
* Fix subtensor factory integration test by @camfairchild in https://github.com/opentensor/bittensor/pull/1400
* Remove neurons by @ifrit98 in https://github.com/opentensor/bittensor/pull/1389
* Merge pull request #1394 from opentensor/fix_axon_requests by @ifrit98 in https://github.com/opentensor/bittensor/pull/1406
* remove hotkey from proto and dendrite by @ifrit98 in https://github.com/opentensor/bittensor/pull/1407
* Weight Utils fix by @mrseeker in https://github.com/opentensor/bittensor/pull/1372
* Extract config to new package by @camfairchild in https://github.com/opentensor/bittensor/pull/1401
* Extract wallet by @camfairchild in https://github.com/opentensor/bittensor/pull/1403
* BTCli integration with new governance protocol by @Rubberbandits in https://github.com/opentensor/bittensor/pull/1398
* Reverting unnecessary commits for next release. by @camfairchild in https://github.com/opentensor/bittensor/pull/1415
* Extract wallet and config by @camfairchild in https://github.com/opentensor/bittensor/pull/1411

## New Contributors
* @mostimasblunderbuss made their first contribution in https://github.com/opentensor/bittensor/pull/1397

**Full Changelog**: https://github.com/opentensor/bittensor/compare/v5.2.0...v5.3.0


## 5.2.0 / 2023-06-28

## What's Changed
Expand Down
133 changes: 68 additions & 65 deletions DEVELOPMENT_WORKFLOW.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,97 +2,102 @@

## Table of contents

1. [Main branches](#main-branches)
1. [Development model](#development-model)
1. [Supporting branches](#supporting-branches)
1. [Feature branches](#feature-branches)
1. [Release branches](#release-branches)
1. [Hotfix branches](#hotfix-branches)
1. [Git operations](#git-operations)
1. [Create a feature branch](#create-a-feature-branch)
1. [Merge feature branch into nobunaga](#merge-feature-branch-into-nobunaga)
1. [Create release branch](#create-release-branch)
1. [Finish a release branch](#finish-a-release-branch)
1. [Create a hotfix branch](#create-a-hotfix-branch)
1. [Finishing a hotfix branch](#finishing-a-hotfix-branch)
- [Development Workflow](#development-workflow)
- [Table of contents](#table-of-contents)
- [Main branches](#main-branches)
- [Development model](#development-model)
- [Feature branches](#feature-branches)
- [Release branches](#release-branches)
- [Hotfix branches](#hotfix-branches)
- [Git operations](#git-operations)
- [Create a feature branch](#create-a-feature-branch)
- [Merge feature branch into staging](#merge-feature-branch-into-staging)
- [Create release branch](#create-release-branch)
- [Finish a release branch](#finish-a-release-branch)
- [Create the hotfix branch](#create-the-hotfix-branch)
- [Finishing a hotfix branch](#finishing-a-hotfix-branch)
- [TODO](#todo)

## Main branches

The repo holds two main branches with an infinite lifetime:
- master
- nobunaga
Bittensor is composed of TWO main branches, **master** and **staging**

We consider `origin/master` to be the main branch where the source code of HEAD always reflects a **__production-ready__** state.
**master**
- master Bittensor's live production branch. This branch should only be touched and merged into by the core develpment team. This branch is protected, but you should make no attempt to push or merge into it reguardless.

We consider `origin/nobunaga` to be the main branch where the source code of HEAD always reflects a state with the **__latest delivered development__** changes for the next release. Some would call this the `"integration branch"`. This is where any automatic nightly builds would be built from.
**staging**
- staging is Bittensor's development branch. This branch is being continuously updated and merged into. This is the branch where you will propose and merge changes.

## Development model

### Supporting branches

Each of these branches have a specific purpose and are bound to strict rules as to which branches may be their originating branch and which branches must be their merge targets. We will walk through them in a minute

#### Feature branches

- May branch off from: `nobunaga`
- Must merge back into: `nobunaga`
- May branch off from: `staging`
- Must merge back into: `staging`
- Branch naming convention:
- Anything except master, nobunaga, finney, release/* or hotfix/*
- Anything except master, staging, finney, release/* or hotfix/*
- Suggested: `feature/<ticket>/<descriptive-sentence>`

When implementing new features, hotfixes, bugfixes, or upgrades, it is wise to adhere to a strict naming and merging convention, whenever possible.

**Branch naming and merging convention:**


Feature branches are used to develop new features for the upcoming or a distant future release. When starting development of a feature, the target release in which this feature will be incorporated may well be unknown at that point.

The essence of a feature branch is that it exists as long as the feature is in development, but will eventually be merged back into `nobunaga` (to definitely add the new feature to the upcoming release) or discarded (in case of a disappointing experiment).
The essence of a feature branch is that it exists as long as the feature is in development, but will eventually be merged into `staging` (to definitely add the new feature to the upcoming release) or discarded (in case of a disappointing experiment).

Generally, you should try to minimize the lifespan of feature branches. As soon as you merge a feature into 'staging', you should immidiately delete the feature branch. This will be strictly enforced. Excess branches creates tech debt and confusion between development teams and parties.

#### Release branches

- May branch off from: `nobunaga`
- Must merge back into: `nobunaga` and `master`
- Please branch off from: `staging`
- Please merge back into: `staging` then into: `master`
- Branch naming convention:
- Suggested format `release/3.4.0/optional-descriptive-message`
- STRONGLY suggested format `release/5.1.0/descriptive-message/creator's-name`

Release branches support preparation of a new production release. Furthermore, they allow for minor bug fixes and preparing meta-data for a release (e.g.: version number, configuration, etc.). By doing all of this work on a release branch, the `nobunaga` branch is cleared to receive features for the next big release.
Release branches support preparation of a new production release. Furthermore, they allow for minor bug fixes and preparing meta-data for a release (e.g.: version number, configuration, etc.). By doing all of this work on a release branch, the `staging` branch is cleared to receive features for the next big release.

This new branch may exist there for a while, until the release may be rolled out definitely. During that time, bug fixes may be applied in this branch, rather than on the `nobunaga` branch. Adding large new features here is strictly prohibited. They must be merged into `nobunaga`, and therefore, wait for the next big release.
This new branch may exist there for a while, until the release may be rolled out definitely. During that time, bug fixes may be applied in this branch, rather than on the `staging` branch. Adding large new features here is strictly prohibited. They must be merged into `staging`, and therefore, wait for the next big release.

#### Hotfix branches

- May branch off from: `master`
- Must merge back into: `nobunaga` and `master`
- Please branch off from: `master` or `staging`
- Please merge back into: `staging` then into: `master`
- Branch naming convention:
- Suggested format: `hotfix/3.3.4/optional-descriptive-message`
- REQUIRED format: `hotfix/3.3.4/descriptive-message/creator's-name`

Hotfix branches are very much like release branches in that they are also meant to prepare for a new production release, albeit unplanned. They arise from the necessity to act immediately upon an undesired state of a live production version. When a critical bug in a production version must be resolved immediately, a hotfix branch may be branched off from the corresponding tag on the master branch that marks the production version.

The essence is that work of team members, on the `nobunaga` branch, can continue, while another person is preparing a quick production fix.
The essence is that work of team members, on the `staging` branch, can continue, while another person is preparing a quick production fix.

### Git operations

#### Create a feature branch

1. Branch from the **nobunaga** branch.
1. Command: `git checkout -b feature/my-feature nobunaga`
1. Branch from the **staging** branch.
1. Command: `git checkout -b feature/my-feature staging`

> Try to rebase frequently with the updated nobunaga branch so you do not face big conflicts before submitting your pull request. Remember, syncing your changes with other developers could also help you avoid big conflicts.
> Rebase frequently with the updated staging branch so you do not face big conflicts before submitting your pull request. Remember, syncing your changes with other developers could also help you avoid big conflicts.
#### Merge feature branch into nobunaga
#### Merge feature branch into staging

In other words, integrate your changes into a branch that will be tested and prepared for release.

- Switch branch to nobunaga: `git checkout nobunaga`
- Merging feature branch into nobunaga: `git merge --no-ff feature/my-feature`
- Pushing changes to nobunaga: `git push origin nobunaga`
- Delete feature branch: `git branch -d feature/my-feature`
- Switch branch to staging: `git checkout staging`
- Merging feature branch into staging: `git merge --no-ff feature/my-feature`
- Pushing changes to staging: `git push origin staging`
- Delete feature branch: `git branch -d feature/my-feature` (alternatively, this can be navigated on the GitHub web UI)

This operation is done by Github when merging a PR.

So, what you have to keep in mind is:
- Open the PR against the `nobunaga` branch.
- After merging a PR you just have to delete your feature branch.
- Open the PR against the `staging` branch.
- After merging a PR you should delete your feature branch. This will be strictly enforced.

#### Create release branch

- Create branch from nobunaga: `git checkout -b release/3.4.0/optional-descriptive-message nobunaga`
- Create branch from staging: `git checkout -b release/3.4.0/descriptive-message/creator's_name staging`
- Updating version with major or minor: `./scripts/update_version.sh major|minor`
- Commit file changes with new version: `git commit -a -m "Updated version to 3.4.0"`

Expand All @@ -106,20 +111,20 @@ In other words, releasing stable code and generating a new version for bittensor
- Pushing changes to master: `git push origin master`
- Pushing tags to origin: `git push origin --tags`

To keep the changes made in the __release__ branch, we need to merge those back into `nobunaga`:
To keep the changes made in the __release__ branch, we need to merge those back into `staging`:

- Switch branch to nobunaga: `git checkout nobunaga`.
- Merging release branch into nobunaga: `git merge --no-ff release/3.4.0/optional-descriptive-message`
- Switch branch to staging: `git checkout staging`.
- Merging release branch into staging: `git merge --no-ff release/3.4.0/optional-descriptive-message`

This step may well lead to a merge conflict (probably even, since we have changed the version number). If so, fix it and commit.

After this the release branch may be removed, since we don’t need it anymore:

- `git branch -d release/3.4.0/optional-descriptive-message`
- `git branch -d release/3.4.0/descriptive-message/creator's-name`

#### Create the hotfix branch

- Create branch from master:`git checkout -b hotfix/3.3.4/optional-descriptive-message master`
- Create branch from master:`git checkout -b hotfix/3.3.4/descriptive-message/creator's-name master`
- Update patch version: `./scripts/update_version.sh patch`
- Commit file changes with new version: `git commit -a -m "Updated version to 3.3.4"`

Expand All @@ -128,37 +133,35 @@ Then, fix the bug and commit the fix in one or more separate commits:

#### Finishing a hotfix branch

When finished, the bugfix needs to be merged back into `master`, but also needs to be merged back into `nobunaga`, in order to safeguard that the bugfix is included in the next release as well. This is completely similar to how release branches are finished.
When finished, the bugfix needs to be merged back into `master`, but also needs to be merged back into `staging`, in order to safeguard that the bugfix is included in the next release as well. This is completely similar to how release branches are finished.

First, update master and tag the release.

- Switch branch to master: `git checkout master`
- Merge changes into master: `git merge --no-ff hotfix/3.3.4/optional-descriptive-message`
- Tag new version: `git tag -a v3.3.4 -m "Releasing v3.3.4: some comment about the hotfix"`
- Tag new version: `git tag -a v3.3.4 -m "Releasing v3.3.4: descriptive comment about the hotfix"`
- Pushing changes to master: `git push origin master`
- Pushing tags to origin: `git push origin --tags`

Next, include the bugfix in `nobunaga`, too:
Next, include the bugfix in `staging`, too:

- Switch branch to nobunaga: `git checkout nobunaga`
- Merge changes into nobunaga: `git merge --no-ff hotfix/3.3.4/optional-descriptive-message`
- Pushing changes to origin/nobunaga: `git push origin nobunaga`
- Switch branch to staging: `git checkout staging`
- Merge changes into staging: `git merge --no-ff hotfix/3.3.4/descriptive-message/creator's-name`
- Pushing changes to origin/staging: `git push origin staging`

The one exception to the rule here is that, **when a release branch currently exists, the hotfix changes need to be merged into that release branch, instead of** `nobunaga`. Back-merging the bugfix into the __release__ branch will eventually result in the bugfix being merged into `develop` too, when the release branch is finished. (If work in develop immediately requires this bugfix and cannot wait for the release branch to be finished, you may safely merge the bugfix into develop now already as well.)
The one exception to the rule here is that, **when a release branch currently exists, the hotfix changes need to be merged into that release branch, instead of** `staging`. Back-merging the bugfix into the __release__ branch will eventually result in the bugfix being merged into `develop` too, when the release branch is finished. (If work in develop immediately requires this bugfix and cannot wait for the release branch to be finished, you may safely merge the bugfix into develop now already as well.)

Finally, we remove the temporary branch:

- `git branch -d hotfix/3.3.4/optional-descriptive-message`

## TODO

- Changing the name of the develop branch from nobunaga to `integration`
- Because sometimes nobunaga are going to have a release branch.
- Knowing if master and nobunaga are different
- Knowing what is in nobunaga that is not merge yet
- Knowing if master and staging are different
- Knowing what is in staging that is not merge yet
- Document with not released developments
- When merged into nobunaga, generate the information exposing what's merged into nobunaga but not release.
- When merged into staging, generate the information exposing what's merged into staging but not release.
- When merged into master, generate github release and release notes.
- CircleCI job
- Merge nobunaga into master and release version (needed to release code)
- Build and Test bittensor (needed to merge PRs)
- Merge staging into master and release version (needed to release code)
- Build and Test Bittensor (needed to merge PRs)
Loading

0 comments on commit f60c2ae

Please sign in to comment.