Release/3.3.4 (#927)

* [feature] external axon flags (#887) * add external axon changes * add defaults for new axon flags * fix args to axon * default to internal ip and port if not specified * add new args and todefaults * add axon unit tests * add description for subtensor integration test * move test to unit test * create new test file add/update copyright notices * don't default to internal ip * add tests for setting the full_address * add tests for subtensor.serve w/external axon info * allow external port config to be None * switch to mock instead of patch * fix test mocks * change mock config create * fix/add default config * change asserts add mesage * fix check call args * fix mock config set * only call once * fix help wording * should be True * [fix] fixes unstake with max-stake flag (#905) * add equality to None to the balance class * add tests for the None case * local train bug fix (#906) * [feature] [CUDA solver] Add multi-GPU and ask for CUDA during btcli run (#893) * added cuda solver * boost versions to fix pip error * allow choosing device id * fix solution check to use keccak * adds params for cuda and dev_id to register * list devices by name during selection * add block number logging * fix calculation of hashrate * fix update interval default * add --TPB arg to register * add update_interval flag * switch back to old looping/work structure * change typing * device count is a function * stop early if wallet registered * add update interval and num proc flag * add better number output * optimize multiproc cpu reg keeping proc until solution * fix test * change import to cubit * fix import and default * up default should have default in CLI call * add comments about params * fix config var access * add cubit as extra * handle stale pow differently check registration after failure * restrict number of processes for integration test * fix stale check * use wallet.is_registered instead * attempt to fix test issue * fix my test * oops typo * typo again ugh * remove print out * fix partly reg test * fix if solution None * fix test? * fix patch * add args for cuda to subtensor * add cuda args to reregister call * add to wallet register the cuda args * fix refs and tests * add for val test also * fix tests with rereg * fix patch for tests * add mock_register to subtensor passed instead * move register under the check for isregistered * use patch obj instead * fit patch object * fix prompt * remove unneeded if * modify POW submit to use rolling submit again * add backoff to block get from network * add test for backoff get block * suppress the dev id flag if not set * remove dest so it uses first arg * fix pow submit loop * move registration status with * fix max attempts check * remove status in subtensor.register * add submit status * change to neuron get instead * fix count * try to patch live display * fix patch * . * separate test cases * add POWNotStale and tests * add more test cases for block get with retry * fix return to None * fix arg order * fix indent * add test to verify solution is submitted * fix mock call * patch hex bytes instead * typo :/ * fix print out for unstake * fix indexing into mock call * call indexing * access dict not with dot * fix other indent * add CUDAException for cubit * up cubit version * [Feature] ask cuda during btcli run (#890) * add ask for cuda reg config in btcli run * suppress unset arg * [Feature] [cuda solver] multi gpu (#891) * change diff display out * remove logging * check cubit support in the check config * allow 1 or more devices in flag * cuda flag should be suppress * modify how cpu count is found * make a solver base class * add a solverbase for CUDA * use mutli process kernel launching, one per GPU * move check under dot get accessor * Feature/cuda solver multi gpu (#892) * change diff display out * remove logging * check cubit support in the check config * allow 1 or more devices in flag * cuda flag should be suppress * modify how cpu count is found * make a solver base class * add a solverbase for CUDA * use mutli process kernel launching, one per GPU * move check under dot get accessor * add All gpus specification * continue trying reg after Stale * catch for OSX * dont use qsize * add test for continue after being stale * patch get_nowait instead of qsize * [Docs] Update old docs link to new link. Change discord invite to custom link (#915) * Update old docs link to new one This change deletes the old gitbooks documentation link and replaces it with the new one. * fix discord links Co-authored-by: Mac Thrasher <[email protected]> * Fix for test_neuron.py (#917) prevents downloading from huggingface * [feature] add --seed option to regen_hotkey (#916) * add seed option to regen hotkey * make seed optional and fix docstring * add tests for both coldkey and hotkey regen w/seed * oops, make seed optional * fix old test, add config.seed * circle ci version update and fix (#920) * Add test_phrases_split unit test Asserts that randomly instantiated compact_topk encodings can be correctly decoded to recover the original topk_tensor. * Update unravel_topk_token_phrases with faster implementation Replaces .tensor_split() with block indexing to avoid extra copy operations. * Rename test_phrases_split to test_random_topk_token_phrases * Unit tests cleanup (#922) * circle ci version update and fix * Test clean up * uncomment test and remove specific test * remove loguru and fix flaky tests * fix syncing * removing tokenizer equivalence + some bug fixes * moving old dataset test * Deactivate test_random_topk_token_phrases unit test * Create topk_tensor on origin device * Normalization Update (#909) * local train bug fix * normalization update * fix tests * remove test * updated normalization * Naming changes, bug fixes * subtensor update for max clip * max weight to a million * Fixes for ordering and comments * additional tests * string fix * numerical stability and testing updates * minor update for division by zero * Naming and spacing fixes * epsilon update * small fix * Adding development workflow documentation and script for bumping the version (#918) BIT-582 Adding development workflow documentation and script for bumping the version * Updated version 3.3.4 * Revert "Normalization Update (#909)" This reverts commit 3990a28. * version update * memory fix to release (#932) . * use force on set_start_method * [fix] Enable the stake blacklist (#931) * uncomment stake blaclist * add test for blacklisted stake * [Fix] use with env to reset start method after (#935) * use with env to reset start method after * . Co-authored-by: Cameron Fairchild <[email protected]> Co-authored-by: Mac Thrasher <[email protected]> Co-authored-by: opentaco <[email protected]> Co-authored-by: opentaco <[email protected]> Co-authored-by: Eduardo García <[email protected]> Co-authored-by: isabella618033 <[email protected]> Co-authored-by: Cameron Fairchild <[email protected]>
opentensor · Oct 3, 2022 · 3885779 · 3885779
1 parent 4e765ac
commit 3885779
Show file tree

Hide file tree

Showing 41 changed files with 1,282 additions and 354 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -116,7 +116,7 @@ workflows:
       - build-and-test:
           matrix:
             parameters:
-              python-version: ["3.7", "3.8", "3.9", "3.10.5"]
+              python-version: ["3.7.14", "3.8.14", "3.9.13", "3.10.6"]
       - unit-tests-all-python-versions:
           requires:
             - build-and-test

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -2,17 +2,16 @@
 
 The following is a set of guidelines for contributing to Bittensor, which are hosted in the [Opentensor Organization](https://github.com/opentensor) on GitHub. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
 
-#### Table Of Contents
+## Table Of Contents
 
-[I don't want to read this whole thing, I just have a question!!!](#i-dont-want-to-read-this-whole-thing-i-just-have-a-question)
-
-[What should I know before I get started?](#what-should-i-know-before-i-get-started)
-
-[How Can I Contribute?](#how-can-i-contribute)
-  * [Reporting Bugs](#reporting-bugs)
-  * [Suggesting Enhancements](#suggesting-enhancements)
-  * [Your First Code Contribution](#your-first-code-contribution)
-  * [Pull Requests](#pull-requests)
+1. [I don't want to read this whole thing, I just have a question!!!](#i-dont-want-to-read-this-whole-thing-i-just-have-a-question)
+1. [What should I know before I get started?](#what-should-i-know-before-i-get-started)
+1. [How Can I Contribute?](#how-can-i-contribute)
+  1. [Reporting Bugs](#reporting-bugs)
+  1. [Suggesting Enhancements](#suggesting-enhancements)
+  1. [Your First Code Contribution](#your-first-code-contribution)
+  1. [Pull Requests](#pull-requests)
+    1. [Development-Workflow](#development-workflow)
 
 ## I don't want to read this whole thing I just have a question!!!
 
@@ -122,10 +121,10 @@ The process described here has several goals:
 
 Please follow these steps to have your contribution considered by the maintainers:
 
-1. Follow all instructions in [the template](https://github.com/opentensor/bittensor/blob/master/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md)
-2. Follow the [styleguides](#styleguides)
-3. After you submit your pull request, verify that all [status checks](https://help.github.com/articles/about-status-checks/) are passing <details><summary>What if the status checks are failing?</summary>If a status check is failing, and you believe that the failure is unrelated to your change, please leave a comment on the pull request explaining why you believe the failure is unrelated. A maintainer will re-run the status check for you. If we conclude that the failure was a false positive, then we will open an issue to track that problem with our status check suite.</details>
+1. Before the PR.
+  1. Read the [development workflow](./DEVELOPMENT_WORKFLOW.md) defined for this repository in order to agree on the ways of working.
+1. While coding, please, add tests relevant to the fixed bug or new feature.
+1. To create the PR follow all instructions in [the template](https://github.com/opentensor/bittensor/blob/master/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md)
+1. After you submit your pull request, verify that all [status checks](https://help.github.com/articles/about-status-checks/) are passing <details><summary>What if the status checks are failing?</summary>If a status check is failing, and you believe that the failure is unrelated to your change, please leave a comment on the pull request explaining why you believe the failure is unrelated. A maintainer will re-run the status check for you. If we conclude that the failure was a false positive, then we will open an issue to track that problem with our status check suite.</details>
 
 While the prerequisites above must be satisfied prior to having your pull request reviewed, the reviewer(s) may ask you to complete additional design work, tests, or other changes before your pull request can be ultimately accepted.
-
-
diff --git a/DEVELOPMENT_WORKFLOW.md b/DEVELOPMENT_WORKFLOW.md
@@ -0,0 +1,164 @@
+# Development Workflow
+
+## Table of contents
+
+1. [Main branches](#main-branches)
+1. [Development model](#development-model)
+    1. [Supporting branches](#supporting-branches)
+        1. [Feature branches](#feature-branches)
+        1. [Release branches](#release-branches)
+        1. [Hotfix branches](#hotfix-branches)
+    1. [Git operations](#git-operations)
+        1. [Create a feature branch](#create-a-feature-branch)
+        1. [Merge feature branch into nobunaga](#merge-feature-branch-into-nobunaga)
+        1. [Create release branch](#create-release-branch)
+        1. [Finish a release branch](#finish-a-release-branch)
+        1. [Create a hotfix branch](#create-a-hotfix-branch)
+        1. [Finishing a hotfix branch](#finishing-a-hotfix-branch)
+
+## Main branches
+
+The repo holds two main branches with an infinite lifetime:
+- master
+- nobunaga
+
+We consider `origin/master` to be the main branch where the source code of HEAD always reflects a **__production-ready__** state.
+
+We consider `origin/nobunaga` to be the main branch where the source code of HEAD always reflects a state with the **__latest delivered development__** changes for the next release. Some would call this the `"integration branch"`. This is where any automatic nightly builds would be built from.
+
+## Development model
+
+### Supporting branches
+
+Each of these branches have a specific purpose and are bound to strict rules as to which branches may be their originating branch and which branches must be their merge targets. We will walk through them in a minute
+
+#### Feature branches
+
+- May branch off from: `nobunaga`
+- Must merge back into: `nobunaga`
+- Branch naming convention:
+    - Anything except master, nobunaga, nakamoto, release/* or hotfix/*
+    - Suggested: `feature/<ticket>/<descriptive-sentence>`
+
+Feature branches are used to develop new features for the upcoming or a distant future release. When starting development of a feature, the target release in which this feature will be incorporated may well be unknown at that point. 
+
+The essence of a feature branch is that it exists as long as the feature is in development, but will eventually be merged back into `nobunaga` (to definitely add the new feature to the upcoming release) or discarded (in case of a disappointing experiment).
+
+#### Release branches
+
+- May branch off from: `nobunaga`
+- Must merge back into: `nobunaga` and `master`
+- Branch naming convention:
+    - Suggested format `release/3.4.0/optional-descriptive-message`
+
+Release branches support preparation of a new production release. Furthermore, they allow for minor bug fixes and preparing meta-data for a release (e.g.: version number, configuration, etc.). By doing all of this work on a release branch, the `nobunaga` branch is cleared to receive features for the next big release.
+
+This new branch may exist there for a while, until the release may be rolled out definitely. During that time, bug fixes may be applied in this branch, rather than on the `nobunaga` branch. Adding large new features here is strictly prohibited. They must be merged into `nobunaga`, and therefore, wait for the next big release.
+
+#### Hotfix branches
+
+- May branch off from: `master`
+- Must merge back into: `nobunaga` and `master`
+- Branch naming convention:
+    - Suggested format: `hotfix/3.3.4/optional-descriptive-message` 
+
+Hotfix branches are very much like release branches in that they are also meant to prepare for a new production release, albeit unplanned. They arise from the necessity to act immediately upon an undesired state of a live production version. When a critical bug in a production version must be resolved immediately, a hotfix branch may be branched off from the corresponding tag on the master branch that marks the production version.
+
+The essence is that work of team members, on the `nobunaga` branch, can continue, while another person is preparing a quick production fix.
+
+### Git operations
+
+#### Create a feature branch
+
+1. Branch from the **nobunaga** branch.
+    1. Command: `git checkout -b feature/my-feature nobunaga`
+
+> Try to rebase frequently with the updated nobunaga branch so you do not face big conflicts before submitting your pull request. Remember, syncing your changes with other developers could also help you avoid big conflicts.
+
+#### Merge feature branch into nobunaga
+
+In other words, integrate your changes into a branch that will be tested and prepared for release.
+
+- Switch branch to nobunaga: `git checkout nobunaga`
+- Merging feature branch into nobunaga: `git merge --no-ff feature/my-feature`
+- Pushing changes to nobunaga: `git push origin nobunaga`
+- Delete feature branch: `git branch -d feature/my-feature`
+
+This operation is done by Github when merging a PR.
+
+So, what you have to keep in mind is:
+- Open the PR against the `nobunaga` branch.
+- After merging a PR you just have to delete your feature branch.
+
+#### Create release branch
+
+- Create branch from nobunaga: `git checkout -b release/3.4.0/optional-descriptive-message nobunaga`
+- Updating version with major or minor: `./scripts/update_version.sh major|minor`
+- Commit file changes with new version: `git commit -a -m "Updated version to 3.4.0"`
+
+#### Finish a release branch
+
+In other words, releasing stable code and generating a new version for bittensor.
+
+- Switch branch to master: `git checkout master`
+- Merging release branch into master: `git merge --no-ff release/3.4.0/optional-descriptive-message`
+- Tag changeset: `git tag -a v3.4.0 -m "Releasing v3.4.0: some comment about it"`
+- Pushing changes to master: `git push origin master`
+- Pushing tags to origin: `git push origin --tags`
+
+To keep the changes made in the __release__ branch, we need to merge those back into `nobunaga`:
+
+- Switch branch to nobunaga: `git checkout nobunaga`.
+- Merging release branch into nobunaga: `git merge --no-ff release/3.4.0/optional-descriptive-message`
+
+This step may well lead to a merge conflict (probably even, since we have changed the version number). If so, fix it and commit.
+
+After this the release branch may be removed, since we don’t need it anymore:
+
+- `git branch -d release/3.4.0/optional-descriptive-message`
+
+#### Create the hotfix branch
+
+- Create branch from master:`git checkout -b hotfix/3.3.4/optional-descriptive-message master`
+- Update patch version: `./scripts/update_version.sh patch`
+- Commit file changes with new version: `git commit -a -m "Updated version to 3.3.4"`
+
+Then, fix the bug and commit the fix in one or more separate commits:
+- `git commit -m "Fixed critical production issue"`
+
+#### Finishing a hotfix branch
+
+When finished, the bugfix needs to be merged back into `master`, but also needs to be merged back into `nobunaga`, in order to safeguard that the bugfix is included in the next release as well. This is completely similar to how release branches are finished.
+
+First, update master and tag the release.
+
+- Switch branch to master: `git checkout master`
+- Merge changes into master: `git merge --no-ff hotfix/3.3.4/optional-descriptive-message`
+- Tag new version: `git tag -a v3.3.4 -m "Releasing v3.3.4: some comment about the hotfix"`
+- Pushing changes to master: `git push origin master`
+- Pushing tags to origin: `git push origin --tags`
+
+Next, include the bugfix in `nobunaga`, too:
+
+- Switch branch to nobunaga: `git checkout nobunaga`
+- Merge changes into nobunaga: `git merge --no-ff hotfix/3.3.4/optional-descriptive-message`
+- Pushing changes to origin/nobunaga: `git push origin nobunaga`
+
+The one exception to the rule here is that, **when a release branch currently exists, the hotfix changes need to be merged into that release branch, instead of** `nobunaga`. Back-merging the bugfix into the __release__ branch will eventually result in the bugfix being merged into `develop` too, when the release branch is finished. (If work in develop immediately requires this bugfix and cannot wait for the release branch to be finished, you may safely merge the bugfix into develop now already as well.)
+
+Finally, we remove the temporary branch:
+
+- `git branch -d hotfix/3.3.4/optional-descriptive-message`
+
+## TODO
+
+- Changing the name of the develop branch from nobunaga to `integration`
+    - Because sometimes nobunaga are going to have a release branch.
+- Knowing if master and nobunaga are different
+- Knowing what is in nobunaga that is not merge yet
+    - Document with not released developments
+    - When merged into nobunaga, generate the information exposing what's merged into nobunaga but not release.
+    - When merged into master, generate github release and release notes.
+- CircleCI job 
+    - Merge nobunaga into master and release version (needed to release code)
+    - Build and Test bittensor (needed to merge PRs)
diff --git a/README.md b/README.md
@@ -1,15 +1,15 @@
 <div align="center">
 
 # **Bittensor** <!-- omit in toc -->
-[![Discord Chat](https://img.shields.io/discord/308323056592486420.svg)](https://discord.gg/3rUr6EcvbB)
+[![Discord Chat](https://img.shields.io/discord/308323056592486420.svg)](https://discord.gg/bittensor)
 [![PyPI version](https://badge.fury.io/py/bittensor.svg)](https://badge.fury.io/py/bittensor)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) 
 
 ---
 
 ### Internet-scale Neural Networks <!-- omit in toc -->
 
-[Discord](https://discord.gg/3rUr6EcvbB) • [Docs](https://app.gitbook.com/@opentensor/s/bittensor/) • [Network](https://www.bittensor.com/metagraph) • [Research](https://drive.google.com/file/d/1VnsobL6lIAAqcA1_Tbm8AYIQscfJV4KU) • [Code](https://github.com/opentensor/BitTensor)
+[Discord](https://discord.gg/bittensor) • [Docs](https://docs.bittensor.com/) • [Network](https://www.bittensor.com/network) • [Research](https://drive.google.com/file/d/1VnsobL6lIAAqcA1_Tbm8AYIQscfJV4KU) • [Code](https://github.com/opentensor/BitTensor)
 
 </div>
 

diff --git a/VERSION b/VERSION
@@ -0,0 +1 @@
+3.3.4
diff --git a/bittensor/__init__.py b/bittensor/__init__.py
@@ -18,7 +18,7 @@
 from rich.console import Console
 
 # Bittensor code and protocol version.
-__version__ = '3.3.3'
+__version__ = '3.3.4'
 version_split = __version__.split(".")
 __version_as_int__ = (100 * int(version_split[0])) + (10 * int(version_split[1])) + (1 * int(version_split[2]))
 

diff --git a/bittensor/_axon/__init__.py b/bittensor/_axon/__init__.py
@@ -2,6 +2,7 @@
 """
 # The MIT License (MIT)
 # Copyright © 2021 Yuma Rao
+# Copyright © 2022 Opentensor Foundation
 
 # Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated 
 # documentation files (the “Software”), to deal in the Software without restriction, including without limitation 
@@ -65,6 +66,8 @@ def __new__(
             server: 'grpc._Server' = None,
             port: int = None,
             ip: str = None,
+            external_ip: str = None,
+            external_port: int = None,
             max_workers: int = None, 
             maximum_concurrent_rpcs: int = None,
             blacklist: 'Callable' = None,
@@ -101,6 +104,10 @@ def __new__(
                     Binding port.
                 ip (:type:`str`, `optional`):
                     Binding ip.
+                external_ip (:type:`str`, `optional`):
+                    The external ip of the server to broadcast to the network.
+                external_port (:type:`int`, `optional`):
+                    The external port of the server to broadcast to the network.
                 max_workers (:type:`int`, `optional`):
                     Used to create the threadpool if not passed, specifies the number of active threads servicing requests.
                 maximum_concurrent_rpcs (:type:`int`, `optional`):
@@ -120,6 +127,8 @@ def __new__(
         config = copy.deepcopy(config)
         config.axon.port = port if port != None else config.axon.port
         config.axon.ip = ip if ip != None else config.axon.ip
+        config.axon.external_ip = external_ip if external_ip != None else config.axon.external_ip
+        config.axon.external_port = external_port if external_port != None else config.axon.external_port
         config.axon.max_workers = max_workers if max_workers != None else config.axon.max_workers
         config.axon.maximum_concurrent_rpcs = maximum_concurrent_rpcs if maximum_concurrent_rpcs != None else config.axon.maximum_concurrent_rpcs
         config.axon.forward_timeout = forward_timeout if forward_timeout != None else config.axon.forward_timeout
@@ -174,6 +183,8 @@ def __new__(
             server = server,
             ip = config.axon.ip,
             port = config.axon.port,
+            external_ip=config.axon.external_ip, # don't use internal ip if it is None, we will try to find it later
+            external_port=config.axon.external_port or config.axon.port, # default to internal port if external port is not set
             forward = forward_text,
             backward = backward_text,
             synapses = synapses,
@@ -214,9 +225,13 @@ def add_args( cls, parser: argparse.ArgumentParser, prefix: str = None  ):
         prefix_str = '' if prefix == None else prefix + '.'
         try:
             parser.add_argument('--' + prefix_str + 'axon.port', type=int, 
-                    help='''The port this axon endpoint is served on. i.e. 8091''', default = bittensor.defaults.axon.port)
+                    help='''The local port this axon endpoint is bound to. i.e. 8091''', default = bittensor.defaults.axon.port)
             parser.add_argument('--' + prefix_str + 'axon.ip', type=str, 
                 help='''The local ip this axon binds to. ie. [::]''', default = bittensor.defaults.axon.ip)
+            parser.add_argument('--' + prefix_str + 'axon.external_port', type=int, required=False,
+                    help='''The public port this axon broadcasts to the network. i.e. 8091''', default = bittensor.defaults.axon.external_port)
+            parser.add_argument('--' + prefix_str + 'axon.external_ip', type=str, required=False,
+                help='''The external ip this axon broadcasts to the network to. ie. [::]''', default = bittensor.defaults.axon.external_ip)
             parser.add_argument('--' + prefix_str + 'axon.max_workers', type=int, 
                 help='''The maximum number connection handler threads working simultaneously on this endpoint. 
                         The grpc server distributes new worker threads to service requests up to this number.''', default = bittensor.defaults.axon.max_workers)
@@ -253,6 +268,8 @@ def add_defaults(cls, defaults):
         defaults.axon = bittensor.Config()
         defaults.axon.port = os.getenv('BT_AXON_PORT') if os.getenv('BT_AXON_PORT') != None else 8091
         defaults.axon.ip = os.getenv('BT_AXON_IP') if os.getenv('BT_AXON_IP') != None else '[::]'
+        defaults.axon.external_port = os.getenv('BT_AXON_EXTERNAL_PORT') if os.getenv('BT_AXON_EXTERNAL_PORT') != None else None
+        defaults.axon.external_ip = os.getenv('BT_AXON_EXTERNAL_IP') if os.getenv('BT_AXON_EXTERNAL_IP') != None else None
         defaults.axon.max_workers = os.getenv('BT_AXON_MAX_WORERS') if os.getenv('BT_AXON_MAX_WORERS') != None else 10
         defaults.axon.maximum_concurrent_rpcs = os.getenv('BT_AXON_MAXIMUM_CONCURRENT_RPCS') if os.getenv('BT_AXON_MAXIMUM_CONCURRENT_RPCS') != None else 400
 
@@ -267,6 +284,7 @@ def check_config(cls, config: 'bittensor.Config' ):
         """ Check config for axon port and wallet
         """
         assert config.axon.port > 1024 and config.axon.port < 65535, 'port must be in range [1024, 65535]'
+        assert config.axon.external_port is None or (config.axon.external_port > 1024 and config.axon.external_port < 65535), 'external port must be in range [1024, 65535]'
         bittensor.wallet.check_config( config )
 
     @classmethod