Release v4.1.1

This release fixes a few issues when increasing the competition count.

• Downloaded model limit is now adjusted dynamically by competition count.
• Competition specific weights rather than subnet weights are used when prioritizing models to keep. (This helps clear out UIDs who swap the competition they are competing in from their previous competition).
• Threshold for model prioritization from other top validators is lowered to 5% weight as weights are spread more thinly over multiple competitions.

Release v4.1.0

Announcing Release v4.1.0

Announcing activation blocks for two new sub-competitions.

Changes

• This version schedules the launch of the 3B model competition, and the 7B* experiment with an epsilon of 0.001. Both are scheduled to be activated on block 3_601_190.

• Sample unpacking is also scheduled to start on block 3_601_190.

• All submitted models for the 7B competition will be automatically attributed to the 7B* experiment sub-competition. The latter will show as competition 3 in the wandb logs. But please use --competition_id 0 when you upload your model. DO NOT use --competition_id 3 when you upload your model.

Note: As we continue to increase the number of competitions, we might observe some fluctuations in vTrust. We will monitor them closely.

Release v4.0.1

This release adds in the necessary subnet 9 specific CompetitionId IntEnum.

Release v4.0.0

Announcing Release v4.0.0

Subnet-9 now supports multiple concurrent competitions.

Changes

• This version implements multi-competition support and a programmed launch for the 700M model competition in parallel with the current 7B model competition. It also contains the 3B competition constraints ready for scheduling.

• The validator version shipped with this release is bumped 3.0.0. This mean that the validator state will be erased and evaluation for all models will be restarted from scratch.

• All submitted models for 7B will be automatically attributed to the 7B sub-competition with a default competition ID of 0.

• Past submissions for 772M models and less will be ignored, even after the launch of the 700M competition on Thursday.

• Miner weights and model losses will be logged to wandb along with the competition ID for each miner: https://wandb.ai/macrocosmos/pretraining-validators?nw=nwusercloudops

• Related discord announcement: https://discord.com/channels/799672011265015819/1162768567821930597/1270164103125663807

• Allowed models list has been adjusted to include new model types:

  • Gemma2ForCausalLM
  • StableLmForCausalLM
  • Phi3ForCausalLM
  • Qwen2ForCausalLM

• The flag --competition_id should now be used with the scripts/upload_model.py script to specify the competition the miner is willing to submit their model to.

• No change to the validator launch command.

Release 3.2.2

Announcing Release 3.2.2

This is a patch release with some minor changes:

Changes

• Validators Logging: The logging for validators has been redirected to the pretraining-validators project within the Macrocosmos Wandb entity.
• Repository Version Logging: The repository version is now logged in Wandb along with the validator version. The version entry in the overview tab on the validator logging page in Wandb now refers to the repository version instead of the validator version.

Release 3.2.1

Announcing Release 3.2.1

This bugfix release addresses a critical issue impacting validator performance and stability. We strongly recommend all users update to this version to ensure optimal functionality.

Bugfix

• Description of the Bug: In the previous release (v3.2.0), all models submitted before block 3256604 were set to be evaluated on RefinedWeb instead of FineWeb-Edu Score-2. This were not the desired behavior. After the specified block, we want all models to be validated on FineWeb-Edu Score-2.
• Resolution: We have fixed the issue, ensuring that all models are now correctly evaluated on the FineWeb-Edu Score-2 dataset starting from block 3307004 instead of block 3256604 (one week earlier), as was originally intended.

Validator

• Validators will now correctly evaluate all models on the FineWeb-Edu Score-2 dataset, as introduced in the previous release.
• The validator version remains decoupled from the package version, maintaining flexibility for future updates.
• We have updated package dependencies, pinning Bittensor to version v6.9.3 to enhance the validator setup experience and stability.

We urge everyone to pull this release to update their validators and benefit from the critical bug fixes included in this release.

Release 3.2.0

Announcing Release 3.2.0

This release introduces several enhancements to improve the validator's performance and stability. The main contribution is the adoption of a new, higher-quality dataset and streamlining the validator setup process.

Validator

• Validators will now default to evaluating miners on the FineWeb Edu (score 2) dataset. This dataset, recently released and open-sourced by HuggingFace, is bigger and of higher quality.
• Evaluation on this new dataset will take effect starting from block 3256604.
• The validator version has been decoupled from the package version, allowing for more flexible and independent updates.
• We have improved the validator setup experience by cleaning up package dependencies.

Other

• We updated the leaderboard links to point to the new stable version hosted by Macrocosmos, ensuring accurate and up-to-date performance tracking.

We urge everyone to pull this release to update their validators and take advantage of these new features and improvements.

What's Changed

• Release 3.2.0 by @alan-aboudib-mc, @RodrigoLPA and @steffencruz in #126

Full Changelog: v3.1.4...v3.2.0

Release 3.1.4

Announcing Release 3.1.4

This release contains improvements to help keep VTrust stable across validators and closes a potential exploit of copying metadata directly from the chain within the same block. This is a proactive fix for subnet 9 after observing this occurring on subnet 6.

Validators:

• Validators now allow for the metadata hash to match the hugging face repo or the hugging face repo + the hotkey that uploaded it.

• Validators now default to evaluating on 18 pages of falcon instead of 12.

• Validators now use weights instead of incentive for shortcutting and retrying top models.

Miners:

• Using the flag –use_hotkey_in_hash will upload metadata that combines the hash of the hugging face repo with their hotkey.

  • We recommend waiting a few days for validators to update first before using this new functionality.

  • To avoid an attacker downloading the repo within 12 seconds to create their own hash you can also upload initially in private and make it public after the block has been committed.

Release 3.1.3

Force redownload of top models on retry.

Release 3.1.2.

Validator only fix to resolve OOM issues.