Paris

The Paris update of Tezos ships with "Adaptive Issuance". As of cycle 749 of Tezos, Adaptive Issuance has kicked in. This significantly impacts staking, rewards and TRD.

For cycle 749 onwards, you must update TRD to this version. If you do not, things will look like they are still working, but you will be paying your delegators too much.

In tezos, rewards are derived from baking rights. In Oxford, baking rights come from delegation (including baker's own balance). In Paris, they are split in 2 unequal parts:

• 2/3 of rights come from staking. Staking is now an explicit operation: both bakers and their delegators can stake their tez (put them at stake). Staking rewards are paid in protocol, only tz account can bake (no smart contract can bake)
• 1/3 of rights come from delegation as before, however the methodology to calculate rewards is changing:
  • there is no more 5 cycle delay between delegation and reward accrual. this is replaced with a 1 cycle delay
  • there are no more concept of "snapshots" with balances taken at random, determining the stake (hence reward distribution). Instead, the lowest balance throughout the cycle is taken into account for reward calculation.
  • also, the fees collected in blocks by bakers (negligible today) are not taken into account for staking rewards. They all go to delegation rewards.

Note: this 2/3 / 1/3 distribution is global, not per baker. It is possible for a baker to have no delegators, only stakers, then their delegation rewards are nil.

There is no RPC to query this lowest balance. Therefore, the ability to implement a simple RPC-based reward distributor has been lost. This was already disabled in TRD - we have now removed the code completely.

For a while, the only supported backend of TRD has been tzkt. This will remain the only supported solution in Paris, however tzkt did not implement the same calculations as the protocol for minimal balance, because it was unable to (more details below).

As you deploy the new release, it is advised to do a dry-run first. Here is how to sanity-check your dry-run:

In the logs for your dry-run, you will see the following line:

INFO - Total rewards before processing is 100,000,000 mutez.

This value is no longer the total rewards. This is only the part of rewards attributable to delegation. You will get more rewards from staking.

This is not the same value as the one you can see in tzkt "rewards" page, which sums up up delegation and staking rewards. You can however check this value in the API by querying the following:

curl -s https://api.tzkt.io/v1/rewards/split/tz1xyz/749 | jq '(.blockRewardsDelegated + .endorsementRewardsDelegated) / 1000000'

(replace tz1xyz with your baker address)

A little below, in the logs, you will see the following:

Total estimated amount to pay out is 90,000,000 mutez.

This is what TRD will pay out.

As a public baker, most of your balance is probably staked (and if it is not, you should fix it by running the stake command). This has another effect: TRD no longer "carves out" some of the rewards for your own stake. Indeed, your own stake is paid for by the protocol, using the new mechanism.

Under AI, most of what TRD pays you is delegation fees. You can sanity check with the 2 values above: if the rewards attributable to delegation are 100 tez, and your fee is 10%, then you should expect TRD to pay out about 90 tez. This is assuming a normal config without any "owners" or "founders" set (I have not tested this). If the values you are seeing in the dry run are significantly outside of this, please reach out on slack.

The amount you will pay out at every cycle will probably be less than what you are used to paying prior to Adaptive Issuance activation. It's normal: in the new model, delegators will earn less, stakers earn more.

How this release was tested

The PR #703 (commit eaad513) was used to pay out rewards for cycle 749 for 2 small-ish mainnet bakers with no issues. One of them is accepting third-party stakers and the other is not. The values are looking good according to the rule of thumb above. Several bakers then used this commit to pay out cycle 749 and node reported issues.

Caveat emptor:

1. CI is failing. We are lacking workforce to fix it - in fact, the changes in Adaptive Issuance are of such magnitude that it might be better to start from an empty codebase and selectively import the python code that we still need, and write a new set of tests. This means, some things might be broken. We simply disabled pytest for this release.
2. The current PR has not been tested in every configuration - especially no distinct "owners" or "founders". I am tempted to delete these features as well, but I don't see why AI would break it, so I left them. use at own risk.
3. TRD accuracy is 100% downstream of tzkt accuracy. However, tzkt does not provide accurate data from the protocol, because this data is simply not accessible. Instead, it approximates it. This is discussed at length in this ticket. Specifially, the delegator balance is measured at the end of the cycle by tzkt. The protocol takes the minimum. This is potentially gameable by delegators.
4. This new way of calculating delegating balance by picking the maximum is breaking smart contracts such as kolibri ovens, for which balance goes to zero when you pay them out. Therefore, paying out kolibri ovens lowers your delegation rewards for the cycle where the payout takes place. A fix is coming for protocol Q. But for the time being, it's a good idea to not pay kolibri ovens, or pay them to the originator address rather than the contract iself. TRD has a feature for that called specials_map, see doc. It's also worth noting that TezPay has implemented some middleware that addresses the problem, but we are not making use of it, due to lack of time, mostly.
5. there is an edge case of rewards unstaked from one baker and pending finalize unstake. these ones are taken into account for delegation rewards by the protocol, but not by tzkt, and therefore these balances do not get payouts from TRD
6. keep in mind that the license for the software says that we make no guarantee, implicit of explicit of proper operation, or suitability for a particular purpose.

Changelog

• change sources for splitting up the delegation rewards
  • look at delegatedBalance instead of balance of bakers and delegators, as numerator
  • look at blockRewardsDelegated + endorsementRewardsDelegated as denominator
• do not include bakers' fees as reward as discussed in issue
• deprecate & remove code for "estimated" and "ideal" rewards. only "actual" rewards remain
• remove code for rpc and tzstats data source. only tzkt remains
• some doc updates
• disable CI - no resource to maintain the testing codeees as reward as discussed in issue
• update next proto Q date - do not prompt for confirrmation if this date is passed as this breaks docker
• updates blocks_per_cycle constant to fix continuous mode

Tezos Reward Distributor: Oxford2

What's Changed

• First payments done since v11.0 by @jdsika in #699
• Update copyright, instructions and version numbers by @jdsika in #698
• Fix: Use Windows compatible signals by @nicolasochem in #695
• Feat: Make tzpro_api_key configuration optional by @nicolasochem in #694
• Fix: Single-shot mode always exiting in error by @nicolasochem in #679
• Fix: Logger recursion by @nicolasochem in #692
• Update: Docker image and Python version by @nicolasochem in #693
• Fix: Twitter plugin to api v2 by @ericlavoie in #691
• Feat: Record playback tests with API calls by @vkresch in #690
• Fix: update constants.py by @jdsika in direct commit
• Fix: Cleanup launch and requirements installation by @jdsika in #686
• Fix: enum comparisons and fix api key by @vkresch in #688
• Feat: Migrate tzstats to tzpro by @jdsika in #685
• Fix: tests_api_consistency by @denver-s in #683

Full Changelog: v11.1...v12.0

Mumbai and Nairobi support

What's Changed

• Mock response block data by @vkresch in #644
• Update RPS for TzKT API by @dmirgaleev in #645
• Added Vlad1mir-D into list of contributors by @Vlad1mir-D in #647
• Set gas limit for tz1-to-tz1 transactions to 1001 by @denver-s in #646
• GitHub Actions: update to Node v16 by @denver-s in #648
• docs: add rvermooten to contributors list by @rvermootenct in #650
• Refactor batch payer by @rvermootenct in #639
• Refactor/configure by @rvermootenct in #651
• fix: confirm all zero balance accounts are dealt with by @rvermootenct in #652
• fix: remove docker arguement for configure by @rvermootenct in #654
• fix: baker address should not be required to be known to the signer by @rvermootenct in #656
• Added multisig doc by @utdrmac in #658
• Bugfix: Founders and Owners balance correctly updated + Reactivate zero by @jdsika in #660
• HotFix Reward API argument by @vkresch in #662
• Fix ignore infinite tests by @vkresch in #665
• disable RPC after Mumbai by @nicolasochem in #664
• adapt ghostnet constats + deprecate -R --release-override by @jdsika in #661
• fix adjusted_early_payout argument by @nicolasochem in #668
• Update twitter.py by @ericlavoie in #671
• Feat/better exit messages by @rvermootenct in #657
• release/v11 cont. by @jdsika in #677
• Mumbai & Nairobi Support by @jdsika in #667

New Contributors

• @dmirgaleev made their first contribution in #645

Full Changelog: v11.0...v11.1

Tezos Reward Distributor: Lima

List of merged pull requests

General remarks:
This release is scheduled in accordance with the TRD grant milestone plan as well as the activation of the Lima protocol. Due to high relevance of the topic, you can find instructions here about how TRD can be used to send funds to the community-driven donation contracts for Ukraine or use it as a template to donate to any other address.

Important:
This update has been tested on Ghostnet after the Lima protocol has been activated. Nevertheless always be careful after a protocol activation. Wait a few hours until the network has stabilized itself and always do a --dry_run before actually executing payments and check the simulated CSV report. Furthermore, please SHUT DOWN THE TRD LINUX SERVICE BEFORE THE UPGRADE GOES LIVE!

Additional Features:

• Attention: TZSTATS data provider has been reactivated in #610 . An API feature had been deprecated which has been re-introduced in summer for TRD.
• Add min_payment_amt configuration option to allow setting minimum payment amount in #613 .
• Additional option for --dry_run to run with or without signer. See --help.
• Extended safety check to avoid usage of master branch after 31.03.2023 due to Mumbai transition.

Breaking Changes:

• Deprecated --dry_run_no_consumers in #631 .

• Packages are available HERE.

Changes:

• Fixed a bug for simulation errors for kt accounts regarding uncaught exceptions.
• Fixed too early disk size warning.
• Fixed consumed_gas calculation after Kathmandu upgrade.
• Adjust fee computation logic for Ithaca and Jakarta.
• Tenderbake compatible.
• Increased test coverage.
• Updated python to 3.10.8 .
• Added pre-commit hooks.
• Documentation improvements.
• Code cleanups.

Tezos Reward Distributor: Hangzhou (stable)

List of merged pull requests

General remarks:
There have been many changes in the development of the TRD since a new grant was awarded by the Tezos Foundation a couple of months ago. The new milestones are now closely tied to upcoming protocol upgrades. Unfortunately, we see a significant amount of bakers using the tool without dedicating time and resources to it. To maintain the tool the way we do now, we need way more resources and support from its users. The grant is intended to serve as an interim solution for an independent open-source tool in the future. To encourage attraction for code contributions, the hourly wage was increased to $70, and the compensation with tez is done via a multi-sig wallet depending on the effort spent. We would also like to appeal to the bakers, who are still developing their own payout tool to finally bundle the resources. Last but not least, here you can find instructions about how TRD can be used to send funds to the community-driven donation contracts for Ukraine or use it as a template to donate to any other address.

Important:
This release serves as a stable release before the Ithaca PR will be merged after the confidence is high enough that there are no severe bugs. This release and the current master branch will automatically shut down after 01.04.2022. We especially need help with testing from the community after the Ithaca upgrade will be live. For testing, please switch to branch "ithaca2" because the master branch will not be able to do payouts on Ithaca. Always use the dry-run option for testing! Furthermore, please SHUT DOWN THE TRD LINUX SERVICE BEFORE THE UPGRADE GOES LIVE!**

Additional Features:

• Attention: Added safety check to avoid usage of master branch after 01.04.2022 due to Ithaca transition. Switch to branch ithaca2 and carefully test.

• Breaking: The reward_type estimated is deprecated in #540.

  • You can now payout early by using --release_override -11 and adjust the payouts afterwards to the reward_types actual or ideal

  • Read about the new feature here

  • Introduced new fields in the calculations csv for overestimate, adjusted_amount and adjustment for new feature

    old	new
    estimated	adjusted_early_payouts
    actual	actual
    ideal	ideal

• Potentially Breaking. Created new trd-data folder structure. Before, in rare cases the calculations csv was overwritten.

  • deprecate --reports_base and --config_dir

  • unified --log-file to --log_file

  • introduce --base_directory

  • logs by default in base directory

    # ~/pymnt/
    # └──cfg
    #    └── tz1xx.yaml
    # └──logs
    #    └── verbose_backup
    #    │   └── app_verbose.gz
    #    └── app.log
    #    └── app_verbose.log
    # └──simulations
    #    └── tz1xxx
    #       ├── calculations
    #       │   └── 449.csv
    #       └── payments
    #           ├── done
    #           │   └── 449.csv
    #           └── failed
    #               └── 449.csv
    # └──reports
    #    └── tz1xxx
    #       ├── calculations
    #       │   └── 449.csv
    #       └── payments
    #           ├── done
    #           │   └── 449.csv
    #           └── failed
    #               └── 449.csv
    

• Payment reporting, see #549 :
  Important debugging advice:

  • The payments have been correct for all tz accounts
  • kt accounts with burn fees have been incorrect as burn was always subtracted from the delegator (edge case)
  • kt burn fee is now also controlled by delegator_pays_ra_fee configuration
  • Reported overall payout amount has been incorrect and is fixed
  • Added simulated transaction fees into the calculations report csv: delegate_transaction_fee and delegator_transaction_fee
  • Added description field with error messages in payment csv for unsuccessful transactions
  • payment_logs had missing items regarding items with the status AVOIDED
  • payments that are below zero_threshold are now set to status DONE according to the definition
  • Using user friendly enum names for the payment status now
  • Changed all occurrences of XTZ and tez to mutez properly as formatted int( ) for consistency

• Plugin for Discord available

• Support all Tezos curves now

• Packages now available HERE

Changes:

• TRD checks disk size to avoid double payments when running out of space
• Improved --helpsection
• Using black formatting now
• Reduced Docker size
• Fixed versioning for docker images on DockerHub
• Optimization/corrections of fees, gas limits to avoid failed payments
• Unit testing has been increased
• Improved logging
• Exception handling and robustness has been increased
• Documentation improvements
• A lot of code cleanups
• Fixed several bugs (see PR list)

Tezos Reward Distributor: Florence (Update 1)

List of merged pull requests

Here is a link to the Agora post explaining the changes in more detail.

Additional Features:

• Breaking: New reward_type ideal introducing payouts like Backerei. Old ideal has been renamed to estimated. The discussion in #415.

  old	new
  ideal	estimated
  actual	actual
  	ideal

• Payment reporting:
  Important debugging advice:

  • New payment status: skipped
  • e.g. liquidated ovens are skipped
  • Payments to kt1 accounts are skipped if fee > payment_amount as a precaution and marked as skipped
  • Consider redirecting kt1 payments to the entitled tz1 address through the configuration
  • Done and failed payouts are now split into two files

• Docker images now available HERE

• Optimization of fees, gas limits and batch sizes ordered by transaction type (tz, kt)

• frozen rewards can now be payed out using RPC

• Log archives are using gzip now

Changes:

• Major: Introduction of a finite state machine, FSM to increase application safety and prepare storing application states for program recovery
• Breaking: -C --initial_cycle has now a default value of -1 meaning start at last unfrozen cycle and now only allows ""positive cycle values**
• Breaking: -R --release_override is now restricted to [-11,0]
• Breaking: -M --run_mode 4 does honor the -C parameter and runs all failed payments by default
• Breaking: fee.ini has been removed
• Bug: tzkt is not paying out accusation rewards. Rationale: The option will be configurable in the future and now in most cases bakers return it due to honest mistakes
• Resolved issue when payments resulted in an invalid branch error
• Unit testing has been increased
• Improved logging
• Exception handling and robustness has been increased
• Documentation improvements
• Using gh-actions now
• Fixed several bugs (see PR list)

Findings

• When using ideal: Tzkt counts missed baking fees, but because it also counts the missed block reward more accurately, it can end up lower than tzstats. Therefore, we currently encourage users when using ideal, to use tzkt. We will reach out to the tzstats team.

Tezos Reward Distributor: Delphi (Update 2)

List of merged pull requests

Here is a link to the Agora post explaining the changes in more detail.

Additional Features:

• Payouts to Kt1 accounts with default entry points.
  Important debugging advice:
  • Payouts will fail if the needed fees are too high
  • The value is currently hard coded as a safety switch in order to protect bakers from being drained by too complex contracts
  • The fee will be configurable in v9.0
  • Investigate failed transactions with the baking bad explorer
  • Check if the contract has a default entry point
• Individual fees for Dexter LPs
• Payout Ideal Rewards
• Admin messages in plugin system

Changes:

• Major: Removal of the (ocaml) tezos-client binary as a dependency
• Breaking: Aliases in the configuration are not supported anymore due to the removal of the tezos-client
• Breaking: Use the tezos-signer in http mode ./tezos-signer launch http signer
• Unit testing has been increased
• Improved logging
• Exception handling and robustness has been increased
• Ask user for permission to install dependencies
• Fixed bug leading to multiple transactions which should have been batched using redirect
• Fixed several bugs (see PR list)

Tezos Reward Distributor: Delphi (Update 1)

List of merged pull requests

Documentation:

• CRITICAL BUG FIX: A bug leading to missing payouts with PRPC has been fixed (see #321)
• Contributors list is now easier to read
• Unit testing has been increased, Pytest and flake8 has been introduced
• Reworked logging mechanism
• Exception handling and robustness has been increased

Features:

• BREAKING CHANGE: Logging and verbose logging has been improved for easier bug reporting, option: -V need argument {on;off}
• Package dependencies are automatically installed (introduced due to customer feedback)

Tezos Reward Distributor: Delphi

List of merged pull requests

Documentation:

• Added better example configuration example
• Added explanation on how to support Dexter

Features:

• Added Dexter liquidity pool payout support
• Added plugin system to use TRD with: email, twitter, webhook, telegram
• Frozen rewards now payable with TzKt
• Code cleanups and increased robustness in edge cases
• More consistent logging
• Fixed several bugs and made improvements (sort PRs by label for more information)

Tezos Reward Distributor: Carthage

List of merged pull requests

Documentation:

• Added guidelines for the contribution in the context of the TRD grant
• Added list of payments for contributors
• Improved README and documentation

Features:

• Improved tzstats provider by up to 50x
• Added TzKT as additional provider
• Added new run mode for retrying payments
• Check balance and warn user in case of insufficient funds (also by email)
• Automatically retry payments in unattended mode
• Extended tests for unattended mode
• Allow deleting lock file at runtime
• Anonymous statistics work again - see documenation
• Improved error messages and introduced several pre-checks
• Fixed several bugs and made improvements (sort PRs by label for more information)