context: introduce new global context API with rerandomization #806
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @Kixunil @TheBlueMatt @dpc @JeremyRubin (if you still care about this) @tcharding This PR is a bit nasty but I scoped it to "just the hard parts" and the rest of it should be cathartic API changes that Tobin and I should be able to power through on our own. |
apoelstra
force-pushed
the
2025-06_context
branch
from
02df2d1 to
879205e
Compare
This introduces the new global context API when std is enabled, using thread locals to allow rerandomizing the context after sensitive operations. As you can see, even the simple case involves some unsafe code and is a bit tricky to implement.
apoelstra
force-pushed
the
2025-06_context
branch
from
879205e to
0026677
Compare
|
I don't particularly care anymore, but from memory: part of what makes this confusing is in WASM contexts we explicitly want "perfect" determinism, so we don't want to give any external observables. probably the only way to do this is to hijack getrandom() and give a entropy transcript through it, and be sure we're not running multi-threaded anything, or we want to e.g. call to a host API to perform signature operations. what we largely want to avoid is that an initialization somewhere from some far-flung library code, or by enabling a feature, means that all of the sudden we do a getrandom call for a context initialization when we're just doing verification work. cheers, jeremy |
apoelstra
force-pushed
the
2025-06_context
branch
from
0026677 to
7e26f33
Compare
|
Force-pushed to update tons of unit tests, and to update the recovery API. The essential code is unchanged. @JeremyRubin this new code only ever calls |
apoelstra
force-pushed
the
2025-06_context
branch
3 times, most recently
from
a66b75a to
dbb164f
Compare
This was referenced Jun 22, 2025
tcharding
reviewed
Jun 25, 2025
src/context.rs
Outdated
| /// Borrows the global context and do some operation on it. | ||
| /// | ||
| /// If provided, after the operation is complete, [`rerandomize_global_context`] | ||
| /// is called on the context. If you have some random data available, |
There was a problem hiding this comment.
Suggested change
| /// is called on the context. If you have some random data available, | |
| /// is called on the context. If you have some random data available. |
In 9330ccd and again in the previous commit for the std version.
There was a problem hiding this comment.
Heh, this sentence is just totally broken. It should say "If some random data is provided, then after the operation is complete, [rerandomize_global_context] is called."
tcharding
previously approved these changes
Jun 25, 2025
My initial plan for this commit was to implement the nostd version without randomization support, and patch it in later. However, I realized that even without rerandomization, I still needed synchronization logic in order to initialize the global context object. (Upstream provides a static "no precomp" context object, but it has no precomputation tables and therefore can't be used for verification, which makes it unusable for our purposes). In order to implement initialization, with ChatGPT's help I implemented a simple spinlock. However, there are a number of problems with spinlocks -- see this article (from Kix in rust-bitcoin#346) for some of them: https://matklad.github.io/2020/01/02/spinlocks-considered-harmful.html To avoid these problems, we tweak the spinlock logic so that we only try spinning a small finite number of times, then give up. Our "give up" logic is: 1. When initializing the global context, if we can't get the lock, we just initialize a new stack-local context and use that. (A parallel thread must be initializing the context, which is wasteful but harmless.) 2. Once we unlock the context, we copy it onto the stack and re-lock it in order to minimize the time holding the lock. (The exception is during initialization where we hold the lock for the whole initialization, in the hopes that other threads will block on us instead of doing their own initialization.) If we rerandomize, we do this on the stack-local copy and then only re-lock to copy it back. 3. If we fail to get the lock to copy the rerandomized context back, we just don't copy it. The result is that we wasted some time rerandomizing without any benefit, which is not the end of the world. Next steps are: 1. Update the API to use this logic everywhere; on validation functions we don't need to rerandomize and on signing/keygen functions we should rerandomize using our secret key material. 2. Remove the existing "no context" API, along with the global-context and global-context-less-secure features. 3. Improve our entropy story on nostd by scraping system time or CPU jitter or something and hashing that into our rerandomization. We don't need to do a great job here -- if we can get even a bit or two per signature, that will completely BTFO a timing attacker.
…d FromStr Since we have a no-feature-gate global context now, we can remove the feature gates from these things. No API change (other than an expansion of the API for users without features enabled).
Sometihng like half the tests in this crate are gated on "rand", most of which are for dumb reasons (we are generating random keys from the thread rng). By adding a non-feature=rand "random key generator" we can enable these tests even without the rand feature. We typically also have a gate on "std", which is needed to get the thread rng, but in some cases this is the *only* reason to have a std gate. So by eliminating the rand requirement we can make tests work in nostd. We do this by implementing a parallel LCG which is obviously not cryptographic but is fine for testing. In the LLM-generated tests in musig2.rs we have some rand feature gates for literally no reason at all :/. My bad. In addition to dramatically increasing nostd test coverage, the new "generate random keys" function also gives us an opportunity to use the new global context API including rerandomization.
This updates a couple functions, and their associated unit tests (which no longer need any std/alloc/global-context feature gates). This runs clean in valgrind, providing some evidence that my new code is sound.
This API is basically unused except for some niche or legacy applications, so I feel comfortable breaking it pretty dramatically. Move all the Secp256k1 functions onto RecoverableSignature and use self/Self as appropriate. Leave the stupid ecdsa_recoverable names even though they are even more redundant, because this module is basically in maintenance mode. We only do these changes since we'll be forced to once we drop the Secp256k1 object.
apoelstra
force-pushed
the
2025-06_context
branch
from
dbb164f to
d2cbb91
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As discussed in #388 and its parent issues, when
stdis enabled we have a fairly straightforward way to enable global contexts. We use thread-local variables and on every access we rerandomize them. When therandcrate is also available the situation is even better, because we don't need to think too hard about where to get entropy from.In the nostd case things are harder. We have no thread locals and basically no synchronization primitives except atomics, which can be used to implement spinlocks but nothing else. Kix has argued strongly against spinlocks but in the following several messages we came to a solution in which do a "soft spinlock" where after a couple iterations we just give up and don't rerandomize.
Kix suggested adding some logging and debugging facilities, which I did not include in my solution here. We can add those in a followup.
Kix also suggested setting the maximum spin count to 0, on the theory that in most cases there will never be any contention except in cases of reentrancy, and in that case spinning is pointless. I think it should be higher than zero to help in situations where there really are multiple threads. I set it to 128 which shouldn't be a noticable (or even measurable) burden even in the case where the spinning is pointless.
This mostly resolves #388. To completely resolve that issue, we need to:
Once we've done that, we will be much better-equipped to address #346. To do that, we should attempt to scrape together some entropy even on nostd without the rand crate. I believe we can do this by reading the system time and CPU jitter. We don't need to do a very good job for this to work; even a bit or two of entropy on each signature will BTFO an attacker attempting to learn timing information from multiple signatures.