Investigate performance of proving for padded height $2^{22}$ and 2^{23}$

[Sword-Smith]

Running both with and without TVM_LDE_TRACE="no_cache", I've seen some quite troubling performance numbers for proving when the padded height is $2^{23}$. We should get performance numbers and RAM consumption for these three padded heights and investigate: $2^{21}$ (for reference), $2^{22}$, and $2^{23}$.

Specifically, a user with 2TB RAM and 256 cores reports that proving for a padded height of $2^{23}$ takes two hours where it, by my calculations, should take less than 600 seconds. And I ran out of RAM when attempting to create these proofs, even with TVM_LDE_TRACE="no_cache" and having 768GB RAM, which should be plenty for this configuration.

[jan-ferdinand]

Do you have insight into the various tables' heights, the program being run, etc.?

[Sword-Smith]

The behavior was observed when running a SingleProof program with many inputs, which translates to the verification of many correct executions of RemovalRecordIntegrity.

This piece of code exhibits the problem, for a padded height of $2^{23}$:
Neptune-Crypto/neptune-core@ec9e693#diff-b03414229c3f4a14c7601dfee113a82c91419c0d1f5fe4181835d6609650f2edR1587-R1600

Edit: The above-marked function call in the test crashes on our hardware, though: Even with 768GB RAM and TVM_LDE_TRACE="no_cache". A way forward would be to run it with live time reporting activated to see where the problems occur.

[jan-ferdinand]

Can you extract the VM's initial state or the (program, input, non-determinism)-triple from this function? That would allow analysis in Triton CLI.