[ZK Stack] - <Prover Build Fails During CUDA Object Compilation (__shared__ Variable Initialization Error)>

[Szczepoo13]

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.83.0-nightly

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

When I run command 'zk_inception prover run' to start prover - I expect the build to go successfully through, however I am encountering an issue when running the prover, the build of crate of boojum-cuda v0.150.4 does not complete (error: failed to run custom build command for 'boojum-cuda v0.150.4'), PTX JIT spills registers and I encounter the following error: initializer not allowed for shared variable.

Expected Behavior

During the run of the command 'zk_inception prover run', and choosing option 'prover' the compilation of the boojum-cuda crate succeeds and prover is started

Current Behavior

During the run of the command 'zk_inception prover run', and choosing option 'prover' the compilation of the boojum-cuda crate, a custom build command related to CUDA fails:

It is disallowing the prover to start successfully.

Repository Link (if applicable)

No response

Additional Details

I am unsure if this is related to a misconfiguration in my CUDA setup or something else. Could you please advise on potential solutions or debugging steps I can take?
My environment details:
GCC: 12.3.0
CMake: 3.30.2
NVCC: 11.5, V11.5.119
Clang: 14.0.0
Rustup - nightly-x86_64-unknown-linux-gnu

I have also tested my env with NVIDIA CUDA samples, and all tests pass successfully (see figure below):

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[gleenai[bot]]

It appears that the issue you're encountering with the boojum-cuda crate during the zk_inception prover run command is related to CUDA programming specifics, particularly with the initialization of __shared__ variables in your CUDA code. This error typically indicates that there's an attempt to initialize a __shared__ variable at the point of declaration, which is not allowed in CUDA.

Here are a few steps you can take to troubleshoot and potentially resolve this issue:

1. Check CUDA Version Compatibility: Ensure that the version of CUDA you are using is compatible with the boojum-cuda crate. Sometimes, newer or older versions of CUDA might have different requirements or behaviors that could lead to compilation issues.

2. Review CUDA Code: Look into the CUDA code where the error occurs. The error message should point you to the specific file and line number where the issue is present. Make sure that __shared__ variables are not being initialized at the point of declaration. Instead, they should be initialized in the kernel code.

3. Update Dependencies: Ensure that all dependencies, including Rust and CUDA, are up to date. Sometimes, updating to the latest versions can resolve unexpected issues.

4. Consult Documentation and Community: Check the ZKsync documentation and community resources for any similar issues or guidance on setting up the prover environment correctly. Sometimes, specific settings or tweaks are necessary to get everything working smoothly.

5. Environment Setup: Since your environment passes all NVIDIA CUDA samples tests, it's likely correctly set up. However, double-check any environment variables and paths related to CUDA to ensure they are correctly configured for your development environment.

If these steps do not resolve the issue, it might be beneficial to reach out directly to the ZKsync community or support channels with details of the error and your setup for more targeted assistance.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[robik75]

Hi @Szczepoo13 , it looks like you are using CUDA 11.5 to compile which would be my first guess for why the compilation is failing.
Please install CUDA 12.x and try again.