Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RV32 U-Mode JIT is nonconformant and slow on ctx switch #133

Open
LekKit opened this issue Mar 29, 2024 · 5 comments
Open

RV32 U-Mode JIT is nonconformant and slow on ctx switch #133

LekKit opened this issue Mar 29, 2024 · 5 comments
Labels
bug Something isn't working inefficiency Better implementation is desired

Comments

@LekKit
Copy link
Owner

LekKit commented Mar 29, 2024

The issue

  • Running riscv32 userland with riscv64 kernel is broken

Steps to reproduce

  • Use those firmware/kernel/rootfs: rv32_umode.zip
  • Run rvvm fw_jump.bin -k linux_6.8 -i rootfs_rv32.ext2 -nojit
  • The userland will crash almost immediately with random unhandled pagefaults inside the kernel or segfaults in userspace

Investigation

  • RV32 U-mode RVJIT is most likely broken. It doesn't sign-extend the dirty 32-bit registers upon spilling them into hart ctx. So just disable it for now and fix later.
  • Interpreter properly sign-extends writes into rv32 registers to full 64-bit register by intermediate cast to sxlen_t:
    static forceinline void riscv_write_reg(rvvm_hart_t* vm, regid_t reg, sxlen_t data)
  • Interpreter load/stores are computing effective virtual address as xlen_t which is unsigned 32-bit in rv32. It's okay for bare or SV32 MMU, but apparently rv32 U-mode uses an SV39/48 MMU so the upper VA half is not properly sign-extended. Either riscv_mmu should manually sign-extend rv32 addresses on SV39, or switch to sxlen_t addresses in interpreter and making Bare/SV32 modes work with this.
  • The CSR subsystem is likely broken. Applying a 32-bit CSR mask when running in rv32 mode allows to see buildroot MOTD, but most other userspace parts are still crashing.
@LekKit LekKit added bug Something isn't working investigating Working to determine what's going on labels Mar 29, 2024
@LekKit
Copy link
Owner Author

LekKit commented Mar 29, 2024

No CSR access is happening in rv32 umode (at least not before it crashes). I'd assume kernel never uses smth like rv32 S-mode so S-mode CSRs can't be messed up this way.

SV32 MMU doesn't seem to be used at all.

@LekKit
Copy link
Owner Author

LekKit commented Mar 29, 2024

Sign-extending or zero-extending rv32 VA pointers in SV39/SV48 translation doesn't help.

The kernel always faults on some magical address 0x3803b0af8. Perhaps this could lead debugging somewhere.

@LekKit
Copy link
Owner Author

LekKit commented Mar 31, 2024

Crashes are caused by riscv_handle_irqs() in RV32 U-Mode. Using a workaround that disables IRQs when hart is in rv32 mode & disabling RVJIT allows RV32 Buildroot to boot properly on RV64 kernel.

This is due to invalid IRQ mask computed for origin priv_mode and not the one we are switched into on IRQ.

@LekKit
Copy link
Owner Author

LekKit commented Mar 31, 2024

Fixed the interpreter side in 8aef67b

@LekKit
Copy link
Owner Author

LekKit commented Mar 31, 2024

RV32 U-Mode somewhat works with JIT since e108450. Note it still isn't spec conformant (RVJIT should sign-extend 32-bit results to 64-bit register, but doesn't now).
Performance is pretty low since the JIT cache is flushed on each XLEN change.

@LekKit LekKit added inefficiency Better implementation is desired and removed investigating Working to determine what's going on labels Apr 1, 2024
@LekKit LekKit changed the title RV32 U-Mode is broken RV32 U-Mode JIT is nonconformant and slow on ctx switch Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working inefficiency Better implementation is desired
Projects
None yet
Development

No branches or pull requests

1 participant