Fuzzing fails when using LightSSS

[Gloria-cpu]

您好，

我想问一下difftest的master的分支是否支持xfuzz以及snapshot等功能？

[poemonsense]

xfuzz is supported by implementing the required interfaces for simulation and coverage-feedback. As shown in the README of xfuzz, DiffTest is supported.

Snapshots (LightSSS) are supported as shown in the README of LightSSS.

[Gloria-cpu]

Hi poemonsense,

Thanks for your previous response. I successfully deployed xfuzz based on rocketchip (DUT) and spike (REF). The spike version I used was downloaded from this repository. However, when I enabled the snapshot functionality, I encountered the following error: difftest_memcpy with DIFFTEST_TO_DUT is not supported yet.

Upon further investigation, I found that the spike implementation in riscv-isa-sim/difftest/difftest.cc does not currently support the difftest_memcpy operation with DIFFTEST_TO_DUT. Is this issue related to the spike version I’m using, or is it something else that needs to be addressed?

[poemonsense]

What do you mean by "enabled the snapshot functionality"?

LightSSS does not require explicit enable.

[Gloria-cpu]

To clarify, here’s what I did:

I compiled difftest using the following command:

make emu REF=$SPIKE_HOME/difftest/build/riscv64-spike-so XFUZZ=1 LLVM_COVER=1 FIRRTL_COVER=mux,control,line,toggle,ready_valid EMU_TRACE=1 EMU_SNAPSHOT=1 -j20

Then, I ran xfuzz with:

./build/fuzzer -f --max-runs 100 --corpus-input $CORPUS -- --max-cycles 10000 --enable-fork --fork-interval 1

The configuration --enable-fork --fork-interval 1 is intended to enable the snapshot functionality. However, shortly after starting the run, the error difftest_memcpy with DIFFTEST_TO_DUT is not supported yet occurs.

[poemonsense]

--enable-fork is for in-memory snapshots (LightSSS). As shown in the README of DiffTest, it does not require any compilation flag (The plugin LightSSS is by default included at compilation time and should be manually enabled during simulation time using --enable-fork.)

EMU_SNAPSHOT is for on-disk snapshots. It requires the support from REF (you may refer to our paper as shown in the README for more details), including copying registers, memories, etc, to DiffTest. It is currently supported by NEMU and not supported by Spike.

[Gloria-cpu]

I see, that makes sense.

I’ve tried multiple versions of NEMU and passed them into the make emu REF=*** command, including directly obtaining riscv64-nemu-interpreter-so from ready-to-run and also compiling it from source using this repository. However, all of these attempts encountered various issues, such as missing function definitions or runtime errors, which made it difficult to get NEMU working properly with RocketChip and xfuzz.

Am I missing any critical steps in the integration process?

[poemonsense]

NEMU does not support rocket configs. We only provide Rocket REF in Spike.

For NutShell or XiangShan, please use the NEMU so in the corresponding repo. For example, https://github.com/OSCPU/NutShell/blob/master/ready-to-run/riscv64-nemu-interpreter-so for NutShell.

To compile them from source, use make riscv64-nutshell-ref_defconfig or make riscv64-xs-ref_defconfig. See https://github.com/OpenXiangShan/NEMU/tree/master/configs for full list of configs.

If you are facing compiling issues, please list here and we can see why.

[Gloria-cpu]

Thank you for your reply.

Our goal is to have DiffTest support both xfuzz and snapshot functionalities simultaneously. Could you provide some guidance or suggestions on this? For example, what would be the recommended DUT and REF configurations to achieve this?

[poemonsense]

This is not a fuzzing issue.

LightSSS is supported by any REFs.

Snapshot is supported by NEMU only.

Diff with any REF and any DUT is supported by DiffTest. You can diff NEMU with Rocket. However, Spike can cosim with rocket without reporting false bugs. For other pairs, such as NEMU vs any DUT, Spike vs NutShell/XiangShan, there are a lot of bugs in the DUT/REF.

[Gloria-cpu]

What is the difference between diff and cosim? Does diff refer to using the --diff parameter in difftest configuration, while cosim refers to the REF specified during compilation with make emu REF=***?

If that’s correct, does this mean that currently, the only stable cosim combination is Spike and Rocket, while other combinations have bugs? Additionally, are xfuzz and snapshot functionalities dependent on cosim?

[poemonsense]

What is the difference between diff and cosim? Does diff refer to using the --diff parameter in difftest configuration, while cosim refers to the REF specified during compilation with make emu REF=***?

If that’s correct, does this mean that currently, the only stable cosim combination is Spike and Rocket, while other combinations have bugs? Additionally, are xfuzz and snapshot functionalities dependent on cosim?

cosim and diff are the same.

REFs: NEMU, Spike. DUTs: Rocket, XiangShan, NutShell, etc. Any pair of REF and DUT is supported by DiffTest.

However, you may find a log of DUT RTL or REF bugs (not DiffTest bugs) when diff between like NEMU and Rocket. This is generally because their configurations are not aligned or their implementation differs.

xfuzz does not depend on diff/cosim. However, if you know what fuzzing is about, you will know that, without diff, you will never know when there is an interesting bug.

Snapshot does not depend on diff/cosim.

[Gloria-cpu]

I see, so any combination of REF and DUT is supported, but due to differences in design and configurations, it can cause various errors, like mismatches, when comparing DUT and REF.

In that case, could you share which DUT and REF combination your team uses when testing the snapshot functionality?

[poemonsense]

I see, so any combination of REF and DUT is supported, but due to differences in design and configurations, it can cause various errors, like mismatches, when comparing DUT and REF.

In that case, could you share which DUT and REF combination your team uses when testing the snapshot functionality?

Snapshot does not depend on diff/cosim.

Snapshot is supported by NEMU only. You may refer to the CI file on how we test it. It is worth noting it has been unused for a long time. We cannot confirm its correctness.

LightSSS is supported by any REF.

[Gloria-cpu]

I understand your points. Based on the CI file you provided, I’m currently using RocketChip (DUT) and Spike (REF) for cosim. I compile and run with the following commands:

make emu REF=$SPIKE_HOME/difftest/build/riscv64-spike-so XFUZZ=1 LLVM_COVER=1 FIRRTL_COVER=mux,control,line,toggle,ready_valid EMU_TRACE=1 EMU_THREAD=8 -j20

Then I run:

./build/fuzzer -f --coverage firrtl.toggle --max-runs 100 --corpus-input $CORPUS -v -- --max-cycles 10000 --enable-fork --fork-interval 1

This runs xfuzz and LightSSS together. However, it stops after just two simulations instead of running 100 times as expected. If I remove --enable-fork --fork-interval 1, it runs normally. Have you encountered this issue before?

[poemonsense]

This runs xfuzz and LightSSS together. However, it stops after just two simulations instead of running 100 times as expected. If I remove --enable-fork --fork-interval 1, it runs normally. Have you encountered this issue before?

This is probably a bug in the code. We will try to fix it.

[Gloria-cpu]

Thank you for looking into it. Please let me know if there’s any update.