Sail Crosscheck Testing with ACT Against Spike

[jjscheel]

Technical Group

Golden Model SIG

ratification-pkg

Technical Debt

Technical Liaison

Bill McSpadden

Task Category

SAIL model

Task Sub Category

• gcc
• binutils
• gdb
• intrinsics
• Java
• KVM
• ld
• llvm
• Linux kernel
• QEMU
• Spike

Ratification Target

3Q2023

Statement of Work (SOW)

Component names:
Sail (infrastructure)

Requirements:
In order to improve functional coverage of the RISC-V Sail model, the RISC-V ACTs need to be run against the model. Since the ACTs are not self-checking tests, another model needs to be checked against. We will use Spike to crosscheck the Sail model.

Configurations of the Sail model should be done with RISCV-Config.

RISCOF should be used for testing.

Deliverables:

1. github repo that contains documentation as to how to set up and run a Sail/Spike crosschecking environment.
2. A sandbox containing examples of running the crosscheck for different configurations of the models.

Acceptance Criteria:

1. Tests run and results are identical between Sail and Spike
2. Accepted PR into the Sail community and any additional projects identified during work

Projected timeframe: (best guess date)
6 person months of work from start date

SOW Signoffs: (delete those not needed)

• Development partner sign-off
• ACT SIG sign-off
• Golden Model SIG sign-off

Waiver

• Freeze
• Ratification

Pull Request Details

• Sail PR

[jjscheel]

@junambi also expressed interest and/or a willingness to help here. @UmerShahidengr, can you and Nambi see if the work can be divided?

[Abdulwadoodd]

I have implemented the methodology in CI that runs ACTs on riscv-sail model and test it against spike, using RISCOF. CI runs tests for all configurations of architecture sizes (rv64i_m, rv32i_m, rv32e). After CI completes the test execution, the artifacts containing the test report of all three categories is uploaded on GitHub space
All the extensions supported in sail-riscv are running under the CI and are passing by matching signature with that of spike.

The implementation is currently on my forked repo of sail and (CI results) available here for the initial review for @billmcspadden-riscv. If this looks good to you and does not require major changes, I'm happy to make a PR official sail-riscv repository.

[UmerShahidengr]

Update ⇾ March 5th, 2024
@Abdulwadoodd is waiting for the initial review, if @billmcspadden-riscv is happy with his implementation, then we can add its PR and complete this SoW.

[jjscheel]

Thank, @UmerShahidengr.

@Abdulwadoodd, I hear your demo is quite impressive. Now begins the "fun" work of finding an acceptable solution for the community. @billmcspadden-riscv will help you with this. It will take time, so please be patient.

@gsterlin, FYI. This work plays into the broader Compatibility Testing delivery plan we trying to develop.

[Abdulwadoodd]

Thanks @jjscheel,
The next step is to add a Readme Document for the sandbox, as instructed by @billmcspadden-riscv after the demo. After that, I'll raise a sail PR for the golden model community review.

[UmerShahidengr]

@Abdulwadoodd can you please give any update on it?

[UmerShahidengr]

@Abdulwadoodd has been busy in other assignments, he will update on this issue soon

[jjscheel]

@Abdulwadoodd, if you get a chance, I'd appreciate an update on Monday or Tuesday. Thanks!

[Abdulwadoodd]

@jjscheel sorry for the delay.
I have raised a PR (riscv/sail-riscv#481) that runs the ACTs in CI, cross-checks sail and spike, generates reports and uploads the reports to GitHub workspace. suggestions/changes/questions are welcome on this PR.

Currently, CI uses arch-tests release of 3.8.10 while the later releases (including current 3.8.20) is a bit broken for RISCOF. The PR to fix this issue is underway (riscv-non-isa/riscv-arch-test#464)

[jjscheel]

Thanks for the update, @Abdulwadoodd!

[UmerShahidengr]

Update June 11th, 2024:
The PR by @Abdulwadoodd is still in review. Work is completed on our side

[jjscheel]

@billmcspadden-riscv, as discussed in yesterday's meeting, please review this SOW and help us understand what's next in the context of Sail CI testing. An update here or in a future DevPartners meeting would be helpful when ready.