Add devcontainer #576
Open
Add devcontainer #576
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pbreuer
reviewed
Jan 6, 2025
There was a problem hiding this comment.
OK, first of all you need to list the four files in this (patch?) and say what they are and what they are for.
The shell file is a one-liner, does something with git, and therefore I have no idea what it is for or why anyone would want it. Looks like it sets some parameters. As a shell script it is missing the bang line at top (#! /bin/sh). It is missing an author claim so people know who to blame and ask (blah, blah, licence, blah) and the biggie is teh lack of an explanation of what it is/is for. To me it looks dangerous! It mentions "/workspaces" which does not exist on my machine, and is a root directory! Oww!
There is a short json file, again I have no idea what it is for, and it needs to say what it is, who made it, and why. It mentions the shell file so it is presumably some link between something and something ... likely a description of the shell file for some gui, as it mentions an X display number (that's not a number of a display I am working on!)
There is a "docker file" which seems to contain four commands to run in sequence in order to install support infrastructure for building something (what?). Seem to be commands with apt for ubuntu to pull some ordinary build stuff, then a one liner to pull with wget a shell script that maybe is some decorative setup for a zsh running "in" docker, if that has a gui, as I suppose it does.
WHAT is all this for? It seems aimed at a gui, which is something I would not use. The idea in open source is that you engage with source code, not distance yourself from it via a gui!
FINALLY .. there is a Makefile! Hurray. This is the ONLY useful item. You need to describe what it is for, who is the author, etc. It should set parameters at the beginning and follow with rules later. Everything is all mixed up here (aka "unreadable"!).
The first thing it does is set PATH, which is completely unacceptable, because that must be configurable to a users taste. I imagine docker has some fixed paths and these are they, largely.
At this point you should ask yourself what good this is doing you. People can run makefiles on their own! They just type "make"! WHY would anyone need to run it inside docker? Please explain! In the first place one must not install stuff on one's machine that is not under the control of the system's installation and configuration manager, and this is doing just that, apparently. It is entirely unacceptable! Unless ... docker can figure out what the distro is, and build packages for it on the fly, and install them. Can it?
That would be extremely unreliable - its decisions would inevitably conflict with the distro's packagers own, so not likely.
Sigh .. you need to determine what this is FOR. There is no difficulty in downloading stuff from the riscv area and putting in all in some directory in /usr/local/src, for example. The problem then is that all that stuff has been developed by people who seem to have little idea how to code, or how to develop code that is installable and maintainable (two sides of the same coin!). You need to HELP solve that, so you should be doing something that adds the intelligence and knowhow that they have lacked. You need to have a makefile that build what they want to build, yes, but WITHOUT messing up your machine with all these extraneous local extra packages etc. You will say that docker manages its own area, but I at least want no area that is not under the control of the package manager on the machine. The question is how to integrate with that.
The simplest method is to mount a transparant file system over the real file system in a sandbox, build whatever you need with docker in there, install it into /usr/local in the transparant system, then take a tar of the binary installation, move the tar out of the transparant file system and step out of the sandbox, destroy the sandbox and all the docker stuff, and then convert the tar to a distro package using alien or whatever you prefer, then install with the distro package manager. If you like you can do without the transparant file system mount, there are plenty of applications that replace "install" with a script that logs where things go and you can make a list of things to tar in /usr/local from that.
But the problem with that and the similar method via docker that it looks like you are proposing is that you are not adding any intelligence or knowledge, so it's got no content. I don't think you will know what the various things you are installing do or where they put things or why or how they make them. That is the knowledge that one wants added via a makefile or other tool. It should organise all that with purpose and design, and that can't be done without knowing.
But to look at the makefile you have supplied, exeunt the disorganisation, it says it runs some setups for a variety of things. You need to explain what is meant by "setup", and what it does ... I have no idea! Please do explain.
You also need to explain what the things are it does setup for, and allow someone to modify all this intelligently with that information to hand via appropriate comment.
You then generate some sense of what of those things is installed via running "command", which is a Ubuntu-only thing and WILL NOT WORK anywhere else, so it is useless. Didn't you just download all these things via docker anyway? So why are you testing? Or is this stuff that docker didn't download and is about to become a victim of the stuff that was downloaded? I suppose so! You do know that "./configure" is generally used to discover what is available, and that will generate a Makefile set up accordingly? You likely don't want the Makefile itself to do discovery.
Actually, if the things whose presence is tested for aren't available, the Makefile seems to try to run "install" on some things using pip. Well! that's what the instructions say to do on the riscv site! Is the Makefile only intended to save people the trouble of reading?
So far the only thing that has happened is that docker has build pip (I suppose) and pip is now building whatever the various things you want installed are, somehow, and I at least am no wiser as to what is going where or what it is for.
Please put lots of writing in to explain what the users choices are and what the consequences of those choices are.
Actually, as far as I can see, no actual building is done? Just whatever they are are got via curl or pip (I don't know what pip does, but I imagine it gets stuff from remote python repositories). Why is that a help to anyone? I can do that!
What one needs is help in building whatever those things are, and help in choosing whether one wants them or not, and help in configuring them to go in the right places and integrating them into the installed system. And that should be done without adding to the system, or modifying it in any way. Provide explanations that allow for informed choice and leave that to the user.
It doesn't help me. Choose one thing to help install, explain what it does, figure out how it can be built in a standard fashion without whatever weirdism the author has misconceived, and do it. For bonus points record where it put things, build a binary tar out of it and convert to the distro package with alien and install that post-hoc.
How about telling ME what those ref.elf.dump files are supposed to be?
|
Thank you very much for your reply. I have never had such a detailed discussion, I need some time to carefully consider these issues |
Actually I don't quite understand what you mean by if you run tree riscof-plugins/rv64_cmo/riscof_work/rv64i_m/CMO/src/cbo.zero-01.S
riscof-plugins/rv64_cmo/riscof_work/rv64i_m/CMO/src/cbo.zero-01.S
├── cbo.zero-01.log
├── coverage.rpt
├── ref.cgf
├── ref.disass
├── ref.elf
├── Reference-sail_c_simulator.signature
└── ref.mdif you run |
|
Hi, thanks for the reply .. let me clarify:
On Mon, 06 Jan 2025 08:16:32 -0800 MingZhu Yan ***@***.***> wrote:
Actually, I don't quite understand what you mean by `ref.elf.dump`, because if you run `riscof coverage`
I am looking at the wally code for a riscv processor ( https://github.com/openhwgroup/cvw ) and there they have a testbench.sv in which it says:
```
if (riscofTest) begin
elffilename = { ...
, "/ref/ref.elf"
};
memfilename = { ...
, "/ref/ref.elf.memfile"
};
ProgramAddrMapFile = { ...
, "/ref/ref.elf.objdump.addr"
};
ProgramLabelMapFile = { ...
, "/ref/ref.elf.objdump.lab"
};
```
and those are used later on in the code as follows:
```
if(elffilename == "buildroot")
...
if(elffilename != "none") begin
...
memFile = $fopen(memfilename, "rb");
...
uncoreMemFile = $fopen(memfilename, "r");
...
$readmemh(memfilename, dut.uncore.ram.memory.RAM);
...
updateProgramAddrLabelArray( ProgramAddrMapFile,...
...
ProgramAddrMapFile(ProgramAddrMapFile)
...
ProgramAddrMapFP = $fopen(ProgramAddrMapFile, "r");
...
ProgramLabelMapFP = $fopen(ProgramLabelMapFile, "r");
...
```
so THEY (the authors of wally) seem to think those files are needed. Therefore they must be generated, somehow.
Yes, I haven't been able to get any sense out of the authors of wally so far (:-). It might be that those I talk to don't know how the tests should be run and what they consist of, and aren't saying. I am operating on guesswork for way too much of this (the .objdump files are just output of disassembly with objdump -D, I know, and I suppose the other files must be the objdump output from particular sections of the disassembled elf file, but which and in
what format?) ...
So YOU are telling ne to run "riscof coverage". Why should I do that, with what objective, and what does it do, and/or is intended to do? Please do explain.
To explain my context, my overall intention is to see if a processor design consisting of a modified wally can execute some instructions individually correctly, but failing that unit test availability, seeing if it can pass the integrated test of running a loaded program in simulation would be ok ... preferably in verilator simulation). The
word "coverage" is not encouraging. "riscof" is neutral. The "risc" part is fine, the "of" part I don't know what to make of.
There are some .S files scattered around the wally tests subdirectory, and failing instructions as to what to do with them I have compiled them into elf files using cpp, riscv as and riscv ld. There were some header files I guessed were meant to provide macros, and some linker files I guessed were meant to be used which basically said to align sectors at 4KB boundaries and start at 0x800...00 in address space, and I supplied some parameters to the macros as seemed fit going by the names of the directories in the paths to the .S files. End result, I have ref.elf (and ref.elf.objdump) files.
you will get
```shell
No, I don't. I get error messages terminating with
``` ...
csr = re.sub('{\d*}','', csr)
/usr/local/lib/python3.12/dist-packages/riscv_config-3.18.3-py3.12.egg/riscv_config/warl.py:621: SyntaxWarning: invalid escape sequence '\s'
csr_in_dependencies = list(filter(re.compile(f'[a-zA-Z0-9_]*[:]*{csr}\s*$').match, dependencies))
INFO | ****** RISCOF: RISC-V Architectural Test Framework 1.25.3 *******
INFO | using riscv_isac version : 0.18.0
INFO | using riscv_config version : 3.18.3
Usage: riscof coverage [OPTIONS]
Try 'riscof coverage --help' for help.
```
Can you start from the beginning please and tell me WHAT riscof is, WHY I might like to use it, WHAT I should use it on, HOW to provide whatever it needs as input, and HOW to cause it to generate output and HOW to
interpret that output.
tree riscof-plugins/rv64_cmo/riscof_work/rv64i_m/CMO/src/cbo.zero-01.S
riscof-plugins/rv64_cmo/riscof_work/rv64i_m/CMO/src/cbo.zero-01.S
_________ cbo.zero-01.log
_________ coverage.rpt
_________ ref.cgf
_________ ref.disass
_________ ref.elf
_________ Reference-sail_c_simulator.signature
_________ ref.md
As I said, no I don't get that. I get what I put above.
```
if you run `riscof run` then
Again, why would I do that, with what objective, what does it do, what do I need in the way of prerequisites, how do I provide them, how do I get it to generate output, and how do I interpret the output?
```
tree riscof-plugins/rv64_cmo/riscof_work/rv64i_m/I/src/add-01.S
What is "tree"? Anyway, I get errors finishing:
``` /usr/local/lib/python3.12/dist-packages/riscv_config-3.18.3-py3.12.egg/riscv_config/warl.py:621: SyntaxWarning: invalid escape sequence '\s'
csr_in_dependencies = list(filter(re.compile(f'[a-zA-Z0-9_]*[:]*{csr}\s*$').match, dependencies))
INFO | ****** RISCOF: RISC-V Architectural Test Framework 1.25.3 *******
INFO | using riscv_isac version : 0.18.0
INFO | using riscv_config version : 3.18.3
Usage: riscof run [OPTIONS]
Try 'riscof run --help' for help.
```
riscof-plugins/rv64_cmo/riscof_work/rv64i_m/I/src/add-01.S
_________ dut
___ _________ DUT-spike.log
___ _________ DUT-spike.signature
___ _________ my.elf
_________ ref
_________ add-01.log
_________ ref.disass
_________ ref.elf
_________ Reference-sail_c_simulator.signature
```
I get what I showed above. But the basic questions remain. Why should I be running what you say to run, with what objective, etc?
(By the way, what is a "DUT"? I've seen that several places. I'm going to guess it's in-jargon for "the processor I want to test". How should that be specified to riscof, if that is the case? Do you want system verilog? Verilog? A simulation running in verilator? What?)
I really don't see how riscof do anything as you have described it, if it should be run as a standalone utility! Where are the inputs? Where are the outputs? It can't possibly read my mind to discover the first, I can't read its mind to discover the second!
Shouldn't it explain all that if one does riscof --help? Ah, it does. Trace info comes out, terminating:
``` ....
/usr/local/lib/python3.12/dist-packages/riscv_config-3.18.3-py3.12.egg/riscv_config/warl.py:621: SyntaxWarning: invalid escape sequence '\s'
csr_in_dependencies = list(filter(re.compile(f'[a-zA-Z0-9_]*[:]*{csr}\s*$').match, dependencies))
Usage: riscof [OPTIONS] COMMAND [ARGS]...
Options:
--version Show the version and exit.
-v, --verbose [info|error|debug]
Set verbose level
--help Show this message and exit.
Commands:
arch-test Setup and maintenance for Architectural TestSuite.
coverage Run the tests on DUT and reference and compare signatures
gendb Generate Database for the Suite.
run Run the tests on DUT and reference and compare signatures
setup Initiate Setup for riscof.
testlist Generate the test list for the given DUT and suite.
validateyaml Validate the Input YAMLs using riscv-config.
```
So what is a "setup", what is it setting up for, and what is an "Architectural Test Suite" testing? How can one "setup" for an executable? Do you mean something like "generate standard reference files for use in a subsequent run testing something for real"? Why doesn't it generate those on need if it doesn't have them, or ask? Shouldn't that be the business of some Makefile, anyway? One should not combine different functionalities (ancient rule of code construction!).
What "tests" is it talking about in "coverage" and why is that called "coverage" and not "test"? Is that meant to be a particular kind of set of tests, perhaps, checking implementation and correctness of a range of
instructions and features? What exactly is it looking at? What is this "DUT"? How do I define it in a format acceptable to the utility?
And in "gendb", what "database"? Why would one want a database, and of what? If of results, that surely is my business, not a testing apparatus!
OK, so "run" makes sense, but where is this "DUT" and how do I define it and how do I tell it to look at it?
Hmm .. perhaps riscof is actually a testbench.sv kind of thing, that has to be compiled together with the actual processor code in order to supply it with inputs and read its outputs? Perhaps wally's testbench.sv is actually a part taken from riscof?
I don't know, but please do provide such explanations.
I would have expected that a testing aparatus would have wanted to talk to a standard debugging port on a simulation in verilator, or something like that, not be compiled together with it! Am I guessing wrong now in thinking one might be meant to compile tester with testee?
How do you specify how they are to be linked together? Has this sort of grown by abstraction from a number of examples over the years, and been called a "framework" as a result?
Answers very much appreciated, and my hearty thanks for your replies so far.
PTB
PS. A lot of that Warning trace output from python is because risc_config-3.18.3 specifes pyyaml 5.2 exactly as a requirement, but pyyaml 6.0.3 is installed in debian (`Name: PyYAML Version: 6.0.2 Summary: YAML parser and emitter for Python Home-page: https://pyyaml.org/), and cannot be regressed as dozens of things depend on it. I changed the requirement to >=5.2 from =5.2 simply to allow it to run, hoping it would not be quite disastrous and let me see what is needed - I do not use python for doing anything and do not intend to, I need you to TELL me what you want and I'll make it happen.
|
|
I can answer some of that.
Coverage mode executes a test, parses the log file, and compares it with a
coverage file, building a histogram of the which, (and how
often) coverpoints defined in the coverage file are seen.
Run mode (likely) executes tests twice, once with a DUT (e.g. Wally) and
once with a reference model (Sail/Spike, etc) and compares the signature
values that it extracts from a memory dump.
…On Mon, Jan 6, 2025 at 11:23 AM pbreuer ***@***.***> wrote:
Hi, thanks for the reply .. let me clarify:
On Mon, 06 Jan 2025 08:16:32 -0800
MingZhu Yan ***@***.***> wrote:
> Actually, I don't quite understand what you mean by `ref.elf.dump`,
because if you run `riscof coverage`
I am looking at the wally code for a riscv processor
https://github.com/openhwgroup/cvw
and there they have a testbench.sv in which it says:
if (riscofTest) begin
elffilename = { ...
, "/ref/ref.elf"
};
memfilename = { ...
, "/ref/ref.elf.memfile"
};
ProgramAddrMapFile = { ...
, "/ref/ref.elf.objdump.addr"
};
ProgramLabelMapFile = { ...
, "/ref/ref.elf.objdump.lab"
};
and those are used later on in the code as follows:
if(elffilename == "buildroot")
...
if(elffilename != "none") begin
...
memFile = $fopen(memfilename, "rb");
...
uncoreMemFile = $fopen(memfilename, "r");
...
$readmemh(memfilename, dut.uncore.ram.memory.RAM);
...
updateProgramAddrLabelArray( ProgramAddrMapFile,...
...
ProgramAddrMapFile(ProgramAddrMapFile)
...
ProgramAddrMapFP = $fopen(ProgramAddrMapFile, "r");
...
ProgramLabelMapFP = $fopen(ProgramLabelMapFile, "r");
...
so THEY (the authors of wally) seem to think those files are needed.
Therefore they must be generated, somehow.
Yes, I haven't been able to get any sense out of the authors of wally so
far (:-). It might be that those I talk to don't know how the tests
should be run and what they consist of, and aren't saying. I am
operating on guesswork for way too much of this (.objdump is just output
of a disassembly with objdump -D, I know, and I suppose the others must
be particular sections of the disassembled elf file, but which and in
what format?) ...
So YOU are telling ne to run "riscof coverage". Why should I do that,
with what objective, and what does it do, and/or is intended to do.
Please explain.
To explain my context, my overall intention is to see if a processor
design consisting of a modified wally can execute some instructions
individually correctly, but failing that unit test availability, seeing
if it can pass the integrated test of running a loaded program in
simulation would be ok ... preferably in verilator simulation). The
word "coverage" is not encouraging. "riscof" is neutral. The "risc"
part is fine, the "of" part I don't know what to make of.
There are some .S files scattered around the wally tests subdirectory,
and failing instructions as to what to do with them I have compiled
them into elf files using cpp, riscv as and ld. There were some header
files I guessed were meant to provide macros, and some linker files
I guessed were meant to be used which basically said to align sectors at
4KB boundaries and start at 0x800...00 in address space, and I supplied
some parameters to the macros as seemed fit going by the names of the
directories in the paths to the .S files.
End result, I have ref.elf (and ref.elf.objdump) files.
> you will get
> ```shell
No, I don't. I get error messages terminating with
...
csr = re.sub('{\d*}','', csr)
/usr/local/lib/python3.12/dist-packages/riscv_config-3.18.3-py3.12.egg/riscv_config/warl.py:621:
SyntaxWarning: invalid escape sequence '\s'
csr_in_dependencies =
list(filter(re.compile(f'[a-zA-Z0-9_]*[:]*{csr}\s*$').match, dependencies))
INFO | ****** RISCOF: RISC-V Architectural Test Framework 1.25.3 *******
INFO | using riscv_isac version : 0.18.0
INFO | using riscv_config version : 3.18.3
Usage: riscof coverage [OPTIONS]
Try 'riscof coverage --help' for help.
Can you start from the beginning please and tell me WHAT riscof is, WHY
I might like to use it, WHAT I should use it on, HOW to provide whatever
it needs as input, and HOW to cause it to generate output and HOW to
interpret that output.
> tree riscof-plugins/rv64_cmo/riscof_work/rv64i_m/CMO/src/cbo.zero-01.S
> riscof-plugins/rv64_cmo/riscof_work/rv64i_m/CMO/src/cbo.zero-01.S
> _________ cbo.zero-01.log
> _________ coverage.rpt
> _________ ref.cgf
> _________ ref.disass
> _________ ref.elf
> _________ Reference-sail_c_simulator.signature
> _________ ref.md
As I said, no I don't get that. I get what I put above.
> ```
>
> if you run `riscof run` then
Again, why would I do that, with what objective, what does it do, what
do I need in the way of prerequisites, how do I provide them, how do I
get it to generate output, and how do I interpret the output?
> ```
> tree riscof-plugins/rv64_cmo/riscof_work/rv64i_m/I/src/add-01.S
What is "tree"? Anyway, I get errors finishing:
/usr/local/lib/python3.12/dist-packages/riscv_config-3.18.3-py3.12.egg/riscv_config/warl.py:621:
SyntaxWarning: invalid escape sequence '\s'
csr_in_dependencies =
list(filter(re.compile(f'[a-zA-Z0-9_]*[:]*{csr}\s*$').match, dependencies))
INFO | ****** RISCOF: RISC-V Architectural Test Framework 1.25.3 *******
INFO | using riscv_isac version : 0.18.0
INFO | using riscv_config version : 3.18.3
Usage: riscof run [OPTIONS]
Try 'riscof run --help' for help.
> riscof-plugins/rv64_cmo/riscof_work/rv64i_m/I/src/add-01.S
> _________ dut
> ___ _________ DUT-spike.log
> ___ _________ DUT-spike.signature
> ___ _________ my.elf
> _________ ref
> _________ add-01.log
> _________ ref.disass
> _________ ref.elf
> _________ Reference-sail_c_simulator.signature
> ```
I get what I showed above. But the basic questions remain. Why
should I be running what you say to run, with what objective, etc?
(By the way, what is a "DUT"? I've seen that several places. I'm going
to guess it's in-jargon for "the processor I want to test". How should
that be specified to riscof, if that is the case? Do you want system
verilog? Verilog? A simulation running in verilator? What?)
I really don't see how riscof do anything as you have described it
should be run! Where are the inputs? Where are the outputs? It can't
possibly read my mind to discover the first, I can't read its mind to
discover the second!
Shouldn't it explain all that if one does riscof --help? Ah, it does. Trace
info comes out terminating:
....
/usr/local/lib/python3.12/dist-packages/riscv_config-3.18.3-py3.12.egg/riscv_config/warl.py:621:
SyntaxWarning: invalid escape sequence '\s'
csr_in_dependencies =
list(filter(re.compile(f'[a-zA-Z0-9_]*[:]*{csr}\s*$').match, dependencies))
Usage: riscof [OPTIONS] COMMAND [ARGS]...
Options:
--version Show the version and exit.
-v, --verbose [info|error|debug]
Set verbose level
--help Show this message and exit.
Commands:
arch-test Setup and maintenance for Architectural TestSuite.
coverage Run the tests on DUT and reference and compare signatures
gendb Generate Database for the Suite.
run Run the tests on DUT and reference and compare signatures
setup Initiate Setup for riscof.
testlist Generate the test list for the given DUT and suite.
validateyaml Validate the Input YAMLs using riscv-config.
So what is a "setup", what is it setting up for, and what is an
"Architectural Test Suite" testing? How can one "setup" for an
executable? Do you mean something like "generate standard reference
files for use in a subsequent run testing something for real"? Why
doesn't it generate those on need if it doesn't have them, or ask?
Shouldn't that be the business of some Makefile, anyway? One should
not combine functionalities (ancient rule of code construction!).
What "tests" is it talking about in "coverage" and why is that called
"coverage" and not "test"? Is that meant to be a particular kind of set
of tests, perhaps, checking implementation and correctness of a range of
instructions and features? What exactly is it looking at? What is this
"DUT"? How do I define it in a format acceptable to the utility?
And in "gendb", what "database"? Why would one want a database, and of
what?
If of results, that is my business, not a testing apparatus!
OK, so "run" makes sense, but where is this "DUT" and how do I define
it and how do I tell it to look at it?
Hmm .. perhaps riscof is actually a testbench.sv kind of thing, that
has to be compiled together with the actual processor code in order
to supply it with inputs and read its outputs? Perhaps wally's
testbench.sv is actually a part taken from riscof?
I don't know, but please do provide such explanations.
I would have expected that a testing aparatus would have wanted to
talk to a standard debugging port on a simulation in verilator, or
something like that, not be compiled together with it! Am I guessing
wrong in thinking one is meant to compile tester with testee?
How do you specify how they are to be linked together? Has this sort
of grown like topsy by abstraction from a number of examples over
the years, and been called a "framework" as a result?
Answers very much appreciated, and my hearty thanks for your
replies so far.
PTB
—
Reply to this email directly, view it on GitHub
<#576 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHPXVJQ6RNBS7B7IAPZP7X32JLJ2HAVCNFSM6AAAAABTLBGXGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZTG43DCOJUHA>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
Thank you for that reply Allen, but you are saying here how "coverage" does
what it does, not what it does. What was the question.
On Mon, 06 Jan 2025 13:20:28 -0800 Allen Baum ***@***.***> wrote:
I can answer some of that.
Coverage mode executes a test, parses the log file, and compares it with a
coverage file, building a histogram of the which, (and how
often) coverpoints defined in the coverage file are seen.
So what does it do? Would I be correct in thinking it tries to check
how many instructions are working correctly, or something like that?
Coverage of what is meant?
(what does "execute a test" mean, what is this log file of, and what
is a "coverage file", what is a "coverpoint", etc. More WHATs! Not
hows - even though you are describing a HOW there).
Run mode (likely) executes tests twice, once with a DUT (e.g. Wally) and
Heh, again, you are saying HOW, not WHAT!
Would I be right in guessing that the aim of run mode is to check the
behavior on a certain test program code of the simulated processor is
roughly as expected? Where what "expected" means is "within some
parameters of a given exemplar"?
What is the test program? Is that something I define? What is the means
of telling riscof which program I want to run. There weren't any
arguments to the utility!
once with a reference model (Sail/Spike, etc) and compares the signature
When you say "reference model", are you talking about a simulation at
functional level, or timing level, or other such considerations?
I compiled Spike easily enough (sent a debian source package
specification to the author) but it seems to me to be user-level code
for a state machine, i.e. a single imperative loop executing
instructions one at a time and acting on 32 registers and some CSRs,
while talking to the operating system for execution load/store and
syscalls.
That's "not hard" as a programming task.
I also compiled Sail, and it seemed to be needed for riscof stuff, as
I recall, but I have no clear idea what it does or what it is needed
for. Perhaps to construct reference files that are referenced by riscof
:-)? That is a guess that would make sense, I think.
values that it extracts from a memory dump.
What is a "memory dump"? Do you mean that the "reference model" and
the present test run take snapshots of the state of the simulated
memory at certain intervals, and those are what will be compared?
I suspect you are talking about what is called a "trace", the sequence
of interactions with memory, consisting of load and store instructions
executed and their ordering/timing.
Thanks for that! Any more on the WHAT?
Regards
PTB
|
|
It is primarily used for test development. If you're testing that the "add"
instruction works, you want to do more than tests 1+1=2;
you want to make sure that it does the right thing if either of the sources
is the same as the destination that,
or that if works with any register as either source or destination
(including X0), that the result is available to the next instruction,
that it never traps, that carries propagate correctly, overflows simply
wrap around, etc., etc.
The coverage file defines each of those cases, and coverage mode ensures
that the tests exercise all of the cases.
IF the coverage report shows that some particular case has not been
exercised, then
the test developer has to go back and get one written or generated.
…On Mon, Jan 6, 2025 at 1:57 PM pbreuer ***@***.***> wrote:
Thank you for that reply Allen, but you are saying here how "coverage" does
what it does, not what it does. What was the question.
On Mon, 06 Jan 2025 13:20:28 -0800 Allen Baum ***@***.***> wrote:
> I can answer some of that.
> Coverage mode executes a test, parses the log file, and compares it with
a
> coverage file, building a histogram of the which, (and how
> often) coverpoints defined in the coverage file are seen.
So what does it do? Would I be correct in thinking it tries to check
how many instructions are working correctly, or something like that?
Coverage of what is meant?
(what does "execute a test" mean, what is this log file of, and what
is a "coverage file", what is a "coverpoint", etc. More WHATs! Not
hows - even though you are describing a HOW there).
> Run mode (likely) executes tests twice, once with a DUT (e.g. Wally) and
Heh, again, you are saying HOW, not WHAT!
Would I be right in guessing that the aim of run mode is to check the
behavior on a certain test program code of the simulated processor is
roughly as expected? Where what "expected" means is "within some
parameters of a given exemplar"?
What is the test program? Is that something I define? What is the means
of telling riscof which program I want to run. There weren't any
arguments to the utility!
> once with a reference model (Sail/Spike, etc) and compares the signature
When you say "reference model", are you talking about a simulation at
functional level, or timing level, or other such considerations?
I compiled Spike easily enough (sent a debian source package
specification to the author) but it seems to me to be user-level code
for a state machine, i.e. a single imperative loop executing
instructions one at a time and acting on 32 registers and some CSRs,
while talking to the operating system for execution load/store and
syscalls.
That's "not hard" as a programming task.
I also compiled Sail, and it seemed to be needed for riscof stuff, as
I recall, but I have no clear idea what it does or what it is needed
for. Perhaps to construct reference files that are referenced by riscof
:-)? That is a guess that would make sense, I think.
> values that it extracts from a memory dump.
What is a "memory dump"? Do you mean that the "reference model" and
the present test run take snapshots of the state of the simulated
memory at certain intervals, and those are what will be compared?
I suspect you are talking about what is called a "trace", the sequence
of interactions with memory, consisting of load and store instructions
executed and their ordering/timing.
Thanks for that! Any more on the WHAT?
Regards
PTB
—
Reply to this email directly, view it on GitHub
<#576 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHPXVJV5ULGZQ7ILBC5IHL32JL353AVCNFSM6AAAAABTLBGXGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZTHE4TQNRVGM>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Ah, and ACT tests (are supposed to) ensure that any state updated by an
instruction is exposed by explicitly storing into a region of memory set up
to be the signature region.
Those memory regions are transferred to a file at the completion of a test
and compared to the signature region of the reference model.
This is different from how I understand Wally does it:
a cycle by cycle comparison of internal DUT state to internal reference
model state that is exposed through the RVVI interface.
There are pros and cons; each has issues.
On Mon, Jan 6, 2025 at 3:04 PM Allen Baum ***@***.***>
wrote:
… It is primarily used for test development. If you're testing that the
"add" instruction works, you want to do more than tests 1+1=2;
you want to make sure that it does the right thing if either of the
sources is the same as the destination that,
or that if works with any register as either source or destination
(including X0), that the result is available to the next instruction,
that it never traps, that carries propagate correctly, overflows simply
wrap around, etc., etc.
The coverage file defines each of those cases, and coverage mode ensures
that the tests exercise all of the cases.
IF the coverage report shows that some particular case has not been
exercised, then
the test developer has to go back and get one written or generated.
On Mon, Jan 6, 2025 at 1:57 PM pbreuer ***@***.***> wrote:
> Thank you for that reply Allen, but you are saying here how "coverage"
> does
> what it does, not what it does. What was the question.
>
> On Mon, 06 Jan 2025 13:20:28 -0800 Allen Baum ***@***.***> wrote:
> > I can answer some of that.
> > Coverage mode executes a test, parses the log file, and compares it
> with a
> > coverage file, building a histogram of the which, (and how
> > often) coverpoints defined in the coverage file are seen.
>
> So what does it do? Would I be correct in thinking it tries to check
> how many instructions are working correctly, or something like that?
> Coverage of what is meant?
>
> (what does "execute a test" mean, what is this log file of, and what
> is a "coverage file", what is a "coverpoint", etc. More WHATs! Not
> hows - even though you are describing a HOW there).
>
>
> > Run mode (likely) executes tests twice, once with a DUT (e.g. Wally) and
>
> Heh, again, you are saying HOW, not WHAT!
>
> Would I be right in guessing that the aim of run mode is to check the
> behavior on a certain test program code of the simulated processor is
> roughly as expected? Where what "expected" means is "within some
> parameters of a given exemplar"?
>
> What is the test program? Is that something I define? What is the means
> of telling riscof which program I want to run. There weren't any
> arguments to the utility!
>
> > once with a reference model (Sail/Spike, etc) and compares the signature
>
> When you say "reference model", are you talking about a simulation at
> functional level, or timing level, or other such considerations?
>
> I compiled Spike easily enough (sent a debian source package
> specification to the author) but it seems to me to be user-level code
> for a state machine, i.e. a single imperative loop executing
> instructions one at a time and acting on 32 registers and some CSRs,
> while talking to the operating system for execution load/store and
> syscalls.
>
> That's "not hard" as a programming task.
>
> I also compiled Sail, and it seemed to be needed for riscof stuff, as
> I recall, but I have no clear idea what it does or what it is needed
> for. Perhaps to construct reference files that are referenced by riscof
> :-)? That is a guess that would make sense, I think.
>
> > values that it extracts from a memory dump.
>
> What is a "memory dump"? Do you mean that the "reference model" and
> the present test run take snapshots of the state of the simulated
> memory at certain intervals, and those are what will be compared?
>
> I suspect you are talking about what is called a "trace", the sequence
> of interactions with memory, consisting of load and store instructions
> executed and their ordering/timing.
>
> Thanks for that! Any more on the WHAT?
>
> Regards
>
> PTB
>
> —
> Reply to this email directly, view it on GitHub
> <#576 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHPXVJV5ULGZQ7ILBC5IHL32JL353AVCNFSM6AAAAABTLBGXGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZTHE4TQNRVGM>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
|
|
Hello Alan
You are saying that by coverage is meant checking a representative set of instances of the actual functionality of a
particular single instruction (or sequences?) for conformance with the specification of what it should do according to the standard.
OK (let's forget that it might not work on odd wednesdays :-).
What kind of thing is the (a) specification of those test instances, what kind of thing is (b) the test that is carried out, and what kind of thing are the (c) observations that are checked against specification?
(a) I presume the specification is concrete: a vector of before and after pairs (r,v) of register indices and the values for them, plus a particular instruction to execute.
But perhaps a time-wise sequence may be given rather than just before/after?
(b) I suppose the test consists of a program that sets up the specified before values, runs the instruction, then checks the after values. It will be or could be generated by riscof, and I hope it will be output, so that
it can be run on a different simulator. Is it generated, or is it supplied?
But perhaps the test consists of forcing certain signal values on one clock, and checking them on subsequent clocks, via a closely coupled connection to actual simulation within riscof in real time. That is
gate/signal level testing, rather than functional testing. It is not that, I think? Yes? No?
(c) I suppose the observations are of the concrete final values of registers and so on or possibly the values taken over several cycles, especially if caches and things like that are also checked. I don't know if a tolerance can be given or not (think FP, the ieee standards allow many different answers).
I now need to know what kind of architectural/system arrangement the riscof apparatus has.
The coverage file defines each of those cases, and coverage mode ensures
that the tests exercise all of the cases.
I think you're attributing some magic here that is .. difficult. Are you saying that the user is not responsible for defining the test vectors, but that they are built in to or generated by riscof, and that you believe they are complete with respect to the functionality in some abstract sense?
It can't be concretely complete: testing the right answer to a+b would require checking every possible instance, which is way more than electrons in the universe; it must be that an abstraction previously developed on a
formal abstract state machine model is tested via particular randomly chosen concrete instances for each abstract test case - is that it? Riscof has been preprogrammed with an abstraction for each instruction? And it is interpreted one way for checking and another way for generating test instances?
Actually, you made it sound like a game in which a userdefines the concrete test instances but does not themself know the abstract model riscof is using :-). That is, riscof does not generate the test instances, the user does.
It is possible not to know the abstract model. There are choices in the abstraction over how to slice up the space of concrete instances .. for example are you going to separate off when the sign bit is high? Low? When the middle bit is high? Low? Or assume it makes no difference and regard all those possibilities as being inside the same abstract state?
IF the coverage report shows that some particular case has not been
exercised, then
the test developer has to go back and get one written or generated.
It sounds like you think there is such a thing as a complete set of tests of concrete functionality ... there isn't! I guess what is meant is that the apparatus has a definition of the finitely many abstract states (the abstract cases, defined as logical conditions) and it checks that all those have been exercised once at least by the actual test vectors supplied. That's nontrivial in the worst case, espeically if timed logic is involved ... never mind.
Can it tell the difference between an abstract case being not covered because the answer is wrong, and being not covered because inputs to trigger it were never supplied? It should, for a one-shot change of states specified in propositional logic ..
The problem would be those instructions that require interactions over time with something else, like memory load and store with certain flags set, or just in certain conditions (e.g. close-packed write sequences).
But I think I get the picture of the WHAT it is FOR.
Now there's the question of WHAT it IS :-).
My impression is that it is a standalone utility, it does not engage in any real time interaction or monitoring with a simulation of the concrete machine under test, neither close- nor far-coupled programmatically,
it's just uncoupled: one shows it the results from a set of simulations (or one simulation, many test vectors?) on
something else, and it either checks that against results from a standard model that either it contains within itself, or checks against results that have been provided to it that have been generated on an external standard model.
The latter, I think.
That would be what one would call an "offline" testing, I think? It would be a sensible division of concerns programmatically, but I suspect that users would like the expected results to be generated internally on the fly, so it
may have a standards compliant simuator inside despite my ideas of division of concerns!
Can you clarify that for me, do you think?
Thank you very much. This is extremely helpful!
If things are like that I need to ask next about input and output formats, and where.
If not, something else ...
Regards and thanks again!
PTB
|
|
On Mon, 06 Jan 2025 15:11:06 -0800 Allen Baum ***@***.***> wrote:
Ah, and ACT tests (are supposed to) ensure that any state updated by an
instruction is exposed by explicitly storing into a region of memory set up
I'm afraid that's not meaningful to me! I think you're saying I have to
use loads and stores in test vectors that are aimed from/to addresses in
a particular predefined small region of address space?
I don't see why .. trace information (the sequence of reads/writes seen
and which cell(s) flipped from what to what, when) should be collected,
not images of the whole region.
Shrug. But if you say so! I go by your word!
to be the signature region.
Those memory regions are transferred to a file at the completion of a test
You do say so :-).
This is different from how I understand Wally does it:
a cycle by cycle comparison of internal DUT state to internal reference
model state that is exposed through the RVVI interface.
Can we perhaps move on to the concrete issue of what files I need and
how to make them in order to use the section of the wally testbench.sv
code marked "if (riscofTest) begin"!
Resting on the above discussion, do you think these files are going to
consist of
1: a program _generated_ by riscof called foo.S and compiled to
ref.elf? (e.g. WALLY-ADD.S)
2: an INITIAL memory image of that "signature space" generated
by riscof and called ref.elf.memfile . Yes, it's read into
memory: memFile = $fopen(memfilename, "rb") [opened READ]
$readmemh(memfilename, dut.uncore.ram.memory.RAM, ..)
3: a file I have no idea about called internally ProgramAddrMapFile
and externally ref.elf.objdump.addr .
4: another file I have no idea about called ProgramLabelMapFile
called externally ref.elf.objdump.lab.
I am going to guess and say yes, riscof generates the .S file, the rest
is the work of mankind. Yes?
Yes. Found the mechanism:
# Makefile for testbench to create .memfile, .objdump.addr, and .objdump.lab from an ELF
# ***@***.*** 3 July 2024
# SPDX-License-Identifier: Apache-2.0 WITH SHL-2.1
%.elf.memfile: %.elf
riscv64-unknown-elf-elf2hex --bit-width $(if $(findstring rv32,$*),32,64) --input $< --output $@
%.elf.objdump.addr: %.elf.objdump
extractFunctionRadix.sh $<
%.elf.objdump: %.elf
riscv64-unknown-elf-objdump -S -D $< > $@
OK, we're flying. Now I can at least run testbench in verilator.
I have absolutely no idea at this point if wally will generate some
files during a testbench run. If, as seems likely, it does, do you
think those will be files I have to give to riscof for it to do
comparisons on? Then I will have to ask about where and maybe even ...
how!
As to why I cannot get asnwers out of the wally people, and I have been
_trying_ (very trying, some may joke), it must be that they don't know
and won't ask someone who does know, and won't admit it, and don't read
code, and barely write it.
(the code itself is evidence towards the latter!)
That's weird, but I keep coming across these weird silences in h/w
worlds. It's as though "I don't know" can't be said. Insights?
Thank you for your input. I'll surely have more questions. If the
above is useful for documentation, please use it. If you want to
interatc with me over producing documentation, I can do that (I have
well over a hundred academic papers to my name, so allegedly I can
write - albeit in computer science, mathematics, software engineering,
etc, still, writing is writing ...).
Regards
PTB
|
|
Hi @pbreuer. I'm one of the main contributors to wally and have done a lot of the work on our tool flow, so hopefully I can shed some light here. I'm not sure who you were talking to from the wally team before, but if you have further questions specific to wally the best way to get in touch with us is by opening an issue or discussion in the cvw repository. We monitor everything opened over there pretty closely. Starting off with a high level overview, the main purpose of riscof is to run the tests already in the riscv-arch-test repository (this repo). Each test in the repo has a string that describes which RISC-V extensions are necessary for it to run. Riscof takes a configuration file as an input that defines which "plugins" to run the tests on. One of these is designated the "reference model" (usually Sail or Spike) and the other the "DUT" (device under test). Additional configuration files pointed to by this main config file define relevant architectural aspects of the DUT, most importantly which extensions are supported. Riscof then uses this information to determine what subset of the tests should be run on the model (based on the provided list of supported extensions), compiles each of the tests, runs them each on both selected plugins (the reference model and the DUT), and compares the final signature that they dump out. Each plugin has a python file that tells riscof what command to use to compile tests for it and what command to use to run a test on it. The tests are each designed so that all of the architecturally important information that it is testing for ends up getting stored to a particular region of memory (dubbed the "signature region"). At the end of the test, this signature region of memory is dumped to a file (the specific means of doing this dump are plugin dependent and are part of the previously mentioned plugin python files) so that riscof can compare the two. Riscof is capable of several other things (like measuring coverage), but as previously mentioned, this is more relevant for test development and is not necessary if you are just trying to run the riscv-arch-tests. In the case of wally specifically, we do things slightly differently. We've found that running the tests through Sail and having riscof do the signature comparison is slower than we would like, especially considering how often we end up running the tests. To get around this we have riscof run with Sail as the reference model and Spike as the DUT. At the end of this it dumps out the signature region from Sail, which should be the officially correct and expected results from the test. We then use our own Makefiles to convert the compiled elf files into hex memfiles that get read into our Verilator simulations by our testbench. At the end of the test, the testbench reads in the expected signature region generated earlier and compares it to the actual state of wally's memory directly in the Verilog of the testbench. This avoids recompiling and regenerating all of the tests and signatures every time we want to run the tests of wally. The different testing methodology that @allenjbaum was referring to with internal state and the RVVI interface is something that we use for a different set of tests and is not relevant for the riscv-arch-tests. All of the relevant files for this for wally are in the tests/riscof directory (other than the testbench). The Makefile has the actual riscof commands we use along with all of the flags. The main config file is called Hopefully this helps a little and feel free to reach out more here for general riscv-arch-test questions or over on the cvw repo for wally specific questions. |
|
Inline comments
On Mon, Jan 6, 2025 at 4:16 PM pbreuer ***@***.***> wrote:
(a) I presume the specification is concrete: a vector of before and after
pairs
(r,v) of register indices and the values for them, plus an instruction.
No; for each test, there is a file that (can) look for particular sources
and targets (register numbers ), their values, and other state.
Those can be explicit or implicit sources and targets (CSR values are
implicit, for example)
There is quite a bit of syntax defined in the ISAC documentation, I
believe.
There is also a regular expression-like translator that expands a compact
format of coverpoint to individual coverpoints.
But perhaps a time-wise sequence may be given rather than just before/after?
yes, there is some of that as well.
(b) I suppose the test consists of a program that sets up the specified
before values, runs the instruction, then checks the after values. It will
be or could be generated by riscof, and I hope it will be output, so that
it can be run on a different simulator.
ACT tests can be handwritten, or generated by the CTG tool (Compatibility
Test Generator) which uses a SAT solver to ensure tests will generator all
the coverpoints
ACT tests are assembly level, and you can see what they look like in the
rvtest_test_suite director of the ACT repo.
ACT tests don't check the value afterward. the store all state that the
instruction has changed into a signature area of memory, which is store in
a file when the test conpletes.
Riscof is responsible for orchestrating the tests to be run, based on the
extensions implemented, and running them on a reference model, and one the
device under test, and comparing the signatures
ACT tests can be quite simple, or very complex (when the involve PMPs and
MMU, for example)
But perhaps the test consists of forcing certain signal values
on one clock, and checking them on subsequent clocks, via a closely
coupled connection to actual simulation within riscof in real time. That is
gate/signal level testing, rather than functional testing.
ACTs do not use that approach. They are designed to be minimally invasive
and to leave RTL unchanged.
(c) I suppose the observations are of the final values of registers and
so on or possibly the values taken over several cycles, especially if
caches and things like that are also checked. I don't know if a
tolerance can be given or not (think FP, the ieee standards allow many
different answers).
ACTs specifically do not test caches; cachs are not architectural state.
The final values are only what the test explicitly stores into the
signature area.
If the reference model has architectural options configured to match the
DUT, it is expected that the signatures will match.
There are currently few exceptions (vector unordered reduction being a big
one; tests can be written to do the right thing for vector agnostic
behavior)
The primary issue we have is our reference model is not totally
configurable yet.
I now need to know what kind of arrangement the riscof apparatus has.
> The coverage file defines each of those cases, and coverage mode ensures
> that the tests exercise all of the cases.
I think you're attributing some magic here that is .. difficult. Are
you saying that the user is not responsible for defining the test
vectors, but that they are built in to riscof, and that you
believe they are complete with respect to the functionality in some
abstract sense?
Again: riscof is a framework that runs tools. As mentioned about CTG takes
coverpoints - which can be values, or properties of values, and genreates
the test.
Read the CTG documentation for details about the magic. This little
different than what many other companies do for design verification
It can't be concretely complete: testing the right answer to a+b would
require
checking every possible instance, which is way more than electrons in
the universe; it must be that an abstraction previously developed on a
formal abstract state machine model is tested via particular randomly
chosen concrete instances for each abstract test case - is that it?
Actually,
you made it sound like a game in which the person who defines the concrete
test instances does not know the abstract model :-).
You have an extremely incomplete picture about the state of the art, and
what is necessary to test for compatibility
I am not going to educate you about this; you'll have to look at the
documentation provided, or a wealth of conference paper to get educated.
There are choices in the abstraction over how to slice up the space of
concrete
instances .. for example are you going to separate off when the sign bit
is high? Low?
When the middle bit is high? Low? Or assume it makes no difference and
regard all those
possibilities as being inside the same abstract state?
> IF the coverage report shows that some particular case has not been
> exercised, then
> the test developer has to go back and get one written or generated.
It sounds like you think there is such a thing as a complete set of
tests of concrete functionality ... there isn't. I guess what is meant is
that the apparatus has a list of criteria (the abstract cases, defined
as logical conditions) and it checks that all those have been exercised
once
at least by the actual test vectors supplied.
Can it tell the difference between an abstract case being not covered
because the answer is wrong, and being not covered because inputs to
trigger it were never supplied? It should, for a one-shot change of
state ..
The problem would be those instructions that require interactions
over time with something else, like memory load and store with
certain flags set, or just in certain conditions (e.g. close-packed
writes).
But I think I get the picture of the WHAT it is for.
Now there's the question of WHAT it IS :-).
My impression is that it is a standalone utility, it does not
engage in any real time interaction or monitoring with a simulation of
the concrete machine under test, neither close- nor far-coupled
programmatically,
it's just uncoupled: one shows it the results
from a set of simulations (or one simulation, many test vectors?) on
something else, and it either checks that against results from a
standard model that either it contains within itself, or checks against
results
that have been provided to it that have been generated on an external
standard model.
The latter, I think.
That would be what one would call an "offline" testing, I think? It would
be the
most sensible division of concerns programmatically, but I suspect that
users
would like the expected results to be generated internally on the fly, so
it
may have a standards compliant simuator inside.
Can you clarify that for me, do you think?
Thank you very much. This is extremely helpful!
If things are like that I need to ask next about input and output formats,
and where.
If not, something else ...
Regards and thanks again!
You can do cycle by cycle testing, self testing, or signature based testing.
There are pros and cons to each.
ACTs chose the latter (that was before my time, I'm not sure of the
criteria was)
I think you need to start looking at the file in the github repositor
firsty; there is quite a bit of detail and examples (the actual tests, the
actual coverpoints).
… PTB
—
Reply to this email directly, view it on GitHub
<#576 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHPXVJW7KXLYGPEYKGSB4M32JMMGJAVCNFSM6AAAAABTLBGXGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZUGE2TIMJSGQ>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
On Tue, 07 Jan 2025 00:25:07 -0800 Allen Baum ***@***.***> wrote:
> (a) I presume the specification is concrete: a vector of before and after
> pairs (r,v) of register indices and the values for them, plus an instruction.
No; for each test, there is a file that (can) look for particular sources
That's what I said! What's the "no"? Is the difference you see that I said
"instruction [under test]" and you said "test"?
and targets (register numbers ), their values, and other state.
Those can be explicit or implicit sources and targets (CSR values are
implicit, for example)
I think you mean "provided by the system" for some readonly CSRs,
but some CSRs are writable, so why wouldn't one specify an initial
value for those? I'm puzzled, but the point/question was that those are
_concrete_ values that one has to supply for checking against riscof's
criteria for "coverage", not a logical condition such as "t4 < 2 * t9"
(that's what I mean by an _abstract_ state), or even something simple
like sp < 0.
Temporal logic specifications of the input state are right out of the
picture, I imagine, but it's quite routine in a specification language
to allow abstract specifications (even TL) such as those, so I was
merely trying to get a feel for what kind of thing the inputs are.
I still imagine from the above that the answer is "yes", :-).
"Concrete" values, not abstract?
There is quite a bit of syntax defined in the ISAC documentation, I
believe.
Possibly. Concrete value pairs would cover it?
There is also a regular expression-like translator that expands a compact
format of coverpoint to individual coverpoints.
Wait, you are saying SOME abstract specification is allowed? Can these
regular expressions involve more that one register (or whatever, memory
site, CSR, etc)? That's an important theoretical and practical question
"No" (no "\1", "\2" stuff in the RE) means that would boil down to
abstract conditions that are machine checkable in linear time as empty
or nonempty (equally, for which one can machine generate a random
concrete instance ... or not, so likely it's like that).
That's interesting, if so.
I'm a bit overcome at the thought of hardware engineers dealing with
any kind of abstract specification required as inputs, but perhaps if
one doesn't tell they'll be OK. Just don't say a word ...
[even one named variable in a "RE" pushes the language beyond computably
determinable, eg a*\1b\1, whee, its such fun to make a parser spin on
the spot]
But perhaps a time-wise sequence may be given rather than just before/after?
yes, there is some of that as well.
Well, that would be problematic in terms of complexity bounds, but
hopefully everything will be protected by `don't do that then'.
To be PRACTICAL, am I supposed to supply ANYTHING AT ALL to riscof
for the purpose of getting wally's testbench to run, can you tell?
I presume I can now generate the files it needs to start, and can fix up
everything it yells about as it runs, and in the end it will produce
something?
Would I be supposed to pass that output to riscof? How would I tag it
as belonging to such and such a test? All I have are .S files! Where
would be the specifications of the test that you have described above?
Are they in files with a certain suffix? Is it in comments in the .S
files? ...
I see nothing. But they #include "model_test.h" and "arch_test.h", perhaps
some macros in those ... not obvious, but there are indications that
perhaps, because there are clearly switches:
#define RVMODEL_SET_MSW_INT
#define RVMODEL_CLEAR_MSW_INT
#define RVMODEL_CLEAR_MTIMER_INT
#define RVMODEL_CLEAR_MEXT_INT
(that looks something to do with "sail", I forget if I compiled it, my
guess is "no, I didn't", if it had anything to do with python, "yes, I
did" if it was in C or C++, no bells are ringing)
> (b) I suppose the test consists of a program that sets up the specified
> before values, runs the instruction, then checks the after values. It will
> be or could be generated by riscof, and I hope it will be output, so that
> it can be run on a different simulator.
ACT tests can be handwritten, or generated by the CTG tool (Compatibility
ACT? (not an easy acronyum to google for!)
Test Generator) which uses a SAT solver to ensure tests will generator all
the coverpoints
If it's a plain SAT solver then yes, one can have logical ("abstract")
specifications as inputs, but time and space complexity can be bad.
If it does the computable fraction of linear algebra (the a<b stuff)
too, I would be pleasantly surprised, but it's possible, especially in
a finite domain (aka 32/64-bit).
But somebody would have had to have gone to a lot of trouble to specify
what the ieee means by "<" of floating point numbers! I've not seen
that, but maybe. Certainly there have been formal verifications of FPUs
since about year 2000, and that implies specifications exist! All hail
finite state space.
ACT tests are assembly level, and you can see what they look like in the
You're mixing jargon! What is a "ACT test"? And what is "assembly
level"? Do you mean "written in assembly language"? I thought you
said test input specification by a user was in the form of pairs
of register/sites and values, with some regular expressions thrown in
and maybe a bit of sequencing specification allowed?
That's not assembly language.
rvtest_test_suite director of the ACT repo.
ACT tests don't check the value afterward. the store all state that the
I don't know what is meant by an "ACT test". I think you're informally
describing a class of tests that work by starting with memory in one
state and finishing with it in another, register values go hang?
instruction has changed into a signature area of memory, which is store in
a file when the test conpletes.
Riscof is responsible for orchestrating the tests to be run, based on the
I don't know what is meant by "orchestrating"!
extensions implemented, and running them on a reference model, and one the
device under test, and comparing the signatures
How can riscof do anything like that, it's a piece of software, it's
not in command of anything?
People do the running of things by calling up a simulator, providing it
with inputs they furnish, and people gather up the results. Riscof is
not involved in any way in that, except that - I gather - it has
provided the programs to be run in the simulator, after having been fed
with specifications by the people of the tests that the people want to
run on the simulator of the hardware description they have provided to
it.
(if s/simulation/reality/g , see matrix, the movie)
Is that an accurate description?
ACT tests can be quite simple, or very complex (when the involve PMPs and
MMU, for example)
PMP? (I've seen that acronym in several processor descriptions, seems
to be something to do with configuration registers, maybe of
"peripherals")
> But perhaps the test consists of forcing certain signal values
> on one clock, and checking them on subsequent clocks, via a closely
> coupled connection to actual simulation within riscof in real time. That is
> gate/signal level testing, rather than functional testing.
>
ACTs do not use that approach. They are designed to be minimally invasive
and to leave RTL unchanged.
OK. Functional specification and test.
(c) I suppose the observations are of the final values of registers and
> so on or possibly the values taken over several cycles, especially if
> caches and things like that are also checked. I don't know if a
> tolerance can be given or not (think FP, the ieee standards allow many
> different answers).
ACTs specifically do not test caches; cachs are not architectural state.
The final values are only what the test explicitly stores into the
signature area.
OK, the "ACT" class of test works by comparing initial and finite
states of memory. The program that runs the test will take from
there initially if required and push to there finally.
We are speaking of test programs generated by riscof following certain
test specifications provided by a user.
If the reference model has architectural options configured to match the
DUT, it is expected that the signatures will match.
I have figured that "DUT" means <something> Under Test. Device?
I can't assign much meaning to tnat statement, but it doesn't matter!
There are currently few exceptions (vector unordered reduction being a big
No idea what that refers to. What vector? What reduction? Sounds like
maybe time-wise ordering of events or (abstract) states is being
referenced, and maybe by "reduction" is meant simplification of the
abstract specification in the case when precise time values are not
given , since if one were to cross disjunct a load of different
conjunctive specifications, one per each time-moment, one would get
a huuuuge logic expression. Simplification to a statement with a
minimal number of disjuncts in normal form would be something hugely
exponentially bounded as a computation, wouldn't it? I don't like to
even consider it mentally!
[It's obviously computable on a finite state space, with upper bound
2^O#states .. that's all the possible descriptions there are going
to be, "just" check them all for equivalence!]
one; tests can be written to do the right thing for vector agnostic
behavior)
The primary issue we have is our reference model is not totally
configurable yet.
Oh, that seems to be saying the riscv spec is not really perfectly
or completely formalized yet?
> I now need to know what kind of arrangement the riscof apparatus has.
>
> > The coverage file defines each of those cases, and coverage mode ensures
> > that the tests exercise all of the cases.
>
> I think you're attributing some magic here that is .. difficult. Are
> you saying that the user is not responsible for defining the test
> vectors, but that they are built in to riscof, and that you
> believe they are complete with respect to the functionality in some
> abstract sense?
Again: riscof is a framework that runs tools. As mentioned about CTG takes
I don't know what is meant by that! It seems to be an application that
when I run it produces some noise. In what sense is that something that
runs something? If you mean it makes calls to system tools such as
"make", that's fine.
coverpoints - which can be values, or properties of values, and genreates
the test.
So the CTG (test generator) is a subsystem that generates programs from
test specifications.
That's great! I have got something!
It should be separate. Never mix functionalities in one application, as
anyone who has bought a combination hammer/screwdriver will attest.
Read the CTG documentation for details about the magic. This little
I never want to know details about anything! I generate them :(. I try
as hard as I can to forget everything about everything!
Not magic - it "sounds" like an ordinary domain specific test
specification language, translating internally into propositional and/or
time logic, interpreted by CTG as an instance generator for specific
start and finish test vectors, for which it then generates a program to
take as input and furnish as outputs using a named instruction type for
simulation.
A second different application X should take the same test spec
and the results from running on the simulator and issue a report.
A third application Y should take the group of test specs as a whole
and say if they in theory "cover" the specified functionality
of the particular instruction associated with the tests.
A fourth application Z should evaluate the results from the simulation
to see if in practice covered the functionality too.
As well as checking conformance.
That it is SAT-based is what really tells me most!
different than what many other companies do for design verification
I wouldn't know about industrial things.
> It can't be concretely complete: testing the right answer to a+b would
> mean checking every possible instance, which is way more than electrons in
> the universe; it must be that an abstraction previously developed on a
> formal abstract state machine model is tested via particular randomly
> chosen concrete instances for each abstract test case - is that it?
> Actually,
> you made it sound like a game in which the person who defines the concrete
> test instances does not know the abstract model :-).
You have an extremely incomplete picture about the state of the art, and
Somehow I think you're wrong! Perhaps you'd like to look me up on
google! You're likely immersed in what you call "the state of the art",
and mistaking that for a picture.
what is necessary to test for compatibility
I am not going to educate you about this; you'll have to look at the
documentation provided, or a wealth of conference paper to get educated.
Tough, then you don't get to communicate.
(It sounds like engineering level pique at missing out on high level
understanding! Toys, pram, etc. What can one say)
Weird, but common in low level engineers - if one doesn't see the
picture oneself, one thinks it's not there to be seen and others are
talking about nothings! Understandable, but problematic.
> There are choices in the abstraction over how to slice up the space of
> concrete
> instances .. for example are you going to separate off when the sign bit
> is high? Low?
> When the middle bit is high? Low? Or assume it makes no difference and
> regard all those
> possibilities as being inside the same abstract state?
>
> > IF the coverage report shows that some particular case has not been
> > exercised, then
> > the test developer has to go back and get one written or generated.
I was talking about the "cases", those abstractions that partition the
total state space, and how much is subsumed under any one case among
those. Who designs those?
I thought the example above was illustrative of the design problem! Is
it not?
At any rate, that's just intellectual curiousity and not relevant to
the practical problem of figuring what if anything I will be supposed
to give to riscof as a result of running wall simulation, eventually, if
that's what occurs.
I will just want to know if my modification to computational semantics
still satisfies the classical interpretation in the trivial case.
So I really want to know as little as possible about this! The trouble
I had is that the wally people I talk to seem to know "nothing" about
software, not their software or anyone's, so can't tell me anything
about what is supposed to be needed, or how it works, or anything.
Somebody probably just got it to work, somehow, sometime, and nobody
knows now what it was is - it's just a button to press.
Talkin to you has allowed me to figure what role riscof plays, and
therefore what wally must need, and that has allowed me to _find_
it in those 200+ source files, and I can now go further with that
program. Thank you.
You can do cycle by cycle testing, self testing, or signature based testing.
Those are jargon terms, so I wouldn't know what they mean :(. At any
rate, I am convinced riscof (the application(s)) is standalone and
computationally uncoupled from simulation so I'll take it that somebody
in the wally team has generated tests using it, and I'll face the hurdle
of perhaps using riscof later to interpret the results from simulation
when I get to that. It probably will not be required as I can instead
compare to results on the same tests from the unmodified wally code.
Come to that, I should be able to interpret deviations. I expect there
to be bugs, big bugs, in the modified wally code, which will exercise
zen debugging powers to the max already, so point semantic errors will
be just extra frizz on the edges.
There are pros and cons to each.
ACTs chose the latter (that was before my time, I'm not sure of the
criteria was)
I think you need to start looking at the file in the github repositor
firsty; there is quite a bit of detail and examples (the actual tests, the
actual coverpoints).
I have no idea about that but as far as I recall now riscof is written
in python for installation under pip and that renders it uninstallable
on a managed system. It required pyyaml =5.2, for example, whereas
debian has 6.0.3. Half the admin code on the system is in python, and I
cannot and wil not wreck it.
I had to interpolate among many different python suppliers of the same
module names in many cases, so it was always going to be a long shot for
"working" as regards that compilation. I would urge never using python
for installation. Now if you had written in haskell, I would be more
enthusiastic, but that's got another kind of hell in only = tests for
compatibility.
If you provide a tar snapshot of compiled standalone binaries (or python
scripts?) with integrated dependencies, that would be fine.
Providing source for potential inspection complies with open source.
If posted with dynamic external dependencies like pyyaml, please do try
to be waaaaaaaay on the old and conservative side (in this case too
conservative, however :-) and flexible (oww, that's what sunk).
For fun, here's what pip says if run:
error: externally-managed-environment
This environment is externally managed
To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
Amen to that.
Thanks again.
PTB
|
|
Some of your questions make it obvious you haven't read through the RISC-V
architecture spec. I think that's a first step.
Other statements and questions make it obvious you haven't actually read
riscof, or ctg or isac documentation either.
Basics:
CTG is the Compatibility Test Generator. It takes a YAML file of
coverpoints, some test macros, and fills in parameters to the macros to
generate assembly language tests complete with boilerplate setup/teardown.
ISAC is the ISA Coverage tool. It takes a YAML file of coverpoints, and
parses a log file produced by riscof in coverage mode, producing a
histogram of the coverpoints that were seen (or not)
riscv-config checks for legal configurations. It is given a YAML file
describing an implementation, produces a list of appropriate test for
riscof to run,
Riscof is the framework responsible for taking test inputs, (tests,
models, YAML coverpoints, YAML hart configuration) and running CTG, ISA,
riscv-config, the toolchain, and tests produced
In run mode, it runs riscv-config, uses its output to run appropriate
tests on 2 different models, extracts and compares the signatures, and
produces a test-results doc.
In coverage mode, it runs the tests on a single model with logging
enabled, and feeds to ISAC , which produces the coverage report.
Oh, and ACT == Architectural Compatibility Tests
When you have done that, come back and ask again.
…On Tue, Jan 7, 2025 at 7:31 AM pbreuer ***@***.***> wrote:
On Tue, 07 Jan 2025 00:25:07 -0800
Allen Baum ***@***.***> wrote:
> > (a) I presume the specification is concrete: a vector of before and
after
> > pairs (r,v) of register indices and the values for them, plus an
instruction.
> No; for each test, there is a file that (can) look for particular sources
That's what I said! What's the "no"? Is the difference you see that I said
"instruction [under test]" and you said "test"?
> and targets (register numbers ), their values, and other state.
> Those can be explicit or implicit sources and targets (CSR values are
> implicit, for example)
I think you mean "provided by the system" for some readonly CSRs,
but some CSRs are writable, so why wouldn't one specify an initial
value for those? I'm puzzled, but the point/question was that those are
_concrete_ values that one has to supply for checking against riscof's
criteria for "coverage", not a logical condition such as "t4 < 2 * t9"
(that's what I mean by an _abstract_ state), or even something simple
like sp < 0.
Temporal logic specifications of the input state are right out of the
picture, I imagine, but it's quite routine in a specification language
to allow abstract specifications (even TL) such as those, so I was
merely trying to get a feel for what kind of thing the inputs are.
I still imagine from the above that the answer is "yes", :-).
"Concrete" values, not abstract?
> There is quite a bit of syntax defined in the ISAC documentation, I
> believe.
Possibly. Concrete value pairs would cover it?
> There is also a regular expression-like translator that expands a compact
> format of coverpoint to individual coverpoints.
Wait, you are saying SOME abstract specification is allowed? Can these
regular expressions involve more that one register (or whatever, memory
site, CSR, etc)? That's an important theoretical and practical question
"No" (no "\1", "\2" stuff in the RE) means that would boil down to
abstract conditions that are machine checkable in linear time as empty
or nonempty (equally, for which one can machine generate a random
concrete instance ... or not, so likely it's like that).
That's interesting, if so.
I'm a bit overcome at the thought of hardware engineers dealing with
any kind of abstract specification required as inputs, but perhaps if
one doesn't tell they'll be OK. Just don't say a word ...
[even one named variable in a "RE" pushes the language beyond computably
determinable, eg a*\1b\1, whee, its such fun to make a parser spin on
the spot]
> But perhaps a time-wise sequence may be given rather than just
before/after?
>
> yes, there is some of that as well.
Well, that would be problematic in terms of complexity bounds, but
hopefully everything will be protected by `don't do that then'.
To be PRACTICAL, am I supposed to supply ANYTHING AT ALL to riscof
for the purpose of getting wally's testbench to run, can you tell?
I presume I can now generate the files it needs to start, and can fix up
everything it yells about as it runs, and in the end it will produce
something?
Would I be supposed to pass that output to riscof? How would I tag it
as belonging to such and such a test? All I have are .S files! Where
would be the specifications of the test that you have described above?
Are they in files with a certain suffix? Is it in comments in the .S
files? ...
I see nothing. But they #include "model_test.h" and "arch_test.h", perhaps
some macros in those ... not obvious, but there are indications that
perhaps, because there are clearly switches:
#define RVMODEL_SET_MSW_INT
#define RVMODEL_CLEAR_MSW_INT
#define RVMODEL_CLEAR_MTIMER_INT
#define RVMODEL_CLEAR_MEXT_INT
(that looks something to do with "sail", I forget if I compiled it, my
guess is "no, I didn't", if it had anything to do with python, "yes, I
did" if it was in C or C++, no bells are ringing)
> > (b) I suppose the test consists of a program that sets up the specified
> > before values, runs the instruction, then checks the after values. It
will
> > be or could be generated by riscof, and I hope it will be output, so
that
> > it can be run on a different simulator.
> ACT tests can be handwritten, or generated by the CTG tool (Compatibility
ACT? (not an easy acronyum to google for!)
> Test Generator) which uses a SAT solver to ensure tests will generator
all
> the coverpoints
If it's a plain SAT solver then yes, one can have logical ("abstract")
specifications as inputs, but time and space complexity can be bad.
If it does the computable fraction of linear algebra (the a<b stuff)
too, I would be pleasantly surprised, but it's possible, especially in
a finite domain (aka 32/64-bit).
But somebody would have had to have gone to a lot of trouble to specify
what the ieee means by "<" of floating point numbers! I've not seen
that, but maybe. Certainly there have been formal verifications of FPUs
since about year 2000, and that implies specifications exist! All hail
finite state space.
> ACT tests are assembly level, and you can see what they look like in the
You're mixing jargon! What is a "ACT test"? And what is "assembly
level"? Do you mean "written in assembly language"? I thought you
said test input specification by a user was in the form of pairs
of register/sites and values, with some regular expressions thrown in
and maybe a bit of sequencing specification allowed?
That's not assembly language.
> rvtest_test_suite director of the ACT repo.
> ACT tests don't check the value afterward. the store all state that the
I don't know what is meant by an "ACT test". I think you're informally
describing a class of tests that work by starting with memory in one
state and finishing with it in another, register values go hang?
> instruction has changed into a signature area of memory, which is store
in
> a file when the test conpletes.
> Riscof is responsible for orchestrating the tests to be run, based on the
I don't know what is meant by "orchestrating"!
> extensions implemented, and running them on a reference model, and one
the
> device under test, and comparing the signatures
How can riscof do anything like that, it's a piece of software, it's
not in command of anything?
People do the running of things by calling up a simulator, providing it
with inputs they furnish, and people gather up the results. Riscof is
not involved in any way in that, except that - I gather - it has
provided the programs to be run in the simulator, after having been fed
with specifications by the people of the tests that the people want to
run on the simulator of the hardware description they have provided to
it.
(if s/simulation/reality/g , see matrix, the movie)
Is that an accurate description?
> ACT tests can be quite simple, or very complex (when the involve PMPs and
> MMU, for example)
PMP? (I've seen that acronym in several processor descriptions, seems
to be something to do with configuration registers, maybe of
"peripherals")
> > But perhaps the test consists of forcing certain signal values
> > on one clock, and checking them on subsequent clocks, via a closely
> > coupled connection to actual simulation within riscof in real time.
That is
> > gate/signal level testing, rather than functional testing.
> >
>
> ACTs do not use that approach. They are designed to be minimally invasive
> and to leave RTL unchanged.
OK. Functional specification and test.
> (c) I suppose the observations are of the final values of registers and
> > so on or possibly the values taken over several cycles, especially if
> > caches and things like that are also checked. I don't know if a
> > tolerance can be given or not (think FP, the ieee standards allow many
> > different answers).
>
> ACTs specifically do not test caches; cachs are not architectural state.
> The final values are only what the test explicitly stores into the
> signature area.
OK, the "ACT" class of test works by comparing initial and finite
states of memory. The program that runs the test will take from
there initially if required and push to there finally.
We are speaking of test programs generated by riscof following certain
test specifications provided by a user.
> If the reference model has architectural options configured to match the
> DUT, it is expected that the signatures will match.
I have figured that "DUT" means <something> Under Test. Device?
I can't assign much meaning to tnat statement, but it doesn't matter!
> There are currently few exceptions (vector unordered reduction being a
big
No idea what that refers to. What vector? What reduction? Sounds like
maybe time-wise ordering of events or (abstract) states is being
referenced, and maybe by "reduction" is meant simplification of the
abstract specification in the case when precise time values are not
given , since if one were to cross disjunct a load of different
conjunctive specifications, one per each time-moment, one would get
a huuuuge logic expression. Simplification to a statement with a
minimal number of disjuncts in normal form would be something hugely
exponentially bounded as a computation, wouldn't it? I don't like to
even consider it mentally!
[It's obviously computable on a finite state space, with upper bound
2^O#states .. that's all the possible descriptions there are going
to be, "just" check them all for equivalence!]
> one; tests can be written to do the right thing for vector agnostic
> behavior)
> The primary issue we have is our reference model is not totally
> configurable yet.
Oh, that seems to be saying the riscv spec is not really perfectly
or completely formalized yet?
> > I now need to know what kind of arrangement the riscof apparatus has.
> >
> > > The coverage file defines each of those cases, and coverage mode
ensures
> > > that the tests exercise all of the cases.
> >
> > I think you're attributing some magic here that is .. difficult. Are
> > you saying that the user is not responsible for defining the test
> > vectors, but that they are built in to riscof, and that you
> > believe they are complete with respect to the functionality in some
> > abstract sense?
> Again: riscof is a framework that runs tools. As mentioned about CTG
takes
I don't know what is meant by that! It seems to be an application that
when I run it produces some noise. In what sense is that something that
runs something? If you mean it makes calls to system tools such as
"make", that's fine.
> coverpoints - which can be values, or properties of values, and genreates
> the test.
So the CTG (test generator) is a subsystem that generates programs from
test specifications.
That's great! I have got something!
It should be separate. Never mix functionalities in one application, as
anyone who has bought a combination hammer/screwdriver will attest.
> Read the CTG documentation for details about the magic. This little
I never want to know details about anything! I generate them :(. I try
as hard as I can to forget everything about everything!
Not magic - it "sounds" like an ordinary domain specific test
specification language, translating internally into propositional and/or
time logic, interpreted by CTG as an instance generator for specific
start and finish test vectors, for which it then generates a program to
take as input and furnish as outputs using a named instruction type for
simulation.
A second different application X should take the same test spec
and the results from running on the simulator and issue a report.
A third application Y should take the group of test specs as a whole
and say if they in theory "cover" the specified functionality
of the particular instruction associated with the tests.
A fourth application Z should evaluate the results from the simulation
to see if in practice covered the functionality too.
As well as checking conformance.
That it is SAT-based is what really tells me most!
> different than what many other companies do for design verification
I wouldn't know about industrial things.
> > It can't be concretely complete: testing the right answer to a+b would
> > mean checking every possible instance, which is way more than
electrons in
> > the universe; it must be that an abstraction previously developed on a
> > formal abstract state machine model is tested via particular randomly
> > chosen concrete instances for each abstract test case - is that it?
> > Actually,
> > you made it sound like a game in which the person who defines the
concrete
> > test instances does not know the abstract model :-).
>
> You have an extremely incomplete picture about the state of the art, and
Somehow I think you're wrong! Perhaps you'd like to look me up on
google! You're likely immersed in what you call "the state of the art",
and mistaking that for a picture.
> what is necessary to test for compatibility
> I am not going to educate you about this; you'll have to look at the
> documentation provided, or a wealth of conference paper to get educated.
Tough, then you don't get to communicate.
(It sounds like engineering level pique at missing out on high level
understanding! Toys, pram, etc. What can one say)
Weird, but common in low level engineers - if one doesn't see the
picture oneself, one thinks it's not there to be seen and others are
talking about nothings! Understandable, but problematic.
> > There are choices in the abstraction over how to slice up the space of
> > concrete
> > instances .. for example are you going to separate off when the sign
bit
> > is high? Low?
> > When the middle bit is high? Low? Or assume it makes no difference and
> > regard all those
> > possibilities as being inside the same abstract state?
> >
> > > IF the coverage report shows that some particular case has not been
> > > exercised, then
> > > the test developer has to go back and get one written or generated.
I was talking about the "cases", those abstractions that partition the
total state space, and how much is subsumed under any one case among
those. Who designs those?
I thought the example above was illustrative of the design problem! Is
it not?
At any rate, that's just intellectual curiousity and not relevant to
the practical problem of figuring what if anything I will be supposed
to give to riscof as a result of running wall simulation, eventually, if
that's what occurs.
I will just want to know if my modification to computational semantics
still satisfies the classical interpretation in the trivial case.
So I really want to know as little as possible about this! The trouble
I had is that the wally people I talk to seem to know "nothing" about
software, not their software or anyone's, so can't tell me anything
about what is supposed to be needed, or how it works, or anything.
Somebody probably just got it to work, somehow, sometime, and nobody
knows now what it was is - it's just a button to press.
Talkin to you has allowed me to figure what role riscof plays, and
therefore what wally must need, and that has allowed me to _find_
it in those 200+ source files, and I can now go further with that
program. Thank you.
> You can do cycle by cycle testing, self testing, or signature based
testing.
Those are jargon terms, so I wouldn't know what they mean :(. At any
rate, I am convinced riscof (the application(s)) is standalone and
computationally uncoupled from simulation so I'll take it that somebody
in the wally team has generated tests using it, and I'll face the hurdle
of perhaps using riscof later to interpret the results from simulation
when I get to that. It probably will not be required as I can instead
compare to results on the same tests from the unmodified wally code.
Come to that, I should be able to interpret deviations. I expect there
to be bugs, big bugs, in the modified wally code, which will exercise
zen debugging powers to the max already, so point semantic errors will
be just extra frizz on the edges.
> There are pros and cons to each.
> ACTs chose the latter (that was before my time, I'm not sure of the
> criteria was)
>
> I think you need to start looking at the file in the github repositor
> firsty; there is quite a bit of detail and examples (the actual tests,
the
> actual coverpoints).
I have no idea about that but as far as I recall now riscof is written
in python for installation under pip and that renders it uninstallable
on a managed system. It required pyyaml =5.2, for example, whereas
debian has 6.0.3. Half the admin code on the system is in python, and I
cannot and wil not wreck it.
I had to interpolate among many different python suppliers of the same
module names in many cases, so it was always going to be a long shot for
"working" as regards that compilation. I would urge never using python
for installation. Now if you had written in haskell, I would be more
enthusiastic, but that's got another kind of hell in only = tests for
compatibility.
If you provide a tar snapshot of compiled standalone binaries (or python
scripts?) with integrated dependencies, that would be fine.
Providing source for potential inspection complies with open source.
If posted with dynamic external dependencies like pyyaml, please do try
to be waaaaaaaay on the old and conservative side (in this case too
conservative, however :-) and flexible (oww, that's what sunk).
For fun, here's what pip says if run:
error: externally-managed-environment
This environment is externally managed
To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
Amen to that.
Thanks again.
PTB
—
Reply to this email directly, view it on GitHub
<#576 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AHPXVJSBAXUGPKONAGIVGVD2JPXOPAVCNFSM6AAAAABTLBGXGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZVGU4DKMRWGM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Please take a look at my response above for some details on Wally’s use of all of this. If you have further Wally specific questions please ask them on the CVW GitHub repository. Either open an issue or discussion. I don’t see any posts over there from you. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
resolve #560
Usage:
make setupNote:
I did not put the setup content into the Dockerfile because we cannot reliably obtain all dependencies through apt install at the moment, and users may want to do some customization (like remove/update/custome things) before installation
/workspaces, at the same level as the act repositoryI have tested the entire process locally, and the experience is very good