IO primitive, and best practices.

[rossc719g]

Context: I've been working on a library (in bs classic, if that matters) that enables co-simulation.
The idea is for the bluesim or verilog simulation to be in communication with software, (C++, Rust, Python, whatever) which provides stimulus and checking of results. I'd like it to be language agnostic.

The approach I am using is to lean on file IO primitives (which are pretty much universally present in any language), and use named pipes for the stimulus-to-dut and dut-to-stimulus communication. I've run into a number of complications, most of which I can work though, but one that seems to be biting me the most is on the file-reading side bluespec. Specifically, it seems that all I get is "$fgetc". (And $ungetc).

I'd like for the pipes to cary binary data. Conversion to and from (e.g.) hex, seems like an unneeded step.

Here is where I started. (i is the input type. It is the stimulus I am passing from stim to dut, using its Bits instance, i_bytes is the numeric type with the number of bytes being transmitted):

    i_ch :: Vector i_bytes (Int 32) <- replicateM $ fgetc stim_to_dut)
    if (any (\ch -> ch < 0) i_ch) then do
      ... handle EOF stuff ...
    else do
      let i :: i = unpack $ truncate i_b $ pack i_vb $ map truncate $ map pack i_ch
      ... make use of the input ...

The first issue right off the bat is that replicateM $ fgetc stim_to_dut) does not do what I would expect.
It seems that it reads one byte from the pipe, and then replicates the byte. Not sure why.
But, thankfully, replacing it with mapM (\_ -> $fgetc stim_to_dut) _ seems to get the job done.

The next issue is that it seems that this approach results in extremely strange code generation.

If i is any sufficiently large type, e.g. Vector 256 (UInt 32), the compilation takes a very long time, and eventually fails with a "Stack space overflow".

If I increase the stack space, or reduce the size of the vector, I can get it to pass that phase of the compilation. The Verilog simulation works fine, but the bluesim fails to compile because the c++ compiler gives "fatal error: bracket nesting level exceeded maximum of 256". Again, I can get around this with compiler flags, but it feels like I am basically forcing it down a path where it should not be going.

So, I decided to use import "BDPI"... (in BSV .. I couldn't find the bs-classic equivalent) to wrap the c read and write calls. The bsc docs seemed to suggest that it would do the required magic to make it work in verilog too (with vpi_wrapper_*{c,h} files, etc.), but when I went down that road, I got compiler warnings that it was "not implemented" or "not guaranteed to work" etc.. I was unable to get it to work.

So finally, I decided to go with the import "BDPI"... for bluesim, and import "BVI" for verilog, and then use genC to determine which to implement. But since the BVI approach only imports modules, not tasks, I had to make it a bit more complex, and ran into subtle timing issues and ordering issues. (The order of the two $fopen calls is important, as is the order of the read vs write, and it proved to be difficult to make those assurances in verilog).

I am about to start fresh and try again with the "BDPI + BVI" version hoping I can overcome my earlier issues.. but I get the sense that there must be an easier way.

So... a few questions:

• Is the replicateM $ fgetc stim_to_dut) vs mapM (\_ -> $fgetc stim_to_dut) _ thing expected? Or is that a bug?
• Is there a better way to formulate the 1-byte-at-a-time reading that does not blow up the stack space, nesting level, and compilation time? I have tried a few variants, but not gotten any joy.
• Why is there no $fread? I feel like this is my fundamental issue. There is a $fwrite, though, for now I am only using it for bytes as well (just for symmetry mapM_ (\b -> $fwrite dut_to_stim "%c" b)), but I expect I could write out the complete Bit(TMul o_bytes 8) thing if I wanted, right? Is there a reason $fread is not there?
• Is there a bs-classic approach to the import "BDPI" stuff? I'd prefer to avoid BSV if I can.
• Any advice on getting a BDPI imported c function to work in the verilog side? Is that supposed to be working?
• Any advice at all on the general approach would be appreciated.

[quark17]

I would use imported C functions for this (BDPI in BSV). For BDPI, BSC generates proper code for both Bluesim and Verilog. In Verilog, it's implemented with Verilog's VPI. (However, you can give the new -use-dpi flag, to have BSC generate SystemVerilog DPI instead.)

I assume the warning message you saw was this:

WARNING: IVerilog support for VPI is incomplete.  BSV designs using foreign
         functions are not guaranteed to compile or simulate correctly.

This is emitted in BSC's linking script for IVerilog. It's just a warning, because there are some uses that IVerilog fails to handle (which you'll notice immediately when you try to run), and newer IVerilog versions have fixes for some of them. I seem to recall that the situations which IVerilog didn't support are pure functions (which appear in Verilog continuous assignments); so as long as your imports are of type Action and ActionValue (which appear in a Verilog always-block), you'll be fine. In any case, just try and see if it runs. Alternatively, you can use a different Verilog simulator. Verilator is another free simulator -- but it doesn't support VPI, so you'll need to add the -use-dpi flag.

• Is the replicateM $ fgetc stim_to_dut) vs mapM (\_ -> $fgetc stim_to_dut) _ thing expected? Or is that a bug?

Do you have a small example that I can run? The definition of replicateM in the Vector library is exactly the alternative code that you showed:

replicateM c = mapM (const c) genList

• Is there a better way to formulate the 1-byte-at-a-time reading that does not blow up the stack space, nesting level, and compilation time? I have tried a few variants, but not gotten any joy.

It would help if you can provide an executable example. I'm not sure if you've shown enough code for me to understand what you're doing.

BSC doesn't preserve vectors in the output code. It inlines everything, which can cause the code to get large. This is possibly a scaling issue, when you're trying to do software-like things. And there may be enhancements (or bug fixes) that can improve the scaling. (There are probably also a lot of optimizations that can happen on the Bluesim generated code that we just haven't implemented yet, because it hasn't been a priority.) But I would suggest avoiding the problem by importing a C function. And if you run into scaling after that you can push any software-like processing inside the C function, instead of in the BH/BSV.

For communication with a software process, we have imported C functions that communicate over pipes. I don't see a good example online, but there is an example inside the Flute repo: https://github.com/bluespec/Flute/blob/master/src_Testbench/Top/C_Imports.bsv#L144-L171

Flute is a CPU and the testbench in that repo supports connecting a GDB process to it, using GDB's remote connection capability (over a pipe). There's an intermediate C program that handles the complicated communication protocol with GDB and translates it into simple requests to the hardward. So, on the hardware side, there are C functions for getting requests (from the intermediary) and sending responses, which are of fixed sizes (64-bit request and 32-bit response). And there are imported functions for connecting and disconnecting (to the pipe). The C implementations for these functions are in the same directory, if you want to look.

The hardware handles one request per clock cycle. The GDB connection can be used to load a program into memory, for example. This happens as a series of requests, which happens over many clock cycles. Your code snippet suggests that you want to send in 256 32-bit words at a time. I don't know what you're doing with that data -- if you're trying to parse that in one clock cycle, I could imagine it generating excessive hardware, and that would be something that I'd push into the C function. If you're just passing it to a method that takes an argument that large (for example, it's going on a really wide bus), then that's fine, but probably better to read it all with one C function call that returns the full size, rather than many reads of small sizes that have to be concatenated. And as a last resort, try doing things over multiple clock cycles.

• Why is there no $fread? I feel like this is my fundamental issue. There is a $fwrite, though, for now I am only using it for bytes as well (just for symmetry mapM_ (\b -> $fwrite dut_to_stim "%c" b)), but I expect I could write out the complete Bit(TMul o_bytes 8) thing if I wanted, right? Is there a reason $fread is not there?

I don't recall what $fread looks like in Verilog, but if it can possibly assign values to multiple names, then it doesn't neatly fit into the way that BSC's functions return an argument. I don't recall whether there's a fundamental reason that the parser couldn't make it work, or if it would just be hard -- or whether we could support it in a limited form, that only assigns one value, for example. If someone has ideas, I'm sure we're open to it.

• Is there a bs-classic approach to the import "BDPI" stuff? I'd prefer to avoid BSV if I can.

I don't recall, but I would expect maybe not. We definitely want BH to have the same features as BSV, it just hasn't been done yet. I note that there is a foreign declaration that exists in BH, but I think that it can only be used for passing things through to the Verilog (no Bluesim support) and it may only show up as a module instantiation (unless BSC recognizes it as a known system task, like $write etc, and handles it specially). And there is a foreignImport pragma that the BSV parser generates from BDPI declarations, so maybe you could fake it in BH, but I don't know. One thing to try is to write a BDPI in BSV and then print it out in BH (using the -dparsed flag to BSC, or the bsv2bs program), and see what it looks like -- there's not guarantee that the output is legal BH, though.

FYI, there is a way to embed code from one syntax in the other. I thought we had both directions, but maybe it's only one. I can't see anything for embedding BSV code in BH, off hand. But in BSV, you can use the attribute bluespec_classic_def to embed BH in BSV:

(* bluespec_classic_def="\
  class MyConnect a where {\n\
    mkConnect :: (IsModule m c) => a -> m (IFC a) }\
" *)

I thought that we did the same for BSV in BH, but maybe not.

[rossc719g]

Thanks!
This is all incredibly helpful. I'll give all this a try again, and check back with results.
I really appreciate the advice.

[rossc719g]

Regarding this one side question:

• Is the replicateM $ fgetc stim_to_dut) vs mapM (\_ -> $fgetc stim_to_dut) _ thing expected? Or is that a bug?

Do you have a small example that I can run?

I got this down to a simple repro:

In ReplicateMfgetc.bs I have:

package ReplicateMfgetc where

import Vector

mkReplicateMfgetc :: Module (Empty)
mkReplicateMfgetc = module
  rules
    {-# ASSERT no implicit conditions #-}  {-# ASSERT fire when enabled #-}
    "foo": when True ==> do
      opened :: File <- $fopen "my_file" "r"
      x :: Vector 5 (Int 32) <- replicateM $ $fgetc opened
      -- x :: Vector 5 (Int 32) <- mapM (\_ -> $fgetc opened) _
      if (any (\c -> c < 0) x) then do
        $display "Got EOF"
      else do
        $display "%c%c%c%c%c" (x !! 0) (x !! 1) (x !! 2) (x !! 3) (x !! 4)
      $fclose opened
      $finish

In my_file I just have Hello World!
I run it with: bsc -sim -g mkReplicateMfgetc ReplicateMfgetc.bs && bsc -Xc++ -Wno-deprecated-declarations -sim -e mkReplicateMfgetc && ./a.out

My bsc version is Bluespec Compiler, version 2023.01 (build 52adafa), which I downloaded as bsc-2023.01-macos-12.tar.gz from the latest release.

When I run it as pasted above, the output is ooooo, (which is curious. It is replicating the last byte read, not the first?)
When I run it with the mapM variant I get the expected Hello

[quark17]

Thank you for the example! This is definitely a bug, and I have opened Issue #572 for it.

[rossc719g]

Fantastic, thanks. I'll follow there.

[rossc719g]

Another quick reply to one of your Qs:

• Is there a better way to formulate the 1-byte-at-a-time reading that does not blow up the stack space, nesting level, and compilation time? I have tried a few variants, but not gotten any joy.

It would help if you can provide an executable example. I'm not sure if you've shown enough code for me to understand what you're doing.

For example, using a very similar program to the one above:

package ReadManyBytes where

import Vector

type N = 16

mkReadManyBytes :: Module (Empty)
mkReadManyBytes = module
  rules
    {-# ASSERT no implicit conditions #-}  {-# ASSERT fire when enabled #-}
    "foo": when True ==> do
      opened :: File <- $fopen "my_file" "r"
      x :: Vector N (Int 32) <- mapM (\_ -> $fgetc opened) _
      if (any (\c -> c < 0) x) then do
        $display "Got EOF"
      else do
        mapM_ (\c -> $write "%c" c) x
        $display  -- Newline
      $fclose opened
      $finish

(I used curl -o my_file https://assets.angio.net/pi1000000.txt to fill my_file with the first million digits of pi.. though to compile it you don't even need my_file to exist.. )

The compilation times are very non-linear. I realize that all of those $fgetc, < 0, and $write calls get inlined, but it grows so fast that it makes me think I am simply "doing it wrong":

N	bsc -g ...
16	0.345s
64	1.337s
256	1m 10.490s
1024	(Stack space overflow) >40m 20s

I will try later this week with the BDPI-only approach and see if I can get it working in bluesim and verilog. Maybe that warning just scared me off too early. A fair number of my issues revolve around knowing how to invoke bsc via bazel to get all the right artifacts in the right places, but that's definitely not in scope of this forum.

Thanks again for the advice!

[quark17]

Thanks again for the example!

When debugging compilation that takes a lot of time or memory, the first step is to use the -v flag, to cause BSC to report the stages of compilation as it enters and exits them, and the time spent in each. Compilation is a sequence of stages, and knowing which stage is taking time -- or which stage the compiler was in when it ran out of memory -- can point to different causes. Typical culprits (in the order they occur in compilation) are type checking ("typecheck"); elaboration (the "expanded" stage); optimization of the elaborated design ("transform", which occurs twice, and, less often, "lift"); scheduling ("schedule"); Verilog backend optimization ("aopt"); and writing out the Verilog file ("verilog"). In some stages, there may be flags that can provide further debug information (such as -show-elab-progress in the "expanded" stage) or flags to disable individual optimizations that might be the cause of the issue but not are strictly necessary (such as various flags in "aopt", include -no-apt-ATS which turns off the entire stage).

In your case, it's the "lift" and "transform" stages that are taking time. For 256, I see:

starting transform
types OK
transform done
 elapsed time: CPU 3.42s, real 3.43s

...

starting lift
types OK
lift done
 elapsed time: CPU 23.16s, real 23.19s

starting transform
types OK
transform done
 elapsed time: CPU 30.55s, real 30.54s

For 512, I see that it runs out of stack space, even if I increase it, but I see these results before it runs out:

starting transform
types OK
transform done
 elapsed time: CPU 27.61s, real 27.76s

...

starting lift
types OK
lift done
 elapsed time: CPU 207.35s, real 207.49s

This is probably a scaling issue in the compiler (or the evaluator is generating code in a style that then becomes a scaling issue later). The "lift" stage is looking for actions inside nested if-else statements, and trying to pull common actions out, into a single action with a combined condition. It could be a scaling issue because you've generated a large number of actions, but I'd have to look specifically at the code that is being created by the evaluator and later stages. There is one optimization that can be turned off, with -no-opt-bool, but that doesn't seem to help much here.

[quark17]

I just realized that your example includes both the getting of character (with $fgetc) and the writing of characters (with $write). If I remove the write part, then size 512 compiles in 30 seconds, spent mostly in the first "transform" stage:

starting transform
types OK
transform done
 elapsed time: CPU 25.75s, real 25.78s

...

starting lift
types OK
lift done
 elapsed time: CPU 1.62s, real 1.62s

starting transform
types OK
transform done
 elapsed time: CPU 0.23s, real 0.23s

I get the same result if I leave the write in, but do it with a single call to $write of the entire vector, rather than separate calls to $write for each element of the vector:

      if (any (\c -> c < 0) x) then do
        $display "Got EOF"
      else do
        $write "%h" (pack x)

[quark17]

I've created Issue #573 for this.

[rossc719g]

A few things to report back:

1. I tried again with the BDPI approach, and got both the bluesim and verilog sides working! Yay! Thanks again for the advice.

The issue I was having was because the macos distribution was x86_64, but everything else on my machine is arm64, and it was causing linking issues. I was able to fix it by building my own bsc from source.

Other issues I had were with confusion about where certain files needed to be. vpi_wrapper_*.{ch} and directc_*.so both apparently need to be in the CWD, not in the dir with the other sources... so some symlinks were needed to make that happy in bazel.

For the heck of it along the way, I tried -use-dpi but got error (S0015) Polymorphic types are not yet supported with DPI

Is there any way to squelch the warning?

WARNING: IVerilog support for VPI is incomplete.  BSV designs using foreign
         functions are not guaranteed to compile or simulate correctly.

2. Thanks for pointing me to the bsv2bsc command. Very helpful. Unfortunately, the output it produces for the BDPI code does not want to compile. It generates:

{-# properties cosim_read_ = { foreignImport cosim_read } #-};

cosim_read_ :: (Bits (StructMaybe t) mt_sz) => Bit 32 -> Prelude.ActionValue (StructMaybe t);
cosim_read_ =  ForeignFuncC cosim_read;;

But when I compile it it complains about the properties line saying:

Error: "CosimIO.bs", line 4, column 31: (P0005)
  Unexpected "foreignImport"; expected "alwaysReady", "noReady",
  "alwaysEnabled", "scanInsert", "bitBlast", "CLK", "RSTN", "options",
  "verilog", or "deprecate"

I tried removing the properties lines, and it just fails further on:

Error: "CosimIO.bs", line 5, column 15: (T0003)
  Unbound constructor `ForeignFuncC'

[quark17]

For the heck of it along the way, I tried -use-dpi but got error (S0015) Polymorphic types are not yet supported with DPI

If you import functions with fixed bit widths, that works in DPI; but if you import functions that can return any size, that's not yet supported, and you get the above error message. With Verilog's VPI, a wrapper is needed around the user's C function, to convert between the C and Verilog interfaces; with SystemVerilog's DPI, the simulator's interface is the same as C, so no wrapper is needed -- however, for BDPI that can return a variable size, we unfortunately still need a wrapper (to convert between the special type that DPI defines for this and the array of ints that BDPI uses).

Implementing the wrapper isn't hard, it just hasn't been done yet. And part of the reason is that Verilator doesn't support DPI with open arrays (like bit[]), so you wouldn't be able to simulate the generated code with Verilator anyway.

Is there any way to squelch the warning?

There is not currently a way to silence it. We could add something -- either a flag to BSC that gets passed to the linking script, or have the linking script consult an environment variable. It would also be worth auditing the testsuite to see which examples IVerilog still can't handle, and making sure that they are reported to the iverilog repo, and perhaps having BSC do a check for those situations and only warning (or erroring) in those cases. Another option is to remove the warning (if we're confident that people will discover the issue when they try to run the simulation).

Thanks for pointing me to the bsv2bsc command. Very helpful. Unfortunately, the output it produces for the BDPI code does not want to compile.

Ah, yes, the code is not representable in BH. When the pretty-printer comes to a foreign function declaration, it doesn't have any real BH syntax that it can output, so it just prints ForeignFuncC for the benefit of any humans reading the output.

[quark17]

Also, I'm surprised that you wrote a polymorphic BDPI -- is it because you're using the same function in multiple places in the design, but with different fixed sizes at each place? If you instead declare separate fixed-size BDPI imports for each size, then -use-dpi will work.

[rossc719g]

Thanks for the hints about the warning. Since I had to compile my own bsc (for the arch issue) anyway, I've commented out the two lines that generate it.

Unfortunately, for my use-case, I do need the reads and writes to be polymorphic. (So, no -use-dpi for me)
It is a generic co-simulation test library, which takes any module to test, and 2 data types to serve as the input, and output.
Then an external sw program handles generating stimulus, and checking the results.
Since the library works with multiple modules, the size of the input and output vary.