warning: global JIT code buffer exhausted

[avelure]

When running several simulations in parallel I get this message on some of them with an exit code of -9 just after.

warning: global JIT code buffer exhausted
warning: JIT code buffer for WORK.<...long path...>.EDAC_4096X16_EDAC_4096X16_0_ACTRAM_R0C9.VITALBEHAVIOR too small

It is a bit random on which tests, but it seems to always be the same message. It is in the VITAL handling of a Microchip memory IP.

[nickg]

The -9 thing is probably those processes being killed due to out-of-memory. I think we need to scale the number of background compile threads based on the load average of the machine, not just the number of CPUs.

Is that model publicly available? I haven't done much testing with VITAL before.

[avelure]

The machine has like 20Gb left of memory, but it might be that the memory per docker container is restricted and that's where it runs out, I'll see if I can increase that limit.
I tried setting NVC_MAX_THREADS to 4, but then the none of the tests complete before my 1h timeout (GHDL normally finishes them in 20min). Without NVC_MAX_THREADS some of the tests complete, some fail with the -9 and some don't end within the timeout.

The memory model is in the device library file (unencrypted) which comes with Libero. You can download Libero without a license. For the latest Libero you also get a free 1Y license that can be renewed perpetually.
The EDAC IP that wraps the memory model is free with Libero.
I can upload the core and the library if you prefer.

[avelure]

I tried the same test on native Windows in single thread and it takes a long time to run, I killed it after 2h.
In the transcript I get

C:\proj\libraries\axcelerator\axcelerator.vhd:99937:3: fatal: 84354937700ps+1: interrupted in process :top:i_test_harness:i_dut:i_edac_4096x16:edac_4096x16_0:rtax_ram:edac_4096x16_edac_4096x16_0_actram_0:ax_ram_no_fabric_pipe:edac_4096x16_edac_4096x16_0_ram_0:edac_4096x16_edac_4096x16_0_actram_r0c3:vitalbehavior

If I run without the --jit parameter it finishes in 50min.

[nickg]

I managed to install it and used the SmartGen tool to generate an EDAC core, but not sure what options I should choose. Can you have a look at this to see if it's similar to your case? nickg/vhdl-projects@15a7bda

[avelure]

That looks similar yes, except it seems you got an obfuscated netlist for the module. I guess it might not matter that much, the VITAL stuff is in the RAM64K36 in axcellerator.vhd.

How we use it is we fill up the RAM at the start, then deassert stop_scrub to enable scrubbing, then we sporadically read from the memory. You need to assert stop_scrub in the cycle you are reading.

[nickg]

So is this the same test where GHDL finishes in 20 minutes but NVC takes 50+? I tried a simple testbench for the RAM based on the settings above and it didn't seem particularly slow: 6 seconds vs 8 seconds for GHDL, although NVC is usually a lot faster so there's probably something to optimise there.

Can you see if it's actually the elaboration stage that takes most of the time without --jit? Try nvc -e -V ... without --no-save --jit -r and see how long that takes. It could be something is generating excessive amounts of LLVM IR which causes the slowdown.

At the moment there is an arbitrary limit on the size of code that can be JIT-compiled, that's what causes the "code buffer exhausted" message. I'll try to fix that at the weekend.

[avelure]

The tests are top-level FPGA tests, so it's a bit more than just the memory.
I haven't used NVC on this iteration of the design before, but I used it on another iteration of it about a year a go and then NVC was several times faster than GHDL. I could try to build the version I used back then and test the runtime on the current iteration of the design.

I can check for the elaboration, but I don't think it is the case as I have shorter tests that only lasts for a few minutes. The tests primarily differ in the test sequence, so they should be equally affected if it was the case.

[nickg]

I haven't used NVC on this iteration of the design before, but I used it on another iteration of it about a year a go and then NVC was several times faster than GHDL. I could try to build the version I used back then and test the runtime on the current iteration of the design.

It's quite possible a performance regression slipped through as I did rewrite a lot of the back-end last year. It would be really helpful if you could narrow down approximately when that happened.

[nickg]

The "code buffer exhausted" warning should be gone now but I'd still like to get to the bottom of what caused the performance regression.

[avelure]

"code buffer exhausted" seems to be gone, but the "JIT code buffer for ... too small" and exit with -9 is still present on "nvc 1.10-devel (1.9.0.r76.g66dc008d) (Using LLVM 12.0.0)"
I'll do some dissecting once I have some more spare time.