Intermittent runtime failures on (my?) mac.

[rossc719g]

I am seeing a rare, but reproducible runtime failure when running bluesim executables.
We have not been able to repro on linux, but I seem to be able to get it to fail reliably on my mac.

The failure looks like this:

sem_open: File exists
sem_open: File exists
Error: Unable to load model ./code/mkTrivial.exe.so
    invoked from within
"sim load $model_name $top_module"
    (file "/opt/tools/bsc/latest/lib/tcllib/bluespec/bluesim.tcl" line 188)

I uploaded an example here:
https://github.com/rossc719g/bsc_examples/tree/main/runtime_flake
Which has a little script that runs a trivial example 50000 times, and records failures.
(I usually see 8-10 failures when I run it)

Has anyone seen something like this before?

[quark17]

I have not seen this before. But it probably comes from src/bluesim/portability.{h,cxx}. There are separate implementations for the semaphore functions for macOS (using named semaphores), and that is probably why you don't see the issue on Linux. The sem_open call (for macOS) occurs on line 94 of portability.cxx, in the function create_semaphore. A few lines up, it attempts to create a unique semaphore name, based on the process ID plus a sequence number (which is declared static and is incremented each time through the function.)

You could try inserting a print of the name, to see what's going on. I don't recall why macOS needed to use named semaphore, but if non-named are available now, we could try switching to that. Otherwise, it sounds like there's a name clash, and we'd need to figure out why that's happening.

[rossc719g]

Thanks for the pointer!
I have been working on this a bit (Aided by discussions with @nanavati) and I think we have a working hypothesis:

Assume that at some point in the past bluesim crashed**, after the named semaphore was created, but before it was unlinked.

In that case, the semaphore will live on, essentially leaving a "poisoned" PID on the machine. The next time bluesim gets the same PID, it will fail because that semaphore already exists. I verified the plausibility of this by printing out the name of the semaphore when it fails, and I am seeing repeating PIDs. So I am pretty sure this is the issue.

Some possible solutions:

1. When sem_open fails, just try again (up to some number of retries), after incrementing the seq_number.
   I verified that 10 retries is sufficient to fix the issue in my case to make my tests pass.
   It does not solve the problem of having created the poisoned PIDs in the first place, but it makes it more robust when/if it happens. It also does not prevent whatever caused them to leak in the first place. In theory, it is possible that a very long running machine could leak enough of them to poison 10 with the same PID... but that seems pretty unlikely. So maybe this is good enough? I don't care for this solution, but it does technically work.

2. Call sem_unlink() immediately after sem_open().
   I also verified that this works.
   From the sem_unlink() manpage:
   The named semaphore named name is removed. If the semaphore is in use by other processes, then name is immediately disassociated with the semaphore, but the semaphore itself will not be removed until all references to it have been closed. Subsequent calls to sem_open() using name will refer to or create a new semaphore named name.
   This is attractive because it more closely mimics the intended unnamed semaphore. The name only exists during the creation. It would allow us to use typedef sem_t tSemaphore; unconditionally, and remove the struct currently being used for named semaphores. It also can use a common implementation for post_semaphore, trywait_on_semaphore, and wait_on_semaphore, and removes the need for the seq_number, and getpid().
   This would radically reduce the possibility of leaking poison PIDs. The chances of the process exiting (crashing, or being killed, etc..) between the sem_open and sem_unlink is approximately zero.
   This does not do anything to remedy the situation if/when semaphores are leaked though, and a leaked one would still cause it to fail.

3. Call sem_unlink immediately before sem_open.
   I did verify that this works.
   This is attractive because it will eventually correct the issue if/when semaphores are leaked. However it does not prevent more from being leaked. But leaks are (apparently) pretty rare. This seems like it would be sufficient. The call to sem_unlink will almost always return an (ignored) error, but that's fine.

4. Both 2 and 3. So: sem_unlink, sem_open, sem_unlink.
   I verified this as well.
   This seems like the best of 2 & 3. It corrects the issue if/when it leaks, and make leaks nearly impossible. We can still remove the struct, and use common implementations for the other functions. I like this option the best.

Thoughts? Shall I put option 4 into a pull request?

** I don't know what exactly the circumstances might have been, but since I was writing bdpi functions, it is not inconceivable that during development, I might have crashed it a few times. I estimate that I have about 30 "poisoned" PIDs.

[quark17]

Thanks! I also wondered if dying before calling shutdown (and releasing the semaphores) might be the cause of the poisoning.

Option 4 sounds reasonable to me. If you remove seq_number and getpid(), just make sure that the constant part of the name is unique enough (long enough?) to reduce the chances of being used by anything else.

[rossc719g]

Oh, and I verified that unnamed semaphores still do not exist on mac. (https://www.quora.com/Why-does-OS-X-not-support-unnamed-semaphores seems like a pretty authoritative answer of why)

[rossc719g]

Made a pull request for it: #613