multiple writes with two different processes raise error even with SHARED_MEMORY_USE_LOCK=1

[weiwongfaye]

I was testing the behavior of using two processes to write the dict, even with SHARED_MEMORY_USE_LOCK=1, I got exceptions
Python 3.9.16. Mac OS
run the two python file in separate terminal with SHARED_MEMORY_USE_LOCK=1 set.
export SHARED_MEMORY_USE_LOCK=1 && python shm_process_safe.py
shm_process_saft.py:

import time
from shared_memory_dict import SharedMemoryDict
smd = SharedMemoryDict(name='tokens', size=1024)
smd['a'] = 'xx'
time.sleep(2)
for i in range(1000000):
    smd['b'] = str(i) + 'yy'
print(smd)

# try:
#     smd.shm.close()
#     smd.shm.unlink()
#     del smd
# except Exception as e:
#     pass

in another terminal, run the other python file also with SHARED_MEMORY_USE_LOCK=1 set.
export SHARED_MEMORY_USE_LOCK=1 && python shm_process_safe2.py
shm_process_safe2.py:

import time
from shared_memory_dict import SharedMemoryDict
smd = SharedMemoryDict(name='tokens', size=1024)
print(smd)
time.sleep(1)
# smd['b']= 'yy'
print(smd)
for i in range(1000000):
    smd['b'] = str(i) + 'xx'
print(smd)
# try:
#     smd.shm.close()
#     smd.shm.unlink()
#     del smd
# except Exception as e:
#     pass

Result:

Got following errors:

export SHARED_MEMORY_USE_LOCK=1 && python shm_process_safe.py                                               
Traceback (most recent call last):
  File "/Users/xx/work/code_snipet/shm/shm_process_safe.py", line 7, in <module>
    smd['b'] = str(i) + 'yy'
  File "/Users/xx/.virtualenvs/octopus/lib/python3.9/site-packages/shared_memory_dict/dict.py", line 95, in __setitem__
    with self._modify_db() as db:
  File "/usr/local/Cellar/[email protected]/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/Users/xx/.virtualenvs/octopus/lib/python3.9/site-packages/shared_memory_dict/dict.py", line 87, in _modify_db
    db = self._read_memory()
  File "/Users/xx/.virtualenvs/octopus/lib/python3.9/site-packages/shared_memory_dict/dict.py", line 184, in _read_memory
    return self._serializer.loads(self._memory_block.buf.tobytes())
  File "/Users/xx/.virtualenvs/octopus/lib/python3.9/site-packages/shared_memory_dict/serializers.py", line 50, in loads
    return pickle.loads(data)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 6: invalid start byte
/usr/local/Cellar/[email protected]/3.9.16/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

[weiwongfaye]

the problem seems to be the multiprocessing.Lock, it doesn't share between the two processes spin up within two terminals.