Skip to content

Commit

Permalink
lmdb: Platform-specific default map size
Browse files Browse the repository at this point in the history
The current code works fine on x86 platforms since x86-64 uses at least
48-bit of virtual address space. On other 64-bit platforms like aarch64
or riscv64, the minimum allowed virtual address is 39-bit [1] [2].
Current 2**40 allocation will fail:

```
zict/tests/test_lmdb.py:53:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <zict.lmdb.LMDB object at 0x40043bfc10>
directory = '/tmp/test_lmdb-sfepfnw8'

    def __init__(self, directory: str):
        import lmdb

        # map_size is the maximum database size but shouldn't fill up the
        # virtual address space
        map_size = 1 << 40 if sys.maxsize >= 2**32 else 1 << 28
        # writemap requires sparse file support otherwise the whole
        # `map_size` may be reserved up front on disk
        writemap = sys.platform.startswith("linux")
>       self.db = lmdb.open(
            directory,
            subdir=True,
            map_size=map_size,
            sync=False,
            writemap=writemap,
        )
E       lmdb.Error: /tmp/test_lmdb-sfepfnw8: Operation not supported

zict/lmdb.py:43: Error
```

Switching to 2**37 on aarch64 and riscv64 should fix the issue.

[1]: https://www.kernel.org/doc/html/v5.8/arm64/memory.html
[2]: https://www.kernel.org/doc/html/latest/riscv/vm-layout.html
  • Loading branch information
hack3ric committed Oct 31, 2023
1 parent 7e5dafe commit 6458a8a
Showing 1 changed file with 13 additions and 4 deletions.
17 changes: 13 additions & 4 deletions zict/lmdb.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import pathlib
import sys
import platform
from collections.abc import ItemsView, Iterable, Iterator, ValuesView

from zict.common import ZictBase
Expand All @@ -25,7 +26,8 @@ class LMDB(ZictBase[str, bytes]):
directory: str
map_size: int
On Linux and MacOS, maximum size of the database file on disk.
Defaults to 1 TiB on 64 bit systems and 1 GiB on 32 bit ones.
Defaults to 128 GiB on aarch64 and riscv64, 1 TiB on other 64 bit systems like
x86-64 and 1 GiB on 32 bit ones.
On Windows, preallocated total size of the database file on disk. Defaults to
10 MiB to encourage explicitly setting it.
Expand All @@ -48,10 +50,17 @@ def __init__(self, directory: str | pathlib.Path, map_size: int | None = None):

super().__init__()
if map_size is None:
if sys.platform != "win32":
map_size = min(2**40, sys.maxsize // 4)
else:
machine = platform.machine()
if sys.platform == "win32":
map_size = 10 * 2**20
elif machine in ["x86_64", "x64"]:
map_size = 2**40
elif machine in ["i386", "i686", "x86"]:
map_size = 2**30
elif machine.startswith("aarch64") or machine.startswith("armv8") or machine.startswith("riscv64"):
map_size = 2**37
else:
map_size = min(2**40, sys.maxsize // 4)

self.db = lmdb.open(
str(directory),
Expand Down

0 comments on commit 6458a8a

Please sign in to comment.