Skip to content

Commit

Permalink
docs & var rename
Browse files Browse the repository at this point in the history
  • Loading branch information
matveyvarg committed Jun 10, 2024
1 parent c2f0d01 commit cc9c13d
Show file tree
Hide file tree
Showing 3 changed files with 23 additions and 4 deletions.
6 changes: 3 additions & 3 deletions deker/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ def __init__(
write_lock_check_interval: int = 1,
loglevel: str = "ERROR",
memory_limit: Union[int, str] = 0,
skip_memory_check: bool = False,
skip_collection_create_memory_check: bool = False,
**kwargs: Any,
) -> None:
"""Deker client constructor.
Expand All @@ -210,7 +210,7 @@ def __init__(
converted into bytes. If result is ``<= 0`` - total RAM + total swap is used
.. note:: This parameter is used for early runtime break in case of potential memory overflow
:param skip_memory_check: If we don't want to check size during collection creation
:param skip_collection_create_memory_check: If we don't want to check size during collection creation
:param kwargs: a wildcard, reserved for any extra parameters
"""
try:
Expand All @@ -231,7 +231,7 @@ def __init__(
write_lock_check_interval=write_lock_check_interval,
loglevel=loglevel.upper(),
memory_limit=mem_limit,
skip_memory_check=skip_memory_check,
skip_collection_create_memory_check=skip_collection_create_memory_check,
)
self.__uri: Uri = Uri.create(self.__config.uri)
self.__is_closed: bool = True
Expand Down
2 changes: 1 addition & 1 deletion deker/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ class DekerConfig:
varray_data_directory: str = "varray_data"
array_symlinks_directory: str = "array_symlinks"
varray_symlinks_directory: str = "varray_symlinks"
skip_memory_check: bool = False
skip_collection_create_memory_check: bool = False

@property
def as_dict(self) -> dict:
Expand Down
19 changes: 19 additions & 0 deletions docs/deker/fine_tuning.rst
Original file line number Diff line number Diff line change
Expand Up @@ -93,9 +93,12 @@ If you need to change it on the fly, you may use the following function::
set_logging_level("INFO") # now DEKER™ logs starting from "INFO" level



``memory_limit``
----------------

.. _memory_limit:

This parameter is used for the early run time break in case of potential memory overflow.

DEKER™ operates big amounts of data, and you may be unaware that your machine will probably run out
Expand Down Expand Up @@ -142,6 +145,22 @@ suffix is ignored: ``"1024k"``, ``"512m"``, ``"8g"``, ``"1t"`` will work.
If you set a memory limit to your container, you'd better limit DEKER™ to the same value.
Otherwise your container may be killed because of memory overflow.

``skip_collection_create_memory_check``
---------------------------------------
Currently deker has 3 places, where memory check, described in `memory_limit`_:

* On collection creation via ``client.create_collection()``
* On getting subset e.g ``array[:]``
* On reading array as ``XArray`` e.g ``array[:].read_xarray()``

While, last two prevents memory overflow and are required,
sometimes you may need to be able to skip
first one (e.g when you work with cluster)

You can do so, by providing
``skip_collection_create_memory_check=True`` as argument to
the :meth:`Client <deker.client.Client>` constructor

HDF5 Options
============

Expand Down

0 comments on commit cc9c13d

Please sign in to comment.