Heap growth strategies

CHANGELOG.md

@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Added configurable logging macros to stm32 platform
 - Added support for ULP wakeup on ESP32
+- Added heap growth strategies as a fine-tuning option to `spawn_opt/2,4`
 
 ### Fixed
 

doc/src/memory-management.md

@@ -26,7 +26,7 @@ The heap and stack for each AtomVM process are stored in a single allocated bloc
 
 The heap contains all of the allocated terms in an execution context.  In some cases, the terms occupy more than one word of memory (e.g., a tuple), but in general, the heap contains a record of memory in use by the program.
 
-The heap grows incrementally, as memory is allocated, and terms are allocated sequentially, in increasing memory addresses.  There is, therefore, no memory fragmentation, properly speaking, at least insofar as a portion of memory might be in use and then freed.  However, it is possible that previously allocated blocks of memory in the context heap are no longer referenced by the program.  In this case, the allocated blocks are "garbage", and are reclaimed at the next garbage collection.
+The heap grows incrementally, as memory is allocated, and terms are allocated sequentially, in increasing memory addresses.  There is, therefore, no memory fragmentation, properly speaking, at least insofar as a portion of memory might be in use and then freed.  However, it is possible that previously allocated blocks of memory in the context heap are no longer referenced by the program.  In this case, the allocated blocks are "garbage", and are reclaimed at the next garbage collection. The actual growth of the heap is controlled by a heap growth strategy (`heap_growth` spawn option) as described below.
 
 > Note. It is possible for the AtomVM heap, as provided by the underlying operating system, to become fragmented, as the execution context stack and heap are allocated via `malloc` or equiv.  But that is a different kind of fragmentation that does not refer to the allocated block used by an individual AtomVM process.
 
@@ -64,7 +64,19 @@ The following diagram illustrates an allocated block of memory that stores terms
     |           word[n-1]            |      v      v
     +================================+ <- stack_base --
 
-The initial size of the allocated block for the stack and heap in AtomVM is 8 words.  As heap and stack allocations grow, eventually, the amount of free space will decrease to the point where a garbage collection is required.  In this case, a new but larger (typically by 2x) block of memory is allocated by the AtomVM OS process, and terms are copied from the old stack and heap to the new stack and heap.  Garbage collection is described in more detail below.
+The initial size of the allocated block for the stack and heap in AtomVM is 8 words.  As heap and stack allocations grow, eventually, the amount of free space will decrease to the point where a garbage collection is required.  In this case, a new but larger block of memory is allocated by the AtomVM OS process, and terms are copied from the old stack and heap to the new stack and heap.  Garbage collection is described in more detail below.
+
+### Heap growth strategies
+
+AtomVM aims at minimizing memory footprint and several heap growth strategies are available. The heap is grown or shrunk when an allocation is required and the current execution context allows for a garbage collection (that will move data structures), allows for shrinking or forces shrinking (typically in the case of a call to `erlang:garbage_collect/0,1`).
+
+Each strategy is set at the process level.
+
+Default strategy is bounded free (`{heap_growth, bounded_free}`). In this strategy, when more memory is required, the allocator keeps the free amount between fixed boundaries (currently 16 and 32 terms). If no allocation is required but free space is larger than boundary, a garbage collection is triggered. After copying data to a new heap, if the free space is larger than the maximum, the heap is shrunk within the boundaries.
+
+With minimum strategy (`{heap_growth, minimum}`), when an allocation can happen, it is always adjusted to have the free space at 0.
+
+With fibonacci strategy (`{heap_growth, fibonacci}`), heap size grows following a variation of fibonacci until a large value and then grows by 20%. If free space is larger than 75% of heap size, the heap is shrunk. This strategy is inspired from Erlang/OTP's implementation.
 
 ### Registers
 

libs/estdlib/src/erlang.erl

@@ -130,9 +130,15 @@
 
 -type demonitor_option() :: flush | {flush, boolean()} | info | {info, boolean()}.
 
+-type heap_growth_strategy() ::
+    bounded_free
+    | minimum
+    | fibonacci.
+
 -type spawn_option() ::
     {min_heap_size, pos_integer()}
     | {max_heap_size, pos_integer()}
+    | {atomvm_heap_growth, heap_growth_strategy()}
     | link
     | monitor.
 
@@ -206,7 +212,8 @@ send_after(Time, Dest, Msg) ->
 %%
 %% The following keys are supported:
 %% <ul>
-%%      <li><b>heap_size</b> the number of words used in the heap (integer)</li>
+%%      <li><b>heap_size</b> the number of words used in the heap (integer), including the stack but excluding fragments</li>
+%%      <li><b>total_heap_size</b> the number of words used in the heap (integer) including fragments</li>
 %%      <li><b>stack_size</b> the number of words used in the stack (integer)</li>
 %%      <li><b>message_queue_len</b> the number of messages enqueued for the process (integer)</li>
 %%      <li><b>memory</b> the estimated total number of bytes in use by the process (integer)</li>
@@ -218,6 +225,7 @@ send_after(Time, Dest, Msg) ->
 %%-----------------------------------------------------------------------------
 -spec process_info
     (Pid :: pid(), heap_size) -> {heap_size, non_neg_integer()};
+    (Pid :: pid(), total_heap_size) -> {total_heap_size, non_neg_integer()};
     (Pid :: pid(), stack_size) -> {stack_size, non_neg_integer()};
     (Pid :: pid(), message_queue_len) -> {message_queue_len, non_neg_integer()};
     (Pid :: pid(), memory) -> {memory, non_neg_integer()};

src/libAtomVM/context.c

@@ -66,6 +66,7 @@ Context *context_new(GlobalContext *glb)
 
     ctx->min_heap_size = 0;
     ctx->max_heap_size = 0;
+    ctx->heap_growth_strategy = BoundedFreeHeapGrowth;
     ctx->has_min_heap_size = 0;
     ctx->has_max_heap_size = 0;
 
@@ -236,6 +237,7 @@ bool context_get_process_info(Context *ctx, term *out, term atom_key)
     size_t ret_size;
     switch (atom_key) {
         case HEAP_SIZE_ATOM:
+        case TOTAL_HEAP_SIZE_ATOM:
         case STACK_SIZE_ATOM:
         case MESSAGE_QUEUE_LEN_ATOM:
         case MEMORY_ATOM:
@@ -268,7 +270,15 @@ bool context_get_process_info(Context *ctx, term *out, term atom_key)
         // heap_size size in words of the heap of the process
         case HEAP_SIZE_ATOM: {
             term_put_tuple_element(ret, 0, HEAP_SIZE_ATOM);
-            unsigned long value = memory_heap_memory_size(&ctx->heap) - context_stack_size(ctx);
+            unsigned long value = memory_heap_youngest_size(&ctx->heap);
+            term_put_tuple_element(ret, 1, term_from_int32(value));
+            break;
+        }
+
+        // total_heap_size size in words of the heap of the process, including fragments
+        case TOTAL_HEAP_SIZE_ATOM: {
+            term_put_tuple_element(ret, 0, TOTAL_HEAP_SIZE_ATOM);
+            unsigned long value = memory_heap_memory_size(&ctx->heap);
             term_put_tuple_element(ret, 1, term_from_int32(value));
             break;
         }

src/libAtomVM/context.h

@@ -67,6 +67,13 @@ enum ContextFlags
     Trap = 32,
 };
 
+enum HeapGrowthStrategy
+{
+    BoundedFreeHeapGrowth = 0,
+    MinimumHeapGrowth,
+    FibonacciHeapGrowth
+};
+
 // Max number of x(N) & fr(N) registers
 // BEAM sets this to 1024.
 #define MAX_REG 16
@@ -92,6 +99,7 @@ struct Context
 
     size_t min_heap_size;
     size_t max_heap_size;
+    enum HeapGrowthStrategy heap_growth_strategy;
 
     unsigned long cp;
 

src/libAtomVM/defaultatoms.c

@@ -149,6 +149,12 @@ static const char *const kill_atom = "\x4" "kill";
 static const char *const killed_atom = "\x6" "killed";
 static const char *const links_atom = "\x5" "links";
 
+static const char *const total_heap_size_atom = "\xF" "total_heap_size";
+static const char *const atomvm_heap_growth_atom = "\x12" "atomvm_heap_growth";
+static const char *const bounded_free_atom = "\xC" "bounded_free";
+static const char *const minimum_atom = "\x7" "minimum";
+static const char *const fibonacci_atom = "\x9" "fibonacci";
+
 void defaultatoms_init(GlobalContext *glb)
 {
     int ok = 1;
@@ -282,6 +288,12 @@ void defaultatoms_init(GlobalContext *glb)
     ok &= globalcontext_insert_atom(glb, killed_atom) == KILLED_ATOM_INDEX;
     ok &= globalcontext_insert_atom(glb, links_atom) == LINKS_ATOM_INDEX;
 
+    ok &= globalcontext_insert_atom(glb, total_heap_size_atom) == TOTAL_HEAP_SIZE_ATOM_INDEX;
+    ok &= globalcontext_insert_atom(glb, atomvm_heap_growth_atom) == ATOMVM_HEAP_GROWTH_ATOM_INDEX;
+    ok &= globalcontext_insert_atom(glb, bounded_free_atom) == BOUNDED_FREE_ATOM_INDEX;
+    ok &= globalcontext_insert_atom(glb, minimum_atom) == MINIMUM_ATOM_INDEX;
+    ok &= globalcontext_insert_atom(glb, fibonacci_atom) == FIBONACCI_ATOM_INDEX;
+
     if (!ok) {
         AVM_ABORT();
     }

src/libAtomVM/defaultatoms.h

@@ -158,7 +158,13 @@ extern "C" {
 #define KILLED_ATOM_INDEX 101
 #define LINKS_ATOM_INDEX 102
 
-#define PLATFORM_ATOMS_BASE_INDEX 103
+#define TOTAL_HEAP_SIZE_ATOM_INDEX 103
+#define ATOMVM_HEAP_GROWTH_ATOM_INDEX 104
+#define BOUNDED_FREE_ATOM_INDEX 105
+#define MINIMUM_ATOM_INDEX 106
+#define FIBONACCI_ATOM_INDEX 107
+
+#define PLATFORM_ATOMS_BASE_INDEX 108
 
 #define FALSE_ATOM TERM_FROM_ATOM_INDEX(FALSE_ATOM_INDEX)
 #define TRUE_ATOM TERM_FROM_ATOM_INDEX(TRUE_ATOM_INDEX)
@@ -291,6 +297,12 @@ extern "C" {
 #define KILLED_ATOM TERM_FROM_ATOM_INDEX(KILLED_ATOM_INDEX)
 #define LINKS_ATOM TERM_FROM_ATOM_INDEX(LINKS_ATOM_INDEX)
 
+#define TOTAL_HEAP_SIZE_ATOM TERM_FROM_ATOM_INDEX(TOTAL_HEAP_SIZE_ATOM_INDEX)
+#define ATOMVM_HEAP_GROWTH_ATOM TERM_FROM_ATOM_INDEX(ATOMVM_HEAP_GROWTH_ATOM_INDEX)
+#define BOUNDED_FREE_ATOM TERM_FROM_ATOM_INDEX(BOUNDED_FREE_ATOM_INDEX)
+#define MINIMUM_ATOM TERM_FROM_ATOM_INDEX(MINIMUM_ATOM_INDEX)
+#define FIBONACCI_ATOM TERM_FROM_ATOM_INDEX(FIBONACCI_ATOM_INDEX)
+
 void defaultatoms_init(GlobalContext *glb);
 
 void platform_defaultatoms_init(GlobalContext *glb);

src/libAtomVM/memory.c

@@ -34,6 +34,7 @@
 //#define ENABLE_TRACE
 
 #include "trace.h"
+#include "utils.h"
 
 #ifndef MAX
 #define MAX(a, b) ((a) > (b) ? (a) : (b))
@@ -101,6 +102,22 @@ enum MemoryGCResult memory_erl_nif_env_ensure_free(ErlNifEnv *env, size_t size)
     return MEMORY_GC_OK;
 }
 
+// Follow Erlang/OTP 18 fibonacci series.
+static size_t next_fibonacci_heap_size(size_t size)
+{
+    static const size_t fib_seq[] = { 12, 38, 51, 90, 142, 233, 376, 610, 987, 1598, 2586, 4185, 6772, 10958,
+        17731, 28690, 46422, 75113, 121536, 196650, 318187, 514838, 833026,
+        1347865, 2180892, 3528758, 5709651 };
+    for (size_t i = 0; i < sizeof(fib_seq) / sizeof(fib_seq[0]); i++) {
+        if (size <= fib_seq[i]) {
+            return fib_seq[i];
+        }
+    }
+    return size + size / 5;
+}
+
+#define FIBONACCI_HEAP_GROWTH_REDUCTION_THRESHOLD 10000
+
 enum MemoryGCResult memory_ensure_free_with_roots(Context *c, size_t size, size_t num_roots, term *roots, enum MemoryAllocMode alloc_mode)
 {
     size_t free_space = context_avail_free_memory(c);
@@ -109,22 +126,87 @@ enum MemoryGCResult memory_ensure_free_with_roots(Context *c, size_t size, size_
             return memory_heap_alloc_new_fragment(&c->heap, size);
         }
     } else {
-        size_t maximum_free_space = 2 * (size + MIN_FREE_SPACE_SIZE);
-        if (free_space < size || (alloc_mode == MEMORY_FORCE_SHRINK) || ((alloc_mode == MEMORY_CAN_SHRINK) && free_space > maximum_free_space)) {
-            size_t memory_size = memory_heap_memory_size(&c->heap);
-            if (UNLIKELY(memory_gc(c, memory_size + size + MIN_FREE_SPACE_SIZE, num_roots, roots) != MEMORY_GC_OK)) {
-                // TODO: handle this more gracefully
-                TRACE("Unable to allocate memory for GC.  memory_size=%zu size=%u\n", memory_size, size);
-                return MEMORY_GC_ERROR_FAILED_ALLOCATION;
+        // Target heap size depends on:
+        // - alloc_mode (MEMORY_FORCE_SHRINK takes precedence)
+        // - heap growth strategy
+        bool should_gc = free_space < size || (alloc_mode == MEMORY_FORCE_SHRINK);
+        size_t memory_size = 0;
+        if (!should_gc) {
+            switch (c->heap_growth_strategy) {
+                case BoundedFreeHeapGrowth: {
+                    size_t maximum_free_space = 2 * (size + MIN_FREE_SPACE_SIZE);
+                    should_gc = ((alloc_mode == MEMORY_CAN_SHRINK) && free_space - size > maximum_free_space);
+                } break;
+                case MinimumHeapGrowth:
+                    should_gc = ((alloc_mode == MEMORY_CAN_SHRINK) && free_space - size > 0);
+                    break;
+                case FibonacciHeapGrowth: {
+                    memory_size = memory_heap_memory_size(&c->heap);
+                    should_gc = ((alloc_mode == MEMORY_CAN_SHRINK) && free_space - size > 3 * memory_size / 4);
+                    break;
+                }
+            }
+        }
+        if (should_gc) {
+            if (memory_size == 0) {
+                memory_size = memory_heap_memory_size(&c->heap);
+            }
+            size_t target_size;
+            switch (c->heap_growth_strategy) {
+                case BoundedFreeHeapGrowth:
+                    if (free_space < size) {
+                        target_size = memory_size + size + MIN_FREE_SPACE_SIZE;
+                    } else {
+                        size_t maximum_free_space = 2 * (size + MIN_FREE_SPACE_SIZE);
+                        target_size = memory_size - free_space + maximum_free_space;
+                    }
+                    break;
+                case MinimumHeapGrowth:
+                    target_size = memory_size - free_space + size;
+                    break;
+                case FibonacciHeapGrowth:
+                    target_size = next_fibonacci_heap_size(memory_size - free_space + size);
+                    break;
+                default:
+                    UNREACHABLE();
             }
-            if (alloc_mode != MEMORY_NO_SHRINK) {
+            target_size = MAX(c->has_min_heap_size ? c->min_heap_size : 0, target_size);
+            if (target_size != memory_size) {
+                if (UNLIKELY(memory_gc(c, target_size, num_roots, roots) != MEMORY_GC_OK)) {
+                    // TODO: handle this more gracefully
+                    TRACE("Unable to allocate memory for GC.  target_size=%zu\n", target_size);
+                    return MEMORY_GC_ERROR_FAILED_ALLOCATION;
+                }
+                should_gc = alloc_mode == MEMORY_FORCE_SHRINK;
+                size_t new_memory_size = memory_heap_memory_size(&c->heap);
+                size_t new_target_size = new_memory_size;
                 size_t new_free_space = context_avail_free_memory(c);
-                if (new_free_space > maximum_free_space) {
-                    size_t new_memory_size = memory_heap_memory_size(&c->heap);
-                    size_t new_requested_size = (new_memory_size - new_free_space) + maximum_free_space;
-                    if (!c->has_min_heap_size || (c->min_heap_size < new_requested_size)) {
-                        if (UNLIKELY(memory_gc(c, new_requested_size, num_roots, roots) != MEMORY_GC_OK)) {
-                            TRACE("Unable to allocate memory for GC shrink.  new_memory_size=%zu new_free_space=%zu new_minimum_free_space=%zu size=%u\n", new_memory_size, new_free_space, maximum_free_space, size);
+                switch (c->heap_growth_strategy) {
+                    case BoundedFreeHeapGrowth: {
+                        size_t maximum_free_space = 2 * (size + MIN_FREE_SPACE_SIZE);
+                        should_gc = should_gc || (alloc_mode != MEMORY_NO_SHRINK && new_free_space > maximum_free_space);
+                        if (should_gc) {
+                            new_target_size = (new_memory_size - new_free_space) + maximum_free_space;
+                        }
+                    } break;
+                    case MinimumHeapGrowth:
+                        should_gc = should_gc || (alloc_mode != MEMORY_NO_SHRINK && new_free_space > 0);
+                        if (should_gc) {
+                            new_target_size = new_memory_size - new_free_space + size;
+                        }
+                        break;
+                    case FibonacciHeapGrowth:
+                        should_gc = should_gc || (new_memory_size > FIBONACCI_HEAP_GROWTH_REDUCTION_THRESHOLD && new_free_space >= 3 * new_memory_size / 4);
+                        if (should_gc) {
+                            new_target_size = next_fibonacci_heap_size(new_memory_size - new_free_space + size);
+                        }
+                        break;
+                }
+                if (should_gc) {
+                    new_target_size = MAX(c->has_min_heap_size ? c->min_heap_size : 0, new_target_size);
+                    if (new_target_size != new_memory_size) {
+                        if (UNLIKELY(memory_gc(c, new_target_size, num_roots, roots) != MEMORY_GC_OK)) {
+                            TRACE("Unable to allocate memory for GC shrink.  new_memory_size=%zu new_free_space=%zu size=%u\n", new_memory_size, new_free_space, size);
                             return MEMORY_GC_ERROR_FAILED_ALLOCATION;
                         }
                     }

src/libAtomVM/memory.h

@@ -151,6 +151,18 @@ static inline size_t memory_heap_fragment_memory_size(const HeapFragment *fragme
     return result;
 }
 
+/**
+ * @brief return the size of the youngest generation of the heap.
+ * @details in some condition, this function returns the size of a fragment
+ * where the stack is not.
+ * @param heap the heap to get the youngest size of
+ * @returns the size in terms
+ */
+static inline size_t memory_heap_youngest_size(const Heap *heap)
+{
+    return heap->heap_end - heap->heap_start;
+}
+
 /**
  * @brief return the total memory size of a heap, including fragments.
  *
@@ -159,7 +171,7 @@ static inline size_t memory_heap_fragment_memory_size(const HeapFragment *fragme
  */
 static inline size_t memory_heap_memory_size(const Heap *heap)
 {
-    size_t result = heap->heap_end - heap->heap_start;
+    size_t result = memory_heap_youngest_size(heap);
     if (heap->root->next) {
         result += memory_heap_fragment_memory_size(heap->root->next);
     }