Replace func trie with hashmap

[ChAoSUnItY]

Previously, trie implementation is not consistent, mainly because of using index to point the referencing func_t to FUNCS, additionally, it lacks of dynamic allocation which might cause segmentation fault and results more technical debt to debug on either FUNCS or FUNCS_TRIE. Thus, in this PR, we can resolve this issue by introducing a dynamic hashmap.

Current implementation is using FNV-1a hashing algorithm (32-bit edition to be precise), and due to lack of unsigned integer implementation, hashing result ranges from 0 to 2,147,483,647.

Notice that current implementation may suffer from lookup issue when the function amount keeps increasing since current hashmap implementation doesn't offer rehashing based on load factor (which ideally, 0.75 would be best and currently shecc does not support floating number).

This also enables us to refactor more structures later with hashmap implementation in shecc.

Benchmark for ./tests/hello.c compilation

Before

Command being timed: "./out/shecc tests/hello.c"
        User time (seconds): 0.00
        System time (seconds): 0.02
        Percent of CPU this job got: 76%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.03
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 52112
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 12220
        Voluntary context switches: 0
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 0
        File system outputs: 32
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

After

Command being timed: "./out/shecc tests/hello.c"
        User time (seconds): 0.00
        System time (seconds): 0.02
        Percent of CPU this job got: 71%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.03
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 49916
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 12224
        Voluntary context switches: 1
        Involuntary context switches: 0
        Swaps: 0
        File system inputs: 8
        File system outputs: 32
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Summary by Bito

This pull request replaces the trie structure for function lookups with a dynamic hashmap, improving memory management and addressing segmentation faults. The new implementation uses the FNV-1a hashing algorithm but currently lacks dynamic resizing, which may impact performance as function count increases. Overall, it simplifies function storage and sets the stage for future enhancements.

Unit tests added: False

Estimated effort to review (1-5, lower is better): 2

[jserv]

@visitorckw, can you comment this?

[visitorckw]

Looks good as is.

However, as mentioned, a large number of functions may cause excessive collisions and slow down performance. For smaller function counts, the default 512 buckets might be overkill. Therefore, a radix tree with dynamic memory allocation could still be a method worth exploring in the future.

[ChAoSUnItY]

I'm concerning that dynamic memory allocation at this moment is not reliable and potentially flawed, I've attempted to implement rehashing algorithm before, but on stage 2 the compilation will fail, while the GCC and stage 1 are fine.

[visitorckw]

The multiplication here may cause overflow, leading to undefined behavior. Signed integer overflow is undefined, while unsigned integer overflow is not. Since shecc currently lacks support for unsigned integers, we might consider adding it to address this issue.

[ChAoSUnItY]

I think we can just simply add unsigned type at this moment to simplify the effort of new type? This way, unsigned can still uses current signed's arithmetic algorithm (due to the fact that they both based on 2's complement), since in ARMv7 and RISC-V 32bit assembly, signed overflow is well-defined, the only difference would be intrepretation of most-significant bit.

Edit: I just realized we still need to handle comparison, but I prefer to defer unsigned integer feature since we have an ongoing project which requires full resolution of shecc's specification, and which doesn't include unsigned types at this moment. I think this addition would alters the simplicity of project. @jserv should we postpone this hashmap implementation?

[jserv]

should we postpone this hashmap implementation?

You can simply convert this pull request to draft state.

[ChAoSUnItY]

I think this pull request should be implemented as soon as possible, the reason is that I'm currently working on type_t refactor, but I have encountered the weird free(): invalid pointer issue when adding functions in globals.c, which I assume the reason is that the function trie cannot hold more than certain numbers of functions and probably function name's length could also contribute to this issue, these 2 factors and the flaw is already described here:

shecc/src/globals.c

Lines 103 to 109 in 09bb918

	if (!trie->next[fc]) {
	/* FIXME: The func_tries_idx variable may exceed the maximum number,
	* which can lead to a segmentation fault. This issue is affected by
	* the number of functions and the length of their names. The proper
	* way to handle this is to dynamically allocate a new element.
	*/
	trie->next[fc] = func_tries_idx++;

But after cherry-picked this branch without any changes to function structures, the issue immediately gone.

One possible solution towards this is to add -fwrapv compilation flag to gcc to instruct compiler to wrap signed integer overflow result according to the 2's compliment representation, this ensures defined behavior at least when compiling with gcc. Meanwhile in shecc, it's fine at this moment since both ARM 32bit and RISC-V 32bit assembly wraps the overflow value according to the 2's compliment representation as well.

[vacantron]

Could enlarging limitation in refs.h temporarily fix this problem? I remember the trie count is almost reaching the limitation in the last change.

This workaround could be remove after applying this patch.

[ChAoSUnItY]

Changing macro MAX_FUNC_TRIES in defs.h to 3000 does the trick.

[visitorckw]

I think this pull request should be implemented as soon as possible, the reason is that I'm currently working on type_t refactor, but I have encountered the weird free(): invalid pointer issue when adding functions in globals.c, which I assume the reason is that the function trie cannot hold more than certain numbers of functions and probably function name's length could also contribute to this issue, these 2 factors and the flaw is already described here:

shecc/src/globals.c

Lines 103 to 109 in 09bb918

	if (!trie->next[fc]) {
	/* FIXME: The func_tries_idx variable may exceed the maximum number,
	* which can lead to a segmentation fault. This issue is affected by
	* the number of functions and the length of their names. The proper
	* way to handle this is to dynamically allocate a new element.
	*/
	trie->next[fc] = func_tries_idx++;

I don't necessarily oppose this PR. However, if the issue is that MAX_FUNC_TRIES is too small, causing an out-of-bounds array access, it seems unrelated to switching to a hash table instead of a trie. A hash table can also use an array, and a trie can use dynamic memory allocation. This feels more like adding a new feature unrelated to fixing the bug itself. But I'm fine if we decide to switch to a hash table as a workaround.

[ChAoSUnItY]

The major reason that I would like to replace trie is that some errors are not straightforward to be realized, in this case, it generates free(): invalid pointer, which isn't that obvious to do with insufficient trie size and is not friendly to new-coming contributors in my opinion.

[bito-code-review]

Code Review Agent Run #66bd40

Actionable Suggestions - 4

• src/globals.c - 4

  • Consider adding buckets allocation error check · Line 125-125
  • Missing format specifier in printf call · Line 157-157
  • Consider adding error handling for FUNCS_MAP · Line 713-714
  • Consider adding null check in hashmap_free · Line 746-747

Review Details

• Files reviewed - 1 · Commit Range: b14a10d..35671a8

  • src/globals.c
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • Fb Infer (Static Code Analysis) - ✖︎ Failed

---

AI Code Review powered by

[bito-code-review]

Missing format specifier in printf call

The printf format string is missing the format specifier for len. Consider updating to include %d in the format string.

Code suggestion

Check the AI-generated fix before applying

Suggested change

	printf("Failed to allocate hashmap_node_t key with size %d\n");
	printf("Failed to allocate hashmap_node_t key with size %d\n", len);

Code Review Run #66bd40

---

Is this a valid issue, or was it incorrectly flagged by the Agent?

• it was incorrectly flagged

[bito-code-review]

Code Review Agent Run #4b564e

Actionable Suggestions - 7

• src/globals.c - 7

  • Missing overflow checks before allocations · Line 137-159
  • Consider adding memory cleanup on failure · Line 127-130
  • Consider freeing map on allocation failure · Line 127-130
  • Consider using unsigned integers for hashing · Line 82-92
  • Consider handling duplicate keys in hashmap_put · Line 185-193
  • Consider restructuring hashmap node freeing logic · Line 244-246
  • Consider adding null check for FUNCS_MAP · Line 480-480

Additional Suggestions - 5

• src/globals.c - 3

  • Consider validating size parameter · Line 82-82
  • Consider initializing hashmap pointer to NULL · Line 24-24
  • Consider using NULL for pointer init · Line 132-133
• src/defs.h - 1

  • Consider adding capacity field to hashmap · Line 316-318
• Makefile - 1

  • Consider separating security compiler flags · Line 4-4

Review Details

• Files reviewed - 3 · Commit Range: f1fb5df..f1fb5df

  • Makefile
  • src/defs.h
  • src/globals.c
• Files skipped - 0
• Tools

  • Whispers (Secret Scanner) - ✔︎ Successful
  • Detect-secrets (Secret Scanner) - ✔︎ Successful
  • Fb Infer (Static Code Analysis) - ✖︎ Failed

---

AI Code Review powered by

[jserv]

Thank @ChAoSUnItY for contributing!