Merge pull request #113 from douzzer/20231024-wolfsentry-doc-sync

wolfSentry update for release 1.6.

wolfSentry/src/ChangeLog.md

@@ -2,6 +2,112 @@
 
 <br>
 
+# wolfSentry Release 1.6.0 (October 24, 2023)
+
+Release 1.6.0 of the wolfSentry embedded firewall/IDPS has enhancements,
+additions, and improvements including:
+
+## New Features
+
+This release adds native support for the CAN bus address family, and for
+bitmask-based address matching.  CAN addresses and bitmasks are now handled in
+configuration JSON, as numbers in decimal, octal, or hexadecimal, supporting
+both 11 bit (part A) and 29 bit (part B) identifiers.
+
+
+## Noteworthy Changes and Additions
+
+`wolfsentry/wolfsentry.h`:
+
+* Add `WOLFSENTRY_ROUTE_FLAG_REMOTE_ADDR_BITMASK` and `WOLFSENTRY_ROUTE_FLAG_LOCAL_ADDR_BITMASK` to `wolfsentry_route_flags_t`.
+* Add `WOLFSENTRY_ACTION_RES_USER0`-`WOLFSENTRY_ACTION_RES_USER6` to `wolfsentry_action_res_t` `enum`, add `WOLFSENTRY_ACTION_RES_USER7` macro, and refactor `WOLFSENTRY_ACTION_RES_USER_BASE` as a macro aliased to `WOLFSENTRY_ACTION_RES_USER0`.
+* Remove !`WOLFSENTRY_NO_STDIO` gate around `wolfsentry_kv_render_value()`.
+
+`wolfsentry/wolfsentry_settings.h`:
+
+* Rename `WOLFSENTRY_NO_STDIO` to `WOLFSENTRY_NO_STDIO_STREAMS`.
+* Rename `WOLFSENTRY_HAVE_NONGNU_ATOMICS` to `WOLFSENTRY_NO_GNU_ATOMICS`.
+* Added handling for `WOLFSENTRY_NO_SEM_BUILTIN`, `WOLFSENTRY_NO_ADDR_BITMASK_MATCHING`, and `WOLFSENTRY_NO_IPV6`.
+* Gate inclusion of `stdio.h` on !`WOLFSENTRY_NO_STDIO_H`, formerly !`WOLFSENTRY_NO_STDIO`.
+* Added `WOLFSENTRY_CONFIG_FLAG_ADDR_BITMASKS`, and rename `WOLFSENTRY_CONFIG_FLAG_NO_STDIO` to `WOLFSENTRY_CONFIG_FLAG_NO_STDIO_STREAMS`.
+
+`src/addr_families.c` and `wolfsentry/wolfsentry_af.h`:  Split `WOLFSENTRY_AF_LINK` into `WOLFSENTRY_AF_LINK48` and `WOLFSENTRY_AF_LINK64`, with `WOLFSENTRY_AF_LINK` aliased to `WOLFSENTRY_AF_LINK48`.
+
+`src/kv.c`: remove !`WOLFSENTRY_NO_STDIO` gate around `wolfsentry_kv_render_value()`.
+
+`src/json/load_config.c`: In `convert_sockaddr_address()`, add separate handling for `WOLFSENTRY_AF_LINK48` and `WOLFSENTRY_AF_LINK64`.
+
+`Makefile`:
+
+* Refactor `NO_STDIO`, `NO_JSON`, `NO_JSON_DOM`, `SINGLETHREADED`, `STATIC`, and `STRIPPED` to pivot on definedness, not oneness.
+* Add feature flags `NO_ADDR_BITMASK_MATCHING` and `NO_IPV6`.
+* Rename feature flag `NO_STDIO` to `NO_STDIO_STREAMS`.
+
+
+## Performance Improvements
+
+`src/routes.c`: Added AF-mismatch optimization to `wolfsentry_route_lookup_0()`.
+
+
+## Documentation
+
+Add inline documentation for `WOLFSENTRY_NO_GETPROTOBY`, `WOLFSENTRY_SEMAPHORE_INCLUDE`, `WOLFSENTRY_THREAD_INCLUDE`, `WOLFSENTRY_THREAD_ID_T`, and `WOLFSENTRY_THREAD_GET_ID_HANDLER`.
+
+`doc/json_configuration.md`: add documentation and ABNF grammar for `"bitmask"` node in route endpoints.
+
+
+## Bug Fixes and Cleanups
+
+Fixes for user settings file handling:
+
+* Don't `#include <wolfsentry/wolfsentry_options.h>` if `defined(WOLFSENTRY_USER_SETTINGS_FILE)`.
+* Generate and install `wolfsentry/wolfsentry_options.h` only if `USER_SETTINGS_FILE` is undefined, and if `USER_SETTINGS_FILE` is defined, depend on it where previously the dependency was unconditionally on `wolfsentry/wolfsentry_options.h`.
+* If `USER_SETTINGS_FILE` is set search it to derive JSON build settings.
+
+`Makefile`: Don't add `-pthread` to `LDFLAGS` if `RUNTIME` is `FreeRTOS-lwIP`.
+
+`wolfsentry/wolfsentry_settings.h`:
+
+* Eliminate inclusion of `errno.h` -- now included only in source files that need it.
+* Fix handling for `WOLFSENTRY_SEMAPHORE_INCLUDE` to give it effect in all code paths (previously ignored in POSIX and FreeRTOS paths).
+
+`src/routes.c`:
+
+* in `wolfsentry_route_event_dispatch_0()`, move update of `meta.purge_after` inside the mutex.
+* in `wolfsentry_route_get_metadata()`, conditionalize use of 64 bit `WOLFSENTRY_ATOMIC_LOAD()` on pointer size, to avoid dependency on library implementation of `__atomic_load_8()`.
+
+`src/wolfsentry_internal.c`: fix use-after-free bug in `wolfsentry_table_free_ents()`, using new `table->coupled_ent_fn` mechanism.
+
+`src/json/load_config.c`: In `convert_sockaddr_address()`, handle `sa->addr_len` consistently -- don't overwrite nonzero values.
+
+`src/json/{centijson_dom.c,centijson_sax.c,centijson_value.c}`: eliminate direct calls to heap allocator functions in `WOLFSENTRY` code paths, i.e. use only `wolfsentry_allocator`.
+
+`src/json/centijson_value.c`: fix uninited-variable defect on `cmp` in `json_value_dict_get_or_add_()`.
+
+
+## Self-Test Enhancements
+
+Makefile.analyzers new and enhanced test targets:
+
+* `user-settings-build-test`: construct a user settings file, then build and self-test using it.
+* `library-dependency-singlethreaded-build-test` and `library-dependency-multithreaded-build-test`: comprehensive check for unexpected unresolved symbols in the library.
+* `no-addr-bitmask-matching-test`, `no-ipv6-test`, `linux-lwip-test-no-ipv6`: tests for new feature gates.
+* `freertos-arm32-build-test`: newly refactored to perform a final link of `test_lwip` kernel using lwIP and FreeRTOS kernel files and newlib-nano, followed by a check on the size of the kernel.
+
+Added `wolfsentry/wolfssl_test.h`, containing self-test and example logic relocated from `wolfssl/wolfssl/test.h` verbatim.
+
+`tests/test-config*.json`: added several bitmask-matched routes, added several diagnostic events (`"set-user-0"` through `"set-user-4"`), and added no-bitmasks and no-ipv6 variants.  Also removed AF-wildcard route from `tests/test-config-numeric.json` to increase test coverage.
+
+`tests/unittests.c`:
+
+* Additional tweaks for portability to 32 bit FreeRTOS
+* Add FreeRTOS-specific implementations of `test_lwip()` and `main()`.
+* In `test_json()`, add `wolfsentry_addr_family_handler_install(...,"my_AF2",...)`.
+* In `test_json()`, add bitmask tests.
+* Added stub implementations for various FreeRTOS/newlib dependencies to support final link in `freertos-arm32-build-test` target.
+
+<br>
+
 # wolfSentry Release 1.5.0 (September 13, 2023)
 
 Release 1.5.0 of the wolfSentry embedded firewall/IDPS has enhancements,

wolfSentry/src/README.md

@@ -46,19 +46,18 @@ be generated from the top of the wolfSentry source tree with `make doc-html`.
 This, and the source code itself, are the recommended API references.
 
 The PDF version of the API reference manual is pregenerated and included with source
-distributions in the `doc/` subdirectory.  The latest version is always
+distributions in the `doc/` subdirectory at `doc/wolfSentry_refman.pdf`.  The latest version is always
 available [on GitHub](https://raw.githubusercontent.com/wolfSSL/wolfsentry/master/doc/wolfSentry_refman.pdf).
 
 
+
 ## Dependencies
 
 In its default build, wolfSentry depends on a POSIX runtime, specifically the
 heap allocator, clock_gettime, stdio, semaphore, pthreads, and string APIs.
 However, these dependencies can be avoided with various build-time options.  The recipe
-<br>
-```
-make STATIC=1 SINGLETHREADED=1 NO_STDIO=1 EXTRA_CFLAGS='-DWOLFSENTRY_NO_CLOCK_BUILTIN -DWOLFSENTRY_NO_MALLOC_BUILTIN'
-```
+
+`make STATIC=1 SINGLETHREADED=1 NO_STDIO=1 EXTRA_CFLAGS="-DWOLFSENTRY_NO_CLOCK_BUILTIN -DWOLFSENTRY_NO_MALLOC_BUILTIN"`
 
 builds a libwolfsentry.a that depends on only a handful of basic string
 functions and the `inet_ntop()` library function (from POSIX.1-2001, and also
@@ -102,43 +101,47 @@ topic.
 
 | `make` Option | Macro Option | Description |
 | -------------- | ------------ | ----------- |
-| `V` ||  Verbose `make` output <br> e.g. `make V=1 -j test` |
-| `USER_MAKE_CONF` || User-defined make clauses to include at the top of the main Makefile <br> e.g. `make -j USER_MAKE_CONF=Makefile.settings` |
-| `EXTRA_CFLAGS` || Additional arguments to be passed verbatim to the compiler |
-| `EXTRA_LDFLAGS` || Additional arguments to be passed verbatim to the linker |
-| `SRC_TOP` || The source code top level directory (default `pwd -P`) |
-| `BUILD_TOP` || Build with artifacts in an alternate location (outside or in a subdirectory of the source tree) <br> e.g. `make BUILD_TOP=./build -j test`|
-| `DEBUG` || Compiler debugging flag to use (default `-ggdb`) |
-| `OPTIM` || The optimizer flag to use (default `-O3`) |
-| `HOST` || The target host tuple, for cross-compilation (default unset, i.e. native targeting) |
-| `RUNTIME` || The target runtime ecosystem -- default unset, `FreeRTOS-lwIP` and `Linux-lwIP` are recognized |
-| `C_WARNFLAGS` || The warning flags to use (overriding the generally applicable defaults) |
-| `STATIC` || Build statically linked unit tests |
-| `STRIPPED` || Strip binaries of debugging symbols |
-| `BUILD_DYNAMIC` || Build dynamically linked library |
-| `VERY_QUIET` || Inhibit all non-error output during build |
-| `TAR` || Path to GNU tar binary for `make dist`, should be set to `gtar` for macOS |
-| `VERSION` || The version to package for `make dist` |
+| `V` | |  Verbose `make` output <br> e.g. `make V=1 -j test` |
+| `USER_MAKE_CONF` | | User-defined make clauses to include at the top of the main Makefile <br> e.g. `make -j USER_MAKE_CONF=Makefile.settings` |
+| `EXTRA_CFLAGS` | | Additional arguments to be passed verbatim to the compiler |
+| `EXTRA_LDFLAGS` | | Additional arguments to be passed verbatim to the linker |
+| `SRC_TOP` | | The source code top level directory (default `pwd -P`) |
+| `BUILD_TOP` | | Build with artifacts in an alternate location (outside or in a subdirectory of the source tree) <br> e.g. `make BUILD_TOP=./build -j test`|
+| `DEBUG` | | Compiler debugging flag to use (default `-ggdb`) |
+| `OPTIM` | | The optimizer flag to use (default `-O3`) |
+| `HOST` | | The target host tuple, for cross-compilation (default unset, i.e. native targeting) |
+| `RUNTIME` | | The target runtime ecosystem -- default unset, `FreeRTOS-lwIP` and `Linux-lwIP` are recognized |
+| `C_WARNFLAGS` | | The warning flags to use (overriding the generally applicable defaults) |
+| `STATIC` | | Build statically linked unit tests |
+| `STRIPPED` | | Strip binaries of debugging symbols |
+| `BUILD_DYNAMIC` | | Build dynamically linked library |
+| `VERY_QUIET` | | Inhibit all non-error output during build |
+| `TAR` | | Path to GNU tar binary for `make dist`, should be set to `gtar` for macOS |
+| `VERSION` | | The version to package for `make dist` |
 | `LWIP` | `WOLFSENTRY_LWIP` | True/false -- Activates appropriate build settings for lwIP |
-| `NO_STDIO` | `WOLFSENTRY_NO_STDIO` | Define to omit functionality that depends on `stdio` |
+| `NO_STDIO_STREAMS` | `WOLFSENTRY_NO_STDIO_STREAMS` | Define to omit functionality that depends on `stdio` stream I/O |
+| | `WOLFSENTRY_NO_STDIO_H` | Define to inhibit inclusion of `stdio.h` |
+| `NO_ADDR_BITMASK_MATCHING` | `WOLFSENTRY_NO_ADDR_BITMASK_MATCHING` | Define to omit support for bitmask matching of addresses, i.e. support only prefix matching. |
+| `NO_IPV6` | `WOLFSENTRY_NO_IPV6` | Define to omit support for the IPv6 address family. |
 | `NO_JSON` | `WOLFSENTRY_NO_JSON` | Define to omit JSON configuration support |
 | `NO_JSON_DOM` | `WOLFSENTRY_NO_JSON_DOM` | Define to omit JSON DOM API |
 | `CALL_TRACE` | `WOLFSENTRY_DEBUG_CALL_TRACE` | Define to activate runtime call stack logging (profusely verbose) |
 | `USER_SETTINGS_FILE` | `WOLFSENTRY_USER_SETTINGS_FILE` | A substitute settings file, replacing autogenerated `wolfsentry_settings.h` |
 | `SINGLETHREADED` | `WOLFSENTRY_SINGLETHREADED` | Define to omit thread safety logic, and replace thread safety functions and macros with no-op macros. |
-|| `WOLFSENTRY_NO_PROTOCOL_NAMES` | If defined, omit APIs for rendering error codes and source code files in human readable form. They will be rendered numerically. |
-|| `WOLFSENTRY_NO_GETPROTOBY` | Define to disable lookup and rendering of protocols and services by name. |
-|| `WOLFSENTRY_NO_ERROR_STRINGS` | If defined, omit APIs for rendering error codes and source code files in human readable form. They will be rendered numerically. |
-|| `WOLFSENTRY_NO_MALLOC_BUILTINS` | If defined, omit built-in heap allocator primitives; the `wolfsentry_host_platform_interface` supplied to wolfSentry APIs must include implementations of all functions in `struct wolfsentry_allocator`. |
-|| `WOLFSENTRY_HAVE_NONGNU_ATOMICS` | Define if gnu-style atomic intrinsics are not available. `WOLFSENTRY_ATOMIC_*()` macro definitions for intrinsics will need to be supplied in `WOLFSENTRY_USER_SETTINGS_FILE` (see `wolfsentry_util.h`). |
-|| `WOLFSENTRY_NO_CLOCK_BUILTIN` | If defined, omit built-in time primitives; the `wolfsentry_host_platform_interface` supplied to wolfSentry APIs must include implementations of all functions in `struct wolfsentry_timecbs`. |
-|| `WOLFSENTRY_USE_NONPOSIX_SEMAPHORES` | Define if POSIX semaphore API is not available. If no non-POSIX builtin implementation is present in `wolfsentry_util.c`, then the `wolfsentry_host_platform_interface` supplied to wolfSentry APIs must include a full semaphore implementation (shim set) in its `wolfsentry_semcbs` slot. |
-|| `WOLFSENTRY_SEMAPHORE_INCLUDE` | Define to the path of a header file declaring a semaphore API. |
-|| `WOLFSENTRY_USE_NONPOSIX_THREADS` | Define if POSIX thread API is not available. `WOLFSENTRY_THREAD_INCLUDE`, `WOLFSENTRY_THREAD_ID_T`, and `WOLFSENTRY_THREAD_GET_ID_HANDLER` will need to be defined. |
-|| `WOLFSENTRY_THREAD_INCLUDE` | Define to the path of a header file declaring a threading API. |
-|| `WOLFSENTRY_THREAD_ID_T` | Define to the appropriate type analogous to POSIX `pthread_t`. |
-|| `WOLFSENTRY_THREAD_GET_ID_HANDLER` | Define to the name of a void function analogous to POSIX `pthread_self`, returning a value of type `WOLFSENTRY_THREAD_ID_T`. |
-|| `FREERTOS` | Build for FreeRTOS |
+| | `WOLFSENTRY_NO_PROTOCOL_NAMES` | If defined, omit APIs for rendering error codes and source code files in human readable form. They will be rendered numerically. |
+| | `WOLFSENTRY_NO_GETPROTOBY` | Define to disable lookup and rendering of protocols and services by name. |
+| | `WOLFSENTRY_NO_ERROR_STRINGS` | If defined, omit APIs for rendering error codes and source code files in human readable form. They will be rendered numerically. |
+| | `WOLFSENTRY_NO_MALLOC_BUILTINS` | If defined, omit built-in heap allocator primitives; the `wolfsentry_host_platform_interface` supplied to wolfSentry APIs must include implementations of all functions in `struct wolfsentry_allocator`. |
+| | `WOLFSENTRY_HAVE_NONGNU_ATOMICS` | Define if gnu-style atomic intrinsics are not available. `WOLFSENTRY_ATOMIC_*()` macro definitions for intrinsics will need to be supplied in `WOLFSENTRY_USER_SETTINGS_FILE` (see `wolfsentry_util.h`). |
+| | `WOLFSENTRY_NO_CLOCK_BUILTIN` | If defined, omit built-in time primitives; the `wolfsentry_host_platform_interface` supplied to wolfSentry APIs must include implementations of all functions in `struct wolfsentry_timecbs`. |
+| | `WOLFSENTRY_NO_SEM_BUILTIN` | If defined, omit built-in semaphore primitives; the `wolfsentry_host_platform_interface` supplied to wolfSentry APIs must include implementations of all functions in `struct wolfsentry_semcbs`. |
+| | `WOLFSENTRY_USE_NONPOSIX_SEMAPHORES` | Define if POSIX semaphore API is not available. If no non-POSIX builtin implementation is present in `wolfsentry_util.c`, then #WOLFSENTRY_NO_SEM_BUILTIN must be set, and the `wolfsentry_host_platform_interface` supplied to wolfSentry APIs must include a full semaphore implementation (shim set) in its `wolfsentry_semcbs` slot. |
+| | `WOLFSENTRY_SEMAPHORE_INCLUDE` | Define to the path of a header file declaring a semaphore API. |
+| | `WOLFSENTRY_USE_NONPOSIX_THREADS` | Define if POSIX thread API is not available. `WOLFSENTRY_THREAD_INCLUDE`, `WOLFSENTRY_THREAD_ID_T`, and `WOLFSENTRY_THREAD_GET_ID_HANDLER` will need to be defined. |
+| | `WOLFSENTRY_THREAD_INCLUDE` | Define to the path of a header file declaring a threading API. |
+| | `WOLFSENTRY_THREAD_ID_T` | Define to the appropriate type analogous to POSIX `pthread_t`. |
+| | `WOLFSENTRY_THREAD_GET_ID_HANDLER` | Define to the name of a void function analogous to POSIX `pthread_self`, returning a value of type `WOLFSENTRY_THREAD_ID_T`. |
+| | `FREERTOS` | Build for FreeRTOS |
 
 ### Build and Self-Test Examples
 
@@ -158,8 +161,7 @@ Install from an alternate build location to a non-standard destination:
 
 `make BUILD_TOP=./build INSTALL_DIR=/usr INSTALL_LIBDIR=/usr/lib64 install`
 
-Build libwolfsentry.a and test it under various analyzers (memory and thread
-testing under full battery of valgrind and sanitizer tests):
+Build libwolfsentry.a and test it in various configurations:
 
 `make -j check`
 
@@ -180,15 +182,15 @@ elaborate makefile code including additional rules and dependency mechanisms.)
 
 Build the smallest simplest possible library:
 
-`make -j SINGLETHREADED=1 NO_STDIO=1 DEBUG= OPTIM=-Os EXTRA_CFLAGS='-DWOLFSENTRY_NO_CLOCK_BUILTIN -DWOLFSENTRY_NO_MALLOC_BUILTIN -DWOLFSENTRY_NO_ERROR_STRINGS -Wno-error=inline -Wno-inline'`
+`make -j SINGLETHREADED=1 NO_STDIO=1 DEBUG= OPTIM=-Os EXTRA_CFLAGS="-DWOLFSENTRY_NO_CLOCK_BUILTIN -DWOLFSENTRY_NO_MALLOC_BUILTIN -DWOLFSENTRY_NO_ERROR_STRINGS -Wno-error=inline -Wno-inline"`
 
 Build and test with user settings:
 
 `make -j USER_SETTINGS_FILE=user_settings.h test`
 
 Build for FreeRTOS on ARM32, assuming FreeRTOS and lwIP source trees are located as shown:
 
-`make -j HOST=arm-none-eabi RUNTIME=FreeRTOS-lwIP FREERTOS_TOP=../third/FreeRTOSv202212.00 LWIP_TOP=../third/lwip EXTRA_CFLAGS='-mcpu=cortex-m7'`
+`make -j HOST=arm-none-eabi RUNTIME=FreeRTOS-lwIP FREERTOS_TOP=../third/FreeRTOSv202212.00 LWIP_TOP=../third/lwip EXTRA_CFLAGS=-mcpu=cortex-m7`
 
 
 ## Project Examples
@@ -211,6 +213,3 @@ build with wolfSentry integration, and use `--with-wolfsentry=/the/install/path`
 if wolfSentry is installed in a nonstandard location.  The wolfSSL test
 client/server can be loaded with user-supplied wolfSentry JSON configurations
 from the command line, using `--wolfsentry-config <file>`.
-
-
-

wolfSentry/src/json_configuration.md

@@ -168,12 +168,14 @@ allowed values are as in the ABNF formal syntax later in this document.
           "interface" : uint8,
           "address" : route_address,
           "prefix-bits" : uint16,
+          "bitmask" : route_address,
           "port" : endpoint_port
         },
         "local" : {
           "interface" : uint8,
           "address" : route_address,
           "prefix-bits" : uint16,
+          "bitmask" : route_address,
           "port" : endpoint_port
         }
       }
@@ -271,7 +273,8 @@ Each route is composed of the following elements, all of which are optional.
 * <b>`remote`</b> -- The attributes to match for the remote endpoint of the traffic.
     * <b>`interface`</b> -- Network interface ID, as an arbitrary integer chosen and used consistently by the caller or IP stack integration.
     * <b>`address`</b> -- The network address, in idiomatic form.  IPv4, IPv6, and MAC addresses shall enumerate all octets.  See `route_address` definition in the ABNF grammar below for permissible values.
-    * <b>`prefix-bits`</b> -- The number of bits in the <b>`address`</b> that traffic must match.
+    * <b>`prefix-bits`</b> -- The number of bits in the <b>`address`</b> that traffic must match (mutually exclusive with <b>`bitmask`</b>).
+    * <b>`bitmask`</b> -- A bitmask to be applied to the traffic address before matching with the route <b>`address`</b> (mutually exclusive with <b>`prefix-bits`</b>).
     * <b>`port`</b> -- The port number that traffic must match.
 
 * <b>`local`</b> -- The attributes to match for the local endpoint of the traffic.  The same nodes are available as for <b>`remote`</b>.
@@ -405,7 +408,7 @@ route_local_endpoint_clause = DQUOTE %s"local" DQUOTE ":" route_endpoint
 route_endpoint = "{"
     [ route_interface_clause "," ]
     [ route_address_clause ","
-      [ route_address_prefix_bits_clause "," ]
+      [ (route_address_prefix_bits_clause / route_address_bitmask_clause) "," ]
     ]
     [ route_port_clause "," ]
     -","
@@ -415,6 +418,8 @@ route_interface_clause = DQUOTE %s"interface" DQUOTE ":" uint8
 
 route_address_clause = DQUOTE %s"address" DQUOTE ":" route_address
 
+route_address_bitmask_clause = DQUOTE %s"bitmask" DQUOTE ":" route_address
+
 route_address = DQUOTE (route_address_ipv4 / route_address_ipv6 / route_address_mac / route_address_user) DQUOTE
 
 route_address_ipv4 = uint8 3*3("." uint8)