v0.9.3

What's Changed

• fix: deprecated extension config map not set properly by @brennanjl in #1090
• services/jsonrpc: compress middleware by @jchappelow in #1093
• jsonrpc/usersvc: use best block height/time in view call by @jchappelow in #1095
• update cometbft dep for security alert by @jchappelow in #1098
• add all-scalar flag by @brennanjl in #1101
• fix csv parsing issue by @jchappelow in #1108
• version: update self-report tag for vanilla builds by @jchappelow in #1114

Full Changelog: v0.9.2...v0.9.3

v0.9.2

What's Changed

• added max db connections config by @brennanjl in #1069
• pg: recognize streaming prepared tx msg by @brennanjl in #1073
• sql,pg: catch db backend failure by @jchappelow in #1082
• expose TrustPeriod flag for snapshots by @jchappelow in #1083
• engine/generate: fix repeated numbered arg typecast by @brennanjl in #1077

Full Changelog: v0.9.1...v0.9.2

v0.9.1

What's Changed

• docs: v0.8 release notes by @KwilLuke in #1049
• rpc: fix cors because of blocked OPTIONS by @jchappelow in #1053

v0.9.0

Kwil v0.9

This is a major release of Kwil DB with several new features.

The highlights are:

• Zero downtime migrations
• Data privacy modes: node peer filtering, and private RPC mode with authenticated calls
• Support builds with customizable branding
• Genesis and end-block hooks for extension authors
• Kuneiform updates: many new functions and syntax improvements
• A framework for testing Kuneiform schemas before deployment
• Improved node monitoring
• Removal of the previously-deprecated gRPC server and the legacy HTTP gateway

Contents

• Important Upgrading Notes
• Build Requirements and Dependencies
• Security Policy
• Notable Features and Changes
  • Zero Downtime Migrations
  • Data Privacy Modes
    • Node Peer Filtering Capability
    • Private RPCs and Authenticated Calls
  • Ability to Rebrand Customized Binary Builds
  • Extension Hooks for Genesis Initialization and End-Block Actions
  • Kuneiform Language
    • Kuneiform Additions
    • Kuneiform Changes
  • New Kuneiform Testing Framework
  • Health Check Endpoint and JSON-RPC Methods
  • Log Rotation and Archival
  • Expose CometBFT Prometheus Metrics
  • Extension Interfaces
• Breaking Changes
  • Removal of the gRPC service and its HTTP API gateway
  • Breaking Changes and Deprecations in CLI Applications
  • Breaking Changes to Kuneiform and SQL Rules
  • Breaking Changes to Transaction Execution
  • Breaking Changes to Extension Interfaces
• Minor Features and Changes
  • Minor New Features
    • CLI Apps
    • Go SDK
    • Parser
    • Node Internals
  • Minor Changes
• Fixes
  • Backported to v0.8
• Testing
• Diff summary

Important Upgrading Notes

Upgrading to this release requires a network migration. See the network migration documents for instructions on how to perform a network migration. Only upgrading from v0.8 is supported.

The application's protocol version is bumped from 0 to 1. This prevents nodes that are running different versions of Kwil from operating on the same network. The genesis file no longer includes a "version" section since this is determined by the application logic.

In certain cases, deployed schemas may not be compatible with v0.9. See the Kuneiform Language section for details.

Build Requirements and Dependencies

• The minimum required Go version is now 1.22. (724fe4f)
• Builds and tests use Go 1.23. (3aecf89, d50159e)
• Update the consensus engine (CometBFT) to v0.38.12. (89bad53)

Security Policy

The supported versions are now as follows:

Version	Supported
Latest beta or release candidate	✓
v0.9.x	✓
v0.8.x	✓
< v0.8	❌

See SECURITY.md for details.

Notable Features and Changes

Zero Downtime Migrations

This release adds a new type of migration called a Zero Downtime (ZDT) migration. Unlike the existing offline process, ZDT migrations launch the new network prior to terminating the existing network. There is a migration period during which changes on the existing network are replicated on the new network. See the docs for more information.

Note that the existing network must also support ZDT migrations. Since this is the first release of Kwil that supports such migrations, it cannot be used to go from a v0.8 network to a new v0.9 network. Future major releases will support ZDT migrations allowing this.

These changes also introduced two new transaction types: create resolution and approve resolution. These support generalized resolution processing, migration proposals being one such type.

Relevant code changes: f290d6c, 2f24aae, 0706798, ba6e3e, d175b7f

To support changeset creation from the logical replication stream from PostgreSQL, the type system in the pg package was reworked. See the following relevant changes to the DB internals and exported types in core/types: c983b22, dfa1d8a, 1a16488, 4ece583

NOTE: To support creation of changesets that enable detection and resolution of conflicts between the networks, all tables are now created with "full" replica identity". This changes "apphash" computation, so Kwil v0.9 cannot be installed in place over a v0.8 deployment's data.

Data Privacy Modes

To support use cases where a network needs to maintain data privacy, this release adds two main access control functionalities at both the P2P layer and in the RPC service.

Node Peer Filtering Capability

To run a private network of nodes that only permits inbound P2P connections from certain peers, there is now a private mode that uses peer filtering to only allow certain nodes to connect. The filtering is based on Node ID, which cryptographically ensures the identity of the peer.

A node's whitelist is composed of the current validator set, nodes with active join requests with an approval from the operator, configured seed nodes and persistent peers, and any manually whitelisted peers. This latter case is used to support the addition of authorized sentry (non-validator) nodes to support other functions such as RPC providers, indexers, and seeders.

It is enabled in config.toml with the chain.p2p.private_mode setting. Whitelisted peers may be set in config.toml with chain.p2p.whitelist_peers, or managed at runtime with the kwil-admin whitelist subcommands.
This functionality was added in 559b027.

Private RPCs and Authenticated Calls

A "private" RPC mode has been added with the following features:

• All call requests are authenticated. The Kwil clients handle this automatically when communicating with an RPC provider in private mode. A private key must be configured with the client in order to sign the call requests.
• Ad hoc SQL queries are prohibited.
• Transaction status query responses do not include the full transaction, which could potentially contain data that the user is not authorized to access.

For Go SDK users, note that this adds a signature field to the core/client.CallMessage type.

Related commits: 6537114, 559b027, 177891a

Ability to Rebrand Customized Binary Builds

Projects now have the ability to build custom-branded binaries that group in kwild, kwil-cli, and kwil-admin. These binaries can be given customized default configurations.

The global cmd.BinaryConfig structure is added to allow changing the project name and other details. Projects should set their configuration in an instance of the cmd/custom.CommonCmdConfig and then create their application's root command with the NewCustomCmd function.

This was added in 2aee07c and efc3af4.

Extension Hooks for Genesis Initialization and End-Block Actions

There are now hooks for extension authors to perform arbitrary actions:

• GenesisHook is a function that is run exactly once, ...

v0.8.4

This release of Kwil DB contains a fix to snapshot creation that could occur for records larger than 64kB, and a fix for a SQL parser bug that would prevent in-line expressions from being comparable (e.g. $1 is null without an explicit typecast).

The parse/v0.2.4 tag for the parse module accompanies this Kwil DB release.

What's Changed

• Fix a snapshot creation error with bufio.Scanner exceeding its line scan limit. This affected both kwil-admin, and the background statesync functionality in kwild. (b3460a7)
• Fix: in-line type assertions not being comparable. Actions using in-line statements with type casts for ambiguous types would not generate SQL properly. This would only occur when performing an expression (such as a comparison). Also fix a stack overflow when using the parse debugger tool via kwil-cli. (faf879e)
• Prepare v0.8.4 (version package and go.mod update for parse module) (2c71b80)

Full Changelog: v0.8.3...v0.8.4

Build

The provided application archives on this release are built with Go 1.22.5.

v0.8.3

This release of Kwil DB contains several minor fixes to the parse module, a fix to kwild's procedural language generation with unary expressions, and a fix to a possible startup crash if generated snapshot files on disk are corrupt or unreadable.

The parse/v0.2.3 tag for the parse module accompanies this Kwil DB release.

What's Changed

• engine: fix generated unary where there was a missing space (6d24370)
• kwild: fix handling invalid snapshot files (507e528)
• parse: fix unknown columns in indexes / foreign keys (7eb5a26)
• parse: removed an unneeded uuid case (16efeef)
• parse: fix Execute handling of parameters (9794a49)
• kwild,kwil-cli: use parse/v0.2.3 in the main module (5d29af4)
• version: 0.8.3 (b2d1346)

Full Changelog: v0.8.2...v0.8.3

Build

The provided application archives on this release are built with Go 1.22.4.

v0.8.2

This release of contains several bug fixes and backported feature additions.

What's Changed

• Introduce genesis consensus params to configure max limits for votebody transactions. (3d6772e)
• Add genesis and end block hooks. (881ebfb)
• Added look back for token errors. (624bff9)
• Fix/kwil admin config. (e4c2f82)
• Fixed panic on parser returns. (062c9eb)
• version: bump self report version to 0.8.2 (1130d69)
• Use parse/v0.2.2 (e525214)
• Change log level for failed resolutions (6a09f17)
• CI: Backport release workflow update (082e44d)

Full Changelog: v0.8.1...v0.8.2

v0.8.1

Kwil DB v0.8.1

This is a major new release of Kwil DB.

The highlights are:

• A new Kuneiform element called "procedures" that uses a procedural language for building logic into schemas.
• Added a JSON-RPC server.
• Network migration support.
• Enable state sync mode and snapshot creation.
• Absorb the kuneiform and dependent repositories into the parse module in the kwil-db repository.

Note that v0.8.0 was not published due to issues with the upgrade procedures.

Important Upgrading Notes

Upgrading to this release requires a network migration. See the network migration documents for instructions on how to perform a network migration. Only upgrading from v0.7 is supported.

In certain cases, deployed schemas may not be compatible with v0.8. See UPGRADING.md for details.

Notable Changes

Procedures

In previous releases, SQL queries were specified entirely within Kuneiform "actions". This release introduces the concept of "procedures", which use a basic procedural language for building logic into schemas. Procedures can be used for building access control logic, performing arithmetic, and managing control flow.

Procedures are declared using the procedure keyword. A procedure may return either a single record (a tuple) or a table.

A procedure body may contain variable declarations, basic arithmetic operations, calls to various built-in functions, control flow (e.g. if/then/for), and SQL queries.

See the procedure docs for more information.

Variable Types

Procedures are strongly typed. The parameters, returns, and variables declared within a procedure must be assigned a type. The recognized types are:

• int is a 64-bit signed integer
• text is a variable length character array (a string)
• bool is a boolean (true or false)
• blob is a byte array
• uint256 is a 256-bit unsigned integer
• decimal is a fixed precision type based
• uuid is 128-bit universally unique identifier (UUID)

Arrays of any of the above types are also permitted.

Legacy actions only support int and text in table declarations, and use no types in their signatures or bodies.

Foreign Procedure Calls

It is now possible to interact with procedures in a different database using foreign procedure declarations. For instance, to use a procedure called target_procedure in the database 'x_target_dbid', declare a foreign procedure in your database and specify the target database and procedure name when calling:

foreign procedure get_user($id uuid) returns (text, int)

procedure call_get_user($id uuid) public  {
    $username, $age = get_user['x_target_dbid', 'target_procedure']($id);
    // $username is a text and $age is an int
}

JSON-RPC Server and gRPC Gateway Deprecation

This release of kwild adds a JSON-RPC server. It listens on TCP port 8484 of all network interfaces by default. JSON-RPC requests are handled by the /rpc/v1 path.

The CLI utilities, the Go SDK client, and the JS SDK now use the JSON-RPC server by default. RPC providers should expose TCP port 8484, and client configuration should be updated to use port 8484.

The HTTP API, which is a REST gateway for the gRPC server, is now deprecated, although it listens on all network interfaces on port 8080 by default. The gRPC server, which was previously deprecated, now listens only on a loopback address to support the internal HTTP gateway (Swagger). The gRPC and the HTTP gateway will be removed in the next release.

The OpenRPC specification document for the "user" RPC service may be found in the source code repository. The server itself will also provide the server's active service specification at the /spec/v1 HTTP endpoint and in the response to the rpc.discover JSON-RPC methods.

The "params" value in all requests are named rather than positional. This means that the JSON-RPC 1.0 calling convention with an array [] for the parameters is not supported, only the JSON-RPC convention using an object {}. However, this is a detail that the Kwil clients hide from users. Only if doing manual HTTP POST requests, such as with curl or Postman, is this important.

The "admin" service and the kwil-admin tool are also updated to use JSON-RPC.

SQL Syntax

• Support for more built-in functions. See Supported Functions.
• Conflicting column names in the returned values from a query are disallowed.
  For example, this means that it is now an error if a SELECT * is used with a
  JOIN between tables with an identical column name.
• There is now a requirement that at least one side of a JOIN must be a
  table's column. It is an error to have literal or computed values on both sides. (Q: does every joined column need to be just a column, or all joined columns. What about joining on a function of a column?)

Snapshots and State Sync

State sync is a feature that allows a new node to bootstrap directly from a snapshot of the blockchain state instead of replaying all the blocks from the genesis. This reduces the time needed to sync a new node to a network. However, the node will only be aware of transactions after the snapshot height.

kwild may now be configured to periodically generate state snapshots, which are provided to nodes that are joining the network. The app.snapshots config section adjust this functionality.

To enable the use of state sync on a new node, trusted snapshot providers must be configured. The chain.statesync config section adjust this functionality.

State sync is only used for a new node. Catch up after a previously-running node has restarted uses the standard "block sync" technique, which fully validates and executes all blockchain transactions to rebuild state.

See the State Sync docs for details.

Network Migrations

To launch a new network with an initial genesis state, special state snapshots may be generated using the kwil-admin snapshot create command.

When launching a network from this genesis state, the app_hash field of the new genesis.json file will be non-null. The initial validator nodes must use the genesis_state config.toml setting to specify the snapshot that corresponds to the app_hash value. Once a trusted snapshot provider has generated an initial snapshot for the new running network, other nodes may rely on the state sync mechanism to join the network without having to manually distribute the genesis_state file.

See Network Migration documents for details.

Coordinated Upgrades

This release includes an experimental system for specifying planned changes to code that affects consensus. This is an presently experimental system that is being evaluated to help avoid the need for network migrations when important fixes and updates might otherwise break consensus. The the documentation for details.

Command Line Application Changes

User CLI (kwil-cli) Changes

Moved kwil-cli Configuration Folder

The default config folder for kwil-cli is now called kwil-cli in the user's home directory. It was previously called kwil_cli.

New RPC Provider Setting

Within config.json, the RPC provider is specified with a "provider" field. This was previously called "grpc_url".

The --kwil-provider command line flag is DEPRECATED and will be removed in the subsequent release. Use the --provider flag instead.

When updating for the new provider setting, ensure it corresponds to the JSON-RPC server at port 8484 instead of the REST API at port 8080. Otherwise, ensure that the chosen RPC provider is updated to direct to the JSON-RPC server.

Command Changes

The result from the utils query-tx command no longer includes the transaction details for an unconfirmed transaction. Confirmed transactions will still include the decoded transaction. A new --raw flag may be provided with this command to retrieve the serialized transaction data given a transaction hash.

Added the utils parse command to parse a Kuneiform file and output the schema as JSON.

Added the utils decode-tx command to decode a serialized transaction provided as a base64 string.

Many of the fields in the JSON objects returned by query-tx and other commands with the --output json option are renamed with consistent snake case formatting. See the See UPGRADING.md for details on changes to the core module types and their JSON tags.

The account balance command may now be given a --pending flag to return account information that includes changes that consider any unconfirmed transactions. This is useful for determining the next nonce for an account when broadcasting multiple transaction per block.

Node Application (kwild) Changes

The default setting for the admin service (admin_listen_addr) is now "/tmp/kwild.socket".

The jsonrpc_listen_addr setting replaces the grpc_listen_addr setting. The default value for jsonrpc_listen_addr is 0.0.0.0:8484 (listen on port 8484 on all network interfaces).

In addition to standard output, kwild also writes the log to "kwild.log" in the application's root directory by default. The output_paths setting is used to change this.

Two new timeouts are added:

• rpc_timeout sets a timeout on requests on the user R...

v0.7.6

What's Changed

• kwild,rpc: db and rpc timeout settings (235a1f1). This introduces two timeout settings: rpc_timeout and db_read_timeout. The DB timeout applies to DB read operations in the Call and Query service methods. The RPC timeout applies to all RPCs, and includes the entire request-response roundtrip.
• pg: always release conn on commit/rollback error (5824d29). This resolves a potential leak of DB reader connections, and a shutdown hang.

Full Changelog: v0.7.5...v0.7.6

There are no changes to the core or parse modules.

Build

This release is built with Go 1.22.3

v0.7.5

What's Changed

• Fix loading of environment variables for kwild config. (a325df6). Environment variables for all kwild config settings are now being respected. They were incorrectly bound previously.
• core: kgw cookie on multi providers (5bdd5c3). Add support for kgw cookies that contain tokens for multiple domains.
• Update go.mod for core module version bump (c4e11f5).
• kwild: fix cometbft client not setting "syncing" (2730871). This fixes the "syncing" field of the node status command response always being false.

Full Changelog: v0.7.4...v0.7.5

Build

This release is built with Go 1.22.3