Skip to content

Releases: risingwavelabs/risingwave

v0.18.0

31 Mar 06:54
c02ae2c
Compare
Choose a tag to compare

For installation and running instructions, see Get started.

Main changes

Starting from this version, we’ll respect semantic versioning conventions by using the middle number (y , instead of z, in x.y.z) to indicate minor versions. That is why this is v0.18.0, not v0.1.18.

Administration and troubleshooting

  • Improves error messages by including the location of the statement in question. #8646
  • Initial values of immutable system parameters can be specified via the meta-node command line. The initial values provided in the configuration file will be ignored. #8366

SQL features

  • Adds initial support for user-defined functions. #8597 #8644 #8255 #7943
  • Adds support for JSONB data type. #8256 #8181
  • Adds support for NULLS { FIRST | LAST } in ORDER BY clauses. #8485
  • New commands:
    • ALTER SOURCE RENAME TO #8778
    • SET TIME ZONE #8572
    • ALTER RELATION RENAME #7745
    • ALTER TABLE ADD/DROP COLUMN for regular tables (without connector settings). #8394
  • New functions:
    • array_length : Returns the length of an array. #8636
    • String functions implemented with the help of chatGPT. #8767 #8839
      • chr(integer) -> varchar
      • starts_with(varchar, varchar) -> boolean
      • initcap(varchar) -> varchar
      • lpad(varchar, integer) -> varchar
      • lpad(varchar, integer, varchar) -> varchar
      • rpad(varchar, integer) -> varchar
      • rpad(varchar, integer, varchar) -> varchar
      • reverse(varchar) -> varchar
      • strpos(varchar, varchar) -> integer
      • to_ascii(varchar) -> varchar
      • to_hex(integer) -> varchar
      • to_hex(bigint) -> varchar)
    • Improves the data type values of columns returned by DESCRIBE . #8819
    • UPDATE commands cannot update primary key columns. #8569
    • Adds support for microsecond precision for intervals. #8501
    • Adds an optional parameter offset to tumble() and hop() functions. #8490
    • Data records that has null time values will be ignored by time window functions. #8146
    • Improves the behaviors of the exp operator when the operand is too large or small. #8309
    • Supports process time temporal join, which enables the joining of an append-only stream (such as Kafka) with a temporal table (e.g. a materialized view backed by MySQL CDC). This feature ensures that any updates made to the temporal table will not affect previous results obtained from the temporal join. Supports FOR SYSTEM_TIME AS OF NOW() syntax to express process time temporal join. #8480

Connectors

  • Adds a new field basetime to the load generator connector for generating timestamp data. The load generator will take this field as now and generates data accordingly. #8619
  • Empty cells in CSV are now parsed as null. #8709
  • Adds the Iceberg connector. #8508
  • Adds support for the upsert type to the Kafka sink connector. #8168
  • Removes the message name parameter for Avro data. #8124
  • Adds support for AWS PrivateLink for Kafka source connector. #8247

Full Changelog: v0.1.17...v0.18.0

v0.1.17

28 Feb 03:32
91621f6
Compare
Choose a tag to compare

For installation and running instructions, see Get started.

Main changes

Administration

  • Adds a system catalog view rw_catalog.rw_ddl_progress, with which users can view the progress of a CREATE INDEX, CREATE SINK, or CREATE MATERIALIZED VIEW statement. #7914
  • Adds the pg_conversion and pg_enum system catalogs. #7964, #7706

SQL features

  • Adds the exp() function. #7971
  • Adds the pow() function. #7789
  • Adds support for displaying primary keys in EXPLAIN statements. #7590
  • Adds support for descending order in CREATE INDEX statements. #7822
  • Adds SHOW PARAMETERS and ALTER SYSTEM commands to display and update system parameters. #7882, #7913

Connectors

  • Adds a new parameter match_pattern to the S3 connector. With the new parameter, users can specify the pattern to filter files that they want to ingest from S3 buckets. For documentation updates, see Ingest data from S3 buckets. #7565
  • Adds the PostgreSQL CDC connector. Users can use this connector to ingest data and CDC events from PostgreSQL directly. For documentation updates, see Ingest data from PostgreSQL CDC. [#6869](#6869, #7133
  • Adds the MySQL CDC connector. Users can use this connector to ingest data and CDC events from MySQL directly. For documentation updates, see Ingest data from MySQL CDC. #6689, #6345, #6481, #7133
  • Adds the JDBC sink connector, with which users can sink data to MySQL, PostgreSQL, or other databases that are compliant with JDBC. #6493
  • Add new parameters to the Kafka sink connector.
    • force_append_only : Specifies whether to force a sink to be append-only. #7922
    • use_transaction : Specifies whether to enable Kafka transactions or not. #7500
    • SSL/SASL parameters: Specifies SSL encryption and SASL authentication settings. #7540

Full Changelog: v0.1.16...v0.1.17

v0.1.16

01 Feb 08:44
1e97059
Compare
Choose a tag to compare

For installation and running instructions, see Get started.

Main changes

Administration

  • Adds support for aborting a query in local mode with Ctrl + C. #7444

SQL features

  • Adds support for the to_timestamp function. #7060
  • Adds support for the RETURNING clause in DML statements. #7094
  • Breaking change: Deprecates CREATE MATERIALIZED SOURCE . To create a materialized source, create a table and include the newly added connector settings. #7281, #7110
  • Adds support for the c and i flags in regex_match() and regex_matches() functions. #7135
  • Adds support for SHOW CREATE TABLE . You can use this statement to show the definition of a table. #7152
  • Adds support for the pg_stat_activity system catalog and several system functions. #7274
  • Adds the _rw_kafka_timestamp parameter to show the timestamps of Kafka messages. Users can now specify the scope of Kafka messages by timestamps. #7275, #7150
  • Adds support for displaying PostgreSQL and RisingWave versions in version(). #7314
  • Adds support for displaying internal tables using the SHOW INTERNAL TABLES statement. #7348
  • Adds support for SET VISIBILITY_MODE You can use this session variable to configure whether only checkpoint data is readable for batch query. #5850
  • Adds support for SET STREAMING_PARALLELISM . You can use this session variable to configure parallelism for streaming queries. #7370

Connectors

  • Adds support for generating array and struct data using the datagen connector. #7099
  • Adds the S3 source connector, with which users can ingest data in CSV format from S3 locations. For data ingestion from files, CSV is the only supported format and the files must be placed on S3. #6846

Full Changelog: v0.1.15...v0.1.16

v0.1.15

04 Jan 02:26
fbb3c31
Compare
Choose a tag to compare

For installation and running instructions, see Get started.

Main changes

Installation and deployment

  • Parallelism and available memory of compute nodes are now command-line arguments and removed from the configuration file. #6767
  • The default barrier interval is set to 1 second. #6553
  • Adds support for meta store backup and recovery. #6737

SQL features

  • Adds support for SHOW CREATE MATERIALIZED VIEW and SHOW CREATE VIEW to show how materialized and non-materialized views are defined. #6921
  • Adds support for CREATE TABLE IF NOT EXISTS. #6643
  • A sink can be created from a SELECT query. #6648
  • Adds support for struct casting and comparison. #6552
  • Adds pg_catalog views and system functions. #6982
  • Adds support for CREATE TABLE AS. #6798
  • Ads the initial support for batch query on Kafka source. #6474
  • Adds support for SET QUERY_EPOCH to query historical data based on meta backup. #6840

Connectors

  • Improves the handling of schema errors for Avro and Protobuf data. #6821
  • Adds two options to the datagen connector to make it possible to generate increasing timestamp values. #6591

Observability

  • Adds metrics for the backup manager in Grafana. #6898
  • RisingWave Dashboard can now fetch data from Prometheus and visualize it in charts. #6602

Full Changelog: v0.1.14...v0.1.15

v0.1.14

01 Dec 06:10
5db0a9a
Compare
Choose a tag to compare

For installation and running instructions, see Get started.

Main changes

SQL features

  • PRIMARY KEY constraint checks can be performed on materialized sources and tables but not on non-materialized sources. For tables or materialized sources that enabled PRIMARY KEY constraints, if you insert data to an existing key, the new data will overwrite the old data. #6320 #6435
  • Adds support for timestamp with time zone data type. You can use this data type in time window functions, and convert between it and timestamp (without time zone). #5855 #5910 #5968
  • Adds support for UNION and UNION ALL operators. #6363 #6397
  • Implements the rank() function to support a different mode of Top-N queries. #6383
  • Adds support for logical views (CREATE VIEW). #6023
  • Adds the data_trunc() function. #6365
  • Adds the system catalog schema. #6227
  • Displays error messages when users enter conflicting or redundant command options. #5933

Connectors

  • Adds support for the Maxwell Change Data Capture (CDC) format. #6057
  • Protobuf schema files can be loaded from Web locations in s3://, http://, or https:// formats. #6114 #5964
  • Adds support for Confluent Schema Registry for Kafka data in Avro and Protobuf formats. #6289
  • Adds two options to the Kinesis connector. Users can specify the startup mode and optionally the sequence number to start with. #6317

Full Changelog: v0.1.13...v0.1.14

v0.1.13

17 Oct 10:06
Compare
Choose a tag to compare

For installation and running instructions, see Get started.

Main changes

SQL features

  • SQL commands:

    • Improves the formatting of response messages of EXPLAIN statements. #5541
  • SQL functions:

    • to_char() now supports specifying output format in lowercase. #5032

      to_char(timestamp '2006-01-02 15:04:05', 'yyyy-mm-dd hh24:mi:ss') → 2006-01-02 15:04:05

    • generate_series now supports negative steps. #5231

      SELECT * FROM generate_series(5,1,-2);
      generate_series 
      -----------------
                     5
                     3
                     1
      (3 rows)
    • Adds support for sum/min/max functions over interval-type data. #5105, #5549

    • Adds support for array concatenation. #5060, #5345

    • Adds support for specifying empty arrays. #5402

    • Casting from array to varchar is now supported. #5081

      array[1,2]::varchar{1,2}

    • Casting from varchar to integer allows leading and trailing spaces. #5452

      ' 1 '::int1

  • Adds new system catalog and psql meta-commands. #5127, #5742

    • \d: Lists all relations in the current database. (Materialized) sources are not supported yet.
    • \dt: Lists all tables in the current database.
    • \dm: Lists all materialized views in the current database.
    • \di: Lists all indexes in the current database.
    • pg_catalog.pg_index: Contains information about indexes.

Connectors

  • Nested columns are now supported for the datagen connector. #5550
What's Changed

What's Changed

Read more

v0.1.12

19 Sep 06:38
Compare
Choose a tag to compare

For installation and running instructions, see Get started.

Main changes

SQL features

  • SQL commands:
    • EXPLAIN now supports specifying options. Supported options: trace, verbose, and type. Unlike PostgreSQL, each option should be separated by a comma and wrapped by parentheses as a whole. #4730
    • Adds support for ALTER USER. #4261
    • CREATE/ALTER USER now has new options CREATEUSER and NOCREATEUSER, which specifiy whether or not the user has the privilege to create, alter, or drop other users. #4447
    • Adds support for EXPLAIN CREATE SINK. #4430
  • SQL functions:
    • Adds support for new system information functions: current_schema, current_schema(), and session_user. #4358
  • The pg_namespace catalog now has a new namespace column nspacl for storing access privileges. #4326

Connectors

  • Some connector parameters were renamed. The old parameter names are still functional but may be deprecated in the future. #4503

    • Kafka & Redpanda

      • kafka.brokers -> properties.bootstrap.server
      • kafka.topic -> topic
      • kafka.scan.startup.mode -> scan.starup.mode
      • kafka.time.offset -> scan.startup.timestamp_millis
      • kafka.consumer.group -> consumer.group.id
    • Kinesis

      • kinesis.stream.name -> stream
      • kinesis.stream.region -> aws.region
      • kinesis.endpoint -> endpoint
      • kinesis.credentials.access -> aws.credentials.access_key_id
      • kinesis.credentials.secret -> aws.credentials.secret_access_key
      • kinesis.credentials.session_token -> aws.credentials.session_token
      • kinesis.assumerole.arn -> aws.credentials.role.arn
      • kinesis.assumerole.external_id -> aws.credentials.role.external_id
    • Pulsar

      • pulsar.topic -> topic
      • pulsar.admin.url -> admin.url
      • pulsar.service.url -> service.url
      • pulsar.scan.startup.mode -> scan.startup.mode
      • pulsar.time.offset -> scan.startup.timestamp_millis
  • The row format name, debezium json, for CDC stream sources, has been renamed to debezium_json. #4494

Configuration changes

  • The default batch query execution mode was changed from distributed to local. #4789

New Contributors

Full Changelog: v0.1.11...v0.1.12

What's Changed

What's Changed

Read more

v0.1.11

29 Jul 09:48
d91bdef
Compare
Choose a tag to compare

What's Changed

  • test(nexmark): rewrite flinksql function to the corresponding pg one by @TennyZhuang in #3654
  • refactor: remove OrderedColumnDesc from proto by @xxchan in #3642
  • feat(meta): do not use watermark as expire time of TTL by @Little-Wallace in #3633
  • fix(lookup): fix the semantics of arrange_order_rules by @BowenXiao1999 in #3655
  • chore(docs): update the binary version to v0.1.10 by @huangjw806 in #3661
  • test(batch): add micro benchmark for NestedLoopJoinExecutor by @wzzzzd in #3652
  • feat(meta): no pause checkpoint with building actor by @xxhZs in #3511
  • feat: expand operator by @likg227 in #3563
  • fix(meta): fix internal_table persisted by @Li0k in #3659
  • feat(frontend): support show config by @yezizp2012 in #3664
  • feat(frontend): StreamDynamicFilter by @jon-chuang in #3515
  • feat(storage): support vnode hint in storage table by @BugenZhao in #3628
  • refactor:introduce per crate error for batch executors by @ZENOTME in #3658
  • fix(docs): fix regress test input file path by @lmatz in #3667
  • test(nexmark): support explain nexmark q15 and q20 by @TennyZhuang in #3660
  • fix(metrics): use iter_in_process_count to replace iter_count_duration by @Li0k in #3678
  • fix(expr, executor): Remove RowExpression: use eval_row and unwrap_or(false) on Datum, not panic on Null/None by @jon-chuang in #3587
  • feat(meta): do not persist data on etcd for pin_snapshot by @Little-Wallace in #3557
  • refactor(streaming): better source executor & split change by @BugenZhao in #3669
  • fix: avoid unnecessary cells in max/min (not-append only) by @BowenXiao1999 in #3630
  • perf(hummock manager): apply version delta from version checkpoint by @soundOfDestiny in #3567
  • chore(ci): use random seed for sqlsmith cron test by @huangjw806 in #3688
  • feat: add overlay string function by @ALeitert in #3671
  • feat(batch): enable parallel table scan by @xxchan in #3670
  • refactor(storage): remove async_trait for MergeIteratorNext by @KivenChen in #3673
  • chore(doc): rename cell-based table to storage table by @wcy-fdu in #3691
  • feat(metrics): add metrics for compact task splits count by @Li0k in #3693
  • feat(frontend): support show all (#3665) by @chenzl25 in #3694
  • chore: fix rustdoc for prost by @xxchan in #3699
  • refactor: consolidate duplicated logic in PlanRoot by @xiangjinwu in #3554
  • feat(expr): support regexp_match by @TennyZhuang in #3702
  • chore(deps): bump openssl-src from 111.20.0+1.1.1o to 111.22.0+1.1.1q by @dependabot in #3701
  • doc: add some comments for keys & rename keys -> key by @xxchan in #3700
  • feat(streaming): allow pause source by @skyzh in #3697
  • feat(meta): allow set parallel degree in config by @skyzh in #3703
  • feat(batch): Implement LookupJoinExecutor by @Graphcalibur in #3614
  • fix(array): ListRef Display by @nanderstabel in #3705
  • feat(meta): add monitoring meta process metrics by @wenym1 in #3706
  • fix: reduce logging in the critical path by @neverchanje in #3704
  • refactor(optimizer): cleanup LogicalAgg::prune_col by @xiangjinwu in #3663
  • fix: correctly implement check_state_consistency by @zwang28 in #3711
  • feat(storage): default block size = 1024 by @skyzh in #3463
  • feat(batch): prune scan partition according to scan_range by @xxchan in #3698
  • test(nexmark): explain q16 q17 by @TennyZhuang in #3715
  • doc: add relational table schema doc + move files by @BowenXiao1999 in #3712
  • fix: fix heartbeat req field and meta client impl by @yezizp2012 in #3716
  • fix(metrics): use current actor id for backpressure by @skyzh in #3721
  • test(streaming): enable streaming e2e test for tpch q4, q18, q20 by @yuhao-su in #3718
  • chore(deps): bump dependencies by @skyzh in #3725
  • feat(executor): Basic in-application-memory DynamicFilterExecutor, pass tpch q11, q22 e2e tests by @jon-chuang in #3681
  • refactor(sqlparser): rename parse_string_agg to parse_string_concat by @richardchien in #3733
  • fix(risedev): Revert Makefile.toml changes by @Graphcalibur in #3732
  • fix: support decimal mul interval by @neverchanje in #3734
  • refactor(storage): remove cell_based serialize/deserialize from serde by @wcy-fdu in #3722
  • fix(sqlsmith): completely cover all exprs by @neverchanje in #3737
  • fix(ci): use header fix action for license check by @skyzh in #3739
  • chore(risedev): update grafana version by @wcy-fdu in #3731
  • test(nexmark): add nexmark q10 q14 e2e test by @TennyZhuang in #3740
  • refactor(streaming): use table catalog in hash join by @yuhao-su in #3707
  • fix(batch): fix batch scan distribution & handle more than one table scans in one stage by @xxchan in #3736
  • feat(optimizer): support two phase topn and limit in batch mode by @chenzl25 in #3719
  • feat: deprecate catalog ref id by @yezizp2012 in #3745
  • refactor(streaming): use table catalog for materialize executor by @wcy-fdu in #3742
  • refactor(meta): improve create mview / scheduler readability by @BugenZhao in #3748
  • feat(metrics): Add Fragment Level Exchange Metrics by @marvenlee2486 in #3696
  • fix(risedev): unexpected rebuilds on Linux by @skyzh in #3761
  • perf(hummmock manager): remove unnecessary hummock version deltas by @soundOfDestiny in #3689
  • fix: sync compaction group correctly by @zwang28 in #3743
  • feat(common): move observer_manager to common service by @xxhZs in #3754
  • refactor: remove vnode_mappings in CompactTask and GroupedSstableBuilder by @hzxa21 in #3764
  • refactor(batch): say goodbye to the state machine in NLJ by @wzzzzd in #3724
  • feat(common): add is_set to BitmapBuilder by @wzzzzd in #3768
  • fix(batch): add first-match optimization for left semi join by @wzzzzd in #3773
  • feat(streaming): use multiple dispatcher for mview creation by @BugenZhao in #3758
  • refactor(stream): refactor state of topN executor to one state table by @StrikeW in #3747
  • chore(storage): remove value meta by @Li0k in #3779
  • feat(executor): Use TableCatalog and StateTable in DynamicFilter (only writes) by @jon-chuang in #3757
  • fix(batch): fix order_by with struct table by @StrikeW in htt...
Read more

v0.1.10

05 Jul 08:48
2df3440
Compare
Choose a tag to compare

For a quick summary on this release, see Release Notes for more information.

What's Changed

Read more

v0.1.9

08 Jun 10:37
68c893e
Compare
Choose a tag to compare

This is a special out-of-schedule release for the new build pipeline.

What's Changed

  • feat(stream): support replace source reader at runtime by @tabVersion in #2923
  • feat(expr): Implementation for bitwise operation by @marvenlee2486 in #2884
  • fix(types): reduce [Struct/List]Value memory usage and fix Decimal data_size by @TennyZhuang in #2970
  • feat(expr): Implementation for bitwise operation (Continue PR #2884) by @marvenlee2486 in #2975
  • chore(ci): speedup unit test by @skyzh in #2974
  • feat(build): drop support for TiKV and RocksDB, and speedup compile by @skyzh in #2982
  • fix(risedev): apply-frontend-test on macOS by @skyzh in #2983
  • feat(build): link openssl statically by @skyzh in #2984
  • feat(grafana): use Python script to generate dashboard by @skyzh in #2979
  • chore(risedev): store kafka and frontend config when compose deploy by @skyzh in #2988
  • refactor(storage): avoid KeyRange type conversion by @zwang28 in #2920
  • feat(expr): Implement approx_count_distinct for batch queries by @Graphcalibur in #2977
  • feat(vnode bitmap): implement vnode-aware overlap in compaction by @soundOfDestiny in #2864
  • chore(test): prevent test compile OOM by @skyzh in #2989
  • feat: add hummock rpc unpin_snapshot_before by @BowenXiao1999 in #2964
  • feat(streaming): use table_id as AppendOnlytopN prefix for keyspace by @Li0k in #2992
  • feat(optimizer): pushdown predicate to LogicalScan by @xxchan in #2987
  • feat: enable exchange operator on frontend to send local execution mode request by @lmatz in #2978
  • chore(grafana): add estimated s3 cost by @skyzh in #3000
  • fix: remove unsafe impl Sync trait for LruCacheShard by @MrCroxx in #2996
  • chore: Ignore commits in the blame view by @xxchan in #2995
  • feat(streaming): record internal_table_ids to TableFragments by @Li0k in #3002
  • chore(ci): migrate docker build pipeline by @huangjw806 in #2990
  • feat: support linking to different database by @cykbls01 in #2751
  • fix(test): unit test of chain executor by @lokax in #2817
  • chore(risedev): allow minio to be used on root disk by @skyzh in #3012
  • feat(meta): record total size of files in version by @Little-Wallace in #2965
  • feat(streaming): source executor use table_id for keyspace prefix by @Li0k in #3010
  • fix(streaming): fix some test with keyspace::table_root by @Li0k in #3018
  • chore(build): bump deps by @skyzh in #3019
  • refactor(kinesis): Correct assume_role_external_id spelling in Kine… by @ryanrussell in #3016
  • feat(risedev): support ipv6 in config by @skyzh in #3011
  • fix(connector): use bound channel for source by @skyzh in #3023
  • feat: support grant/revoke privilege in frontend by @yezizp2012 in #3004
  • feat(storage): support write anytime and spill to disk by @wenym1 in #2919
  • feat(stream): introduce allocator_api in JoinHashMap by @TennyZhuang in #3020
  • refactor(DataChunk): use new rather than DataChunkBuilder or try_from by @xiangjinwu in #3026
  • chore(streaming): add assert check for internal_table_id_set to avoid… by @Li0k in #3025
  • fix: should always unpin epoch for dml query by @BowenXiao1999 in #3034
  • fix(config): parameters are case insensitive in set command by @lmatz in #3033
  • feat(compaction): notify meta when hash mapping changes by @soundOfDestiny in #3035
  • feat(meta): initiate static compaction groups by @zwang28 in #2952
  • perf(executor): decode row datums from pk by @kwannoel in #2957
  • feat(storage): introduce non-proto type for vnode bitmap by @xx01cyx in #3030
  • feat(relational_iter): add state_table_iter with pk_prefix and pk_bounds by @wcy-fdu in #3008
  • feat(pgwire): introduce simple extended query mode by @ZENOTME in #2961
  • feat(expr): implementation for char_length operation by @nasnoisaac in #2991
  • feat(storage): support parallel upload by @skyzh in #3031
  • feat: use unpin_snapshot_before in FE by @BowenXiao1999 in #2993
  • chore: remove scan_with_start_key by @wcy-fdu in #3048
  • chore(docs): bump to 0.1.8 and update CI badge by @skyzh in #3056
  • feat(ci): parallelize e2e test by @skyzh in #3046
  • feat(storage): support concurrent upload in capacity split sst builder by @wenym1 in #3052
  • refactor(session): error when setting unrecognized configuration parameter by @lmatz in #3042
  • release: v0.1.9 by @huangjw806 in #3062
  • fix: the signature unmatch in async trait function by @BowenXiao1999 in #3068
  • fix(connector): remove redundant warning by @tabVersion in #3072
  • chore(ci): migrate release pipeline by @huangjw806 in #3070
  • chore(docs): change risingwave logo in readme by @mikechesterwang in #3076

New Contributors

Full Changelog: v0.1.8...v0.1.9