Skip to content

Commit

Permalink
Dynamic Table.
Browse files Browse the repository at this point in the history 
Dynamic Table is a an auto-refreshing materialized view which could
be constructed by base tables, external tables, materialized views
and dynamic tables.
And it could be used to answer query by AQUMV.
As normal tables in CBDB, dynamic tables could also have distribution
keys.

The purpose of Dynamic Tables is to solve the problem often raised by
customers who are big fans of a lakehouse architecture: how can we run
queries on external tables as fast as internal tables?

CREATE DYNAMIC TABLE:

CREATE DYNAMIC TABLE dt0 SCHEDULE '5 * * * *' AS
  SELECT a, b, sum(c) FROM t1 GROUP BY a, b WITH NO DATA DISTRIBUTED
BY(b);
CREATE DYNAMIC TABLE

\d
                 List of relations
 Schema | Name |     Type      |  Owner  | Storage
--------+------+---------------+---------+---------
 public | dt0  | dynamic table | gpadmin | heap
 public | t1   | table         | gpadmin | heap
(2 rows)

CREATE DYNAMIC TABLE xxx AS Query
The Query allows any valid SELECT SQL of Materialized Views: from single
or multiple relations, base tables, materialized views, and dynamic
tables as well, joins, subquery, aggregation, group by and etc.

SCHEDULE:
A string used to schedule background job which auto-refreshes the
dynamic table.
We follow the valid string of pg_cron extension which supports linux
crontab, refer https://crontab.guru

 ┌───────────── min (0 - 59)
 │ ┌────────────── hour (0 - 23)
 │ │ ┌─────────────── day of month (1 - 31) or last day of the month ($)
 │ │ │ ┌──────────────── month (1 - 12)
 │ │ │ │ ┌───────────────── day of week (0 - 6) (0 to 6 are Sunday to
 │ │ │ │ │                  Saturday, or use names; 7 is also Sunday)
 │ │ │ │ │
 │ │ │ │ │
 * * * * *

You can also use '[1-59] seconds' to schedule a job based on an
interval.
The example creates a cron job refreshing the dynamic table at minute 5
of each hour.
For convenience, SCHEDULE is optional. If user didn't specific it, a default
schedule is provided: at every 5th minute.

WITH NO DATA:
Same as Materialized View, will create an empty Dynamic Table if
specified.

DISTRIBUTED BY:
Same as normal tables in CBDB, Dynamic Tables could support distribution
keys as materialized views.

Refresh Dynamic Table
As seen in pg_task, we put a command to auto-refresh dynamic tables.
However, if users want to do a REFRESH manually, exec command REFRESH
DYNAMIC TABLE is also supported.
REFRESH DYNAMIC TABLE dt0;
REFRESH DYNAMIC TABLE

Refresh WITH NO DATA
Same as Materialized Views, Refresh with no data will truncate the
Dynamic Table and make it unpopulated status.
REFRESH DYNAMIC TABLE dt0 WITH NO DATA;
REFRESH DYNAMIC TABLE

Drop Dynamic Table
Drop a Dynamic Table will drop its scheduler job automatically.
DROP DYNAMIC TABLE dt0;
DROP DYNAMIC TABLE

Like Materialized Views, Dynamic Tables could be used to answer query
too. This is limited by AQUMV.

Authored-by: Zhang Mingli [email protected]
  • Loading branch information
@avamingli
avamingli committed Nov 26, 2024
1 parent f92faf0 commit 25faf4a
Show file tree
Hide file tree
Showing 31 changed files with 1,126 additions and 13 deletions.
1 change: 1 addition & 0 deletions contrib/pg_stat_statements/pg_stat_statements.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1153,6 +1153,7 @@ pgss_ProcessUtility(PlannedStmt *pstmt, const char *queryString,
rows = (qc && (qc->commandTag == CMDTAG_COPY ||
qc->commandTag == CMDTAG_FETCH ||
qc->commandTag == CMDTAG_SELECT ||
qc->commandTag == CMDTAG_REFRESH_DYNAMIC_TABLE ||
qc->commandTag == CMDTAG_REFRESH_MATERIALIZED_VIEW)) ?
qc->nprocessed : 0;

Expand Down
3 changes: 3 additions & 0 deletions doc/src/sgml/ref/allfiles.sgml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY createConversion SYSTEM "create_conversion.sgml">
<!ENTITY createDatabase SYSTEM "create_database.sgml">
<!ENTITY createDomain SYSTEM "create_domain.sgml">
<!ENTITY createDynamicTable SYSTEM "create_dynamic_table.sgml">
<!ENTITY createEventTrigger SYSTEM "create_event_trigger.sgml">
<!ENTITY createExtension SYSTEM "create_extension.sgml">
<!ENTITY createForeignDataWrapper SYSTEM "create_foreign_data_wrapper.sgml">
Expand Down Expand Up @@ -114,6 +115,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY dropConversion SYSTEM "drop_conversion.sgml">
<!ENTITY dropDatabase SYSTEM "drop_database.sgml">
<!ENTITY dropDomain SYSTEM "drop_domain.sgml">
<!ENTITY dropDynamicTable SYSTEM "drop_dynamic_table.sgml">
<!ENTITY dropEventTrigger SYSTEM "drop_event_trigger.sgml">
<!ENTITY dropExtension SYSTEM "drop_extension.sgml">
<!ENTITY dropForeignDataWrapper SYSTEM "drop_foreign_data_wrapper.sgml">
Expand Down Expand Up @@ -166,6 +168,7 @@ Complete list of usable sgml source files in this directory.
<!ENTITY prepare SYSTEM "prepare.sgml">
<!ENTITY prepareTransaction SYSTEM "prepare_transaction.sgml">
<!ENTITY reassignOwned SYSTEM "reassign_owned.sgml">
<!ENTITY refreshDynamicTable SYSTEM "refresh_dynamic_table.sgml">
<!ENTITY refreshMaterializedView SYSTEM "refresh_materialized_view.sgml">
<!ENTITY reindex SYSTEM "reindex.sgml">
<!ENTITY releaseSavepoint SYSTEM "release_savepoint.sgml">
Expand Down
185 changes: 185 additions & 0 deletions doc/src/sgml/ref/create_dynamic_table.sgml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
<!--
doc/src/sgml/ref/create_dynamic_table.sgml
PostgreSQL documentation
-->

<refentry id="sql-createdynamictable">
<indexterm zone="sql-createdynamictable">
<primary>CREATE DYNAMIC TABLE</primary>
</indexterm>

<refmeta>
<refentrytitle>CREATE DYNAMIC TABLE</refentrytitle>
<manvolnum>7</manvolnum>
<refmiscinfo>SQL - Language Statements</refmiscinfo>
</refmeta>

<refnamediv>
<refname>CREATE DYNAMIC TABLE</refname>
<refpurpose>define a new dynamic table</refpurpose>
</refnamediv>

<refsynopsisdiv>
<synopsis>
CREATE DYNAMIC TABLE [ IF NOT EXISTS ] <replaceable>table_name</replaceable>
[ (<replaceable>column_name</replaceable> [, ...] ) ]
[ USING <replaceable class="parameter">method</replaceable> ]
[ WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
[ TABLESPACE <replaceable class="parameter">tablespace_name</replaceable> ]
AS <replaceable>query</replaceable>
[ WITH [ NO ] DATA ]
</synopsis>
</refsynopsisdiv>

<refsect1>
<title>Description</title>

<para>
<command>CREATE DYNAMIC TABLE</command> defines a dynamic table of
a query. The query is executed and used to populate the view at the time
the command is issued (unless <command>WITH NO DATA</command> is used) and may be
refreshed later using <command>REFRESH DYNAMIC TABLE</command>.
</para>

<para>
<command>CREATE DYNAMIC TABLE</command> is similar to
<command>CREATE TABLE AS</command>, except that it also remembers the query used
to initialize the view, so that it can be refreshed later upon demand.
A dynamic table has many of the same properties as a table, but there
is no support for temporary dynamic tables.
</para>

<para>
<command>CREATE DYNAMIC TABLE</command> requires
<literal>CREATE</literal> privilege on the schema used for the dynamic
table.
</para>
</refsect1>

<refsect1>
<title>Parameters</title>

<variablelist>
<varlistentry>
<term><literal>IF NOT EXISTS</literal></term>
<listitem>
<para>
Do not throw an error if a dynamic table with the same name already
exists. A notice is issued in this case. Note that there is no guarantee
that the existing dynamic table is anything like the one that would
have been created.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><replaceable>table_name</replaceable></term>
<listitem>
<para>
The name (optionally schema-qualified) of the dynamic table to be
created.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><replaceable>column_name</replaceable></term>
<listitem>
<para>
The name of a column in the new dynamic table. If column names are
not provided, they are taken from the output column names of the query.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><literal>USING <replaceable class="parameter">method</replaceable></literal></term>
<listitem>
<para>
This optional clause specifies the table access method to use to store
the contents for the new dynamic table; the method needs be an
access method of type <literal>TABLE</literal>. See <xref
linkend="tableam"/> for more information. If this option is not
specified, the default table access method is chosen for the new
dynamic table. See <xref linkend="guc-default-table-access-method"/>
for more information.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><literal>WITH ( <replaceable class="parameter">storage_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )</literal></term>
<listitem>
<para>
This clause specifies optional storage parameters for the new
dynamic table; see
<xref linkend="sql-createtable-storage-parameters"/> in the
<xref linkend="sql-createtable"/> documentation for more
information. All parameters supported for <literal>CREATE
TABLE</literal> are also supported for <literal>CREATE DYNAMIC
TABLE</literal>.
See <xref linkend="sql-createtable"/> for more information.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><literal>TABLESPACE <replaceable class="parameter">tablespace_name</replaceable></literal></term>
<listitem>
<para>
The <replaceable class="parameter">tablespace_name</replaceable> is the name
of the tablespace in which the new dynamic table is to be created.
If not specified, <xref linkend="guc-default-tablespace"/> is consulted.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><replaceable>query</replaceable></term>
<listitem>
<para>
A <link linkend="sql-select"><command>SELECT</command></link>, <link linkend="sql-table"><command>TABLE</command></link>,
or <link linkend="sql-values"><command>VALUES</command></link> command. This query will run within a
security-restricted operation; in particular, calls to functions that
themselves create temporary tables will fail.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><literal>WITH [ NO ] DATA</literal></term>
<listitem>
<para>
This clause specifies whether or not the dynamic table should be
populated at creation time. If not, the dynamic table will be
flagged as unscannable and cannot be queried until <command>REFRESH
DYNAMIC TABLE</command> is used. Also, if the view is IMMV,
triggers for maintaining the view are not created.
</para>
</listitem>
</varlistentry>

</variablelist>
</refsect1>

<refsect1>
<title>Compatibility</title>

<para>
<command>CREATE DYNAMIC TABLE</command> is a
<productname>Cloudberry</productname> extension.
</para>
</refsect1>

<refsect1>
<title>See Also</title>

<simplelist type="inline">
<member><xref linkend="sql-createtableas"/></member>
<member><xref linkend="sql-createview"/></member>
<member><xref linkend="sql-dropmaterializedview"/></member>
<member><xref linkend="sql-refreshmaterializedview"/></member>
</simplelist>
</refsect1>

</refentry>
116 changes: 116 additions & 0 deletions doc/src/sgml/ref/drop_dynamic_table.sgml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
<!--
doc/src/sgml/ref/drop_dynamic_table.sgml
PostgreSQL documentation
-->

<refentry id="sql-dropdynamictable">
<indexterm zone="sql-dropdynamictable">
<primary>DROP DYNAMIC TABLE</primary>
</indexterm>

<refmeta>
<refentrytitle>DROP DYNAMIC TABLE</refentrytitle>
<manvolnum>7</manvolnum>
<refmiscinfo>SQL - Language Statements</refmiscinfo>
</refmeta>

<refnamediv>
<refname>DROP DYNAMIC TABLE</refname>
<refpurpose>remove a dynamic table</refpurpose>
</refnamediv>

<refsynopsisdiv>
<synopsis>
DROP DYNAMIC TABLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable> [, ...] [ CASCADE | RESTRICT ]
</synopsis>
</refsynopsisdiv>

<refsect1>
<title>Description</title>

<para>
<command>DROP DYNAMIC TABLE</command> drops an existing dynamic
table. To execute this command you must be the owner of the dynamic
table. Since every dynamic table has a auto refresh process of pg_task
job, drop a dynamic table will drop that job too.
</para>
</refsect1>

<refsect1>
<title>Parameters</title>

<variablelist>
<varlistentry>
<term><literal>IF EXISTS</literal></term>
<listitem>
<para>
Do not throw an error if the dynamic table does not exist. A notice
is issued in this case.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><replaceable class="parameter">name</replaceable></term>
<listitem>
<para>
The name (optionally schema-qualified) of the dynamic table to
remove.
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><literal>CASCADE</literal></term>
<listitem>
<para>
Automatically drop objects that depend on the dynamic table (such as
other materialized views, or regular views or pg_task jobs),
and in turn all objects that depend on those objects
(see <xref linkend="ddl-depend"/>).
</para>
</listitem>
</varlistentry>

<varlistentry>
<term><literal>RESTRICT</literal></term>
<listitem>
<para>
Refuse to drop the dynamic table if any objects depend on it. This
is the default.
</para>
</listitem>
</varlistentry>
</variablelist>
</refsect1>

<refsect1>
<title>Examples</title>

<para>
This command will remove the dynamic table called
<literal>order_summary</literal>:
<programlisting>
DROP DYNAMIC TABLE order_summary;
</programlisting></para>
</refsect1>

<refsect1>
<title>Compatibility</title>

<para>
<command>DROP DYNAMIC TABLE</command> is a
<productname>Cloudberry</productname> extension.
</para>
</refsect1>

<refsect1>
<title>See Also</title>

<simplelist type="inline">
<member><xref linkend="sql-createdynamictable"/></member>
<member><xref linkend="sql-refreshdynamictable"/></member>
</simplelist>
</refsect1>

</refentry>
Loading

0 comments on commit 25faf4a

Please sign in to comment.