diff --git a/docs/MatrixOne/Contribution-Guide/How-to-Contribute/preparation.md b/docs/MatrixOne/Contribution-Guide/How-to-Contribute/preparation.md index 751c1efc1..508a53c7c 100644 --- a/docs/MatrixOne/Contribution-Guide/How-to-Contribute/preparation.md +++ b/docs/MatrixOne/Contribution-Guide/How-to-Contribute/preparation.md @@ -16,7 +16,7 @@ These introductions will help you go through you the key concepts and user detai ### Roadmap -MatrixOne v24.1.2.4 has been released, you can see [Release Notes](../../Release-Notes/v24.1.2.4.md) know more information. +MatrixOne v24.2.0.0 has been released, you can see [Release Notes](../../Release-Notes/v24.2.0.0.md) know more information. For the long-term project roadmap, please refer to [MatrixOne roadmap](https://github.com/matrixorigin/matrixone/issues/613) for a more general overview. diff --git a/docs/MatrixOne/Deploy/deploy-MatrixOne-cluster.md b/docs/MatrixOne/Deploy/deploy-MatrixOne-cluster.md index a0dd0dd65..fcbf54d84 100644 --- a/docs/MatrixOne/Deploy/deploy-MatrixOne-cluster.md +++ b/docs/MatrixOne/Deploy/deploy-MatrixOne-cluster.md @@ -648,7 +648,7 @@ mysql -h $(kubectl get svc/mo-tp-cn -n mo-hn -o jsonpath='{.spec.clusterIP}') -P mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 163 -Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne +Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Copyright (c) 2000, 2023, Oracle and/or its affiliates. diff --git a/docs/MatrixOne/Deploy/deploy-matrixone-single-with-s3.md b/docs/MatrixOne/Deploy/deploy-matrixone-single-with-s3.md index 7107414dc..295b91bf6 100644 --- a/docs/MatrixOne/Deploy/deploy-matrixone-single-with-s3.md +++ b/docs/MatrixOne/Deploy/deploy-matrixone-single-with-s3.md @@ -736,7 +736,7 @@ cnservices = [ ```bash #Configure cn.toml, tn.toml and log.toml first. -mo_ctl set_conf MO_CONTAINER_IMAGE=matrixorigin/matrixone/1.2.4 #Setting up mirroring +mo_ctl set_conf MO_CONTAINER_IMAGE=matrixorigin/matrixone/2.0.0 #Setting up mirroring mo_ctl set_conf MO_CONTAINER_NAME=mo # Setting the container name mo_ctl set_conf MO_CONTAINER_CONF_HOST_PATH=/data/mo_confs/ # Set the directory on the host machine where the mo configuration file is stored mo_ctl set_conf MO_CONTAINER_CONF_CON_FILE="/etc/launch.toml" # Set the path to the configuration file inside the container when the container starts up diff --git a/docs/MatrixOne/Deploy/update-MatrixOne-cluster.md b/docs/MatrixOne/Deploy/update-MatrixOne-cluster.md index 7e2b9ba70..be6a63ab6 100644 --- a/docs/MatrixOne/Deploy/update-MatrixOne-cluster.md +++ b/docs/MatrixOne/Deploy/update-MatrixOne-cluster.md @@ -69,7 +69,7 @@ According to the introduction in [MatrixOne Distributed Cluster Deployment](depl root@master0 ~]# mysql -h $(kubectl get svc/mo-tp-cn -n mo-hn -o jsonpath='{.spec.clusterIP}') -P 6001 -uroot -p111 Welcome to the MariaDB monitor. Commands end with ; or \g. Your MySQL connection id is 1005 - Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne + Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. diff --git a/docs/MatrixOne/Develop/Publish-Subscribe/pub-sub-overview.md b/docs/MatrixOne/Develop/Publish-Subscribe/pub-sub-overview.md index 6a2cf6bfc..e77414b26 100644 --- a/docs/MatrixOne/Develop/Publish-Subscribe/pub-sub-overview.md +++ b/docs/MatrixOne/Develop/Publish-Subscribe/pub-sub-overview.md @@ -1,384 +1,261 @@ -# Publish Subscription +# Publish and subscribe -A database's Publish-Subscribe (Pub/Sub) is a messaging model in which a **publisher** sends a message to one or more **subscribers**, who in turn receive and process the message. The **subscriber** sends the message to one or more subscribers. In this model, publishers and subscribers are loosely coupled and do not need to communicate directly with each other, thus increasing application scalability and flexibility. +Database publish-subscribe (Publish-Subscribe, referred to as Pub/Sub) is a messaging model in which the publisher sends messages to one or more subscribers, and the subscribers The message is received and processed. In this mode, publishers and subscribers are loosely coupled and do not require direct communication between them, thus improving the scalability and flexibility of the application. -In databases, publish subscriptions are often used in scenarios such as real-time data updates, cache synchronization, and business event notifications. For example, when data changes for a table in a database, subscribers can be notified in real time through the publish subscription feature, enabling real-time data synchronization and processing. In addition, notification of business events, such as an order being cancelled, an inventory being insufficient, etc., can be achieved by publishing a subscription function. +In databases, publish and subscribe functions are usually used in real-time data updates, cache synchronization, business event notifications and other scenarios. For example, when data in a certain table in the database changes, subscribers can be notified in real time through the publish and subscribe function, thereby achieving real-time data synchronization and processing. In addition, the publish and subscribe function can also be used to implement notification of business events, such as an order being canceled, a certain inventory quantity being insufficient, etc. -Typically, a database's publish subscription function consists of two parts: **the publisher** and the **subscriber**. The **publisher** is responsible for publishing the message, while the **subscriber** subscribes to the corresponding message for data synchronization purposes. There can be a many-to-many relationship between the publisher and the subscriber, i.e. one publisher can post messages to multiple subscribers and one subscriber can subscribe to multiple messages/data. +Usually, the publish and subscribe function of the database consists of two parts: **Publisher**and **Subscriber**. **Publisher**is responsible for publishing messages, while **Subscribers**subscribe to corresponding messages to achieve data synchronization. There can be a many-to-many relationship between publishers and subscribers, that is, a publisher can publish messages to multiple subscribers, and a subscriber can also subscribe to multiple messages/data. ## Application scenarios -The publish subscription feature has several typical application scenarios: +The publish and subscribe function has a variety of typical application scenarios: -- **Data synchronization**: When one database needs to be synchronized with another, the publish subscription feature can be used to send data changes to the subscriber database. For example, when a website needs to transfer data from one geographic location to another, the publish subscription feature can be used to ensure data synchronization between two databases. +- **Data Synchronization**: When one database needs to be synchronized with another database, the publish-subscribe function can be used to send data changes to the subscriber database. For example, when a website needs to transfer data from one geographical location to another, the publish-subscribe feature can be used to ensure data synchronization between the two databases. -- **Business data distribution**: The publish subscription feature can be used to distribute business data to different systems or business processes. For example, when a bank needs to distribute customer account information to multiple business systems, the publish subscription feature can be used to distribute data to the appropriate systems, ensuring data consistency across business processes. +- **Business Data Distribution**: The publish and subscribe function can be used to distribute business data to different systems or business processes. For example, when a bank needs to distribute customer account information to multiple business systems, the publish and subscribe function can be used to distribute data to the corresponding systems to ensure data consistency between various business processes. -- **Data backup**: The publish subscription feature can be used to back up data. For example, when one database needs to be backed up to another, the publish subscription feature can be used to back up data to the subscriber database to restore data in the event of a primary database failure. +- **Data Backup**: The publish and subscribe function can be used to back up data. For example, when one database needs to be backed up to another database, the publish-subscribe feature can be used to back up the data to the subscriber database so that the data can be restored in the event of a failure of the primary database. -- **Real-time data processing**: The publish subscription feature can be used to enable real-time data processing. For example, when a website needs to process data from different users, the publish subscription feature can be used to transfer the data to a handler for processing in order to enable real-time data analysis and decision-making. +- **Real-time data processing**: The publish and subscribe function can be used to implement real-time data processing. For example, when a website needs to process data from different users, the publish and subscribe function can be used to transfer the data to the handler for processing in order to achieve real-time data analysis and decision-making. -## Noun interpretation +## Glossary -- **Publishing**: In a database, publishing usually refers to setting a database object to a state accessible to other tenants. This is an important step in data sharing and replication, where published objects can be subscribed to and acquired by other tenants. +- **Publish**: In databases, publishing usually refers to setting a database object to a state that can be accessed by other tenants. This is an important step in data sharing and replication. The published objects can be subscribed by other tenants and obtain the data. -- **Subscription**: A subscription is when a database chooses to receive and copy data from published database objects. +- **Subscription**: A subscription is when a database chooses to receive and replicate data from published database objects. -- **Publisher (Pub**): A publisher is a database that performs publishing operations. The publishing side is responsible for creating and managing published objects, as well as managing access to databases that subscribe to that published object. +- **Publisher (Pub)**: The publisher is the database that performs publishing operations. The publisher is responsible for creating and managing published objects, as well as managing access rights to databases that subscribe to the published objects. -- **Subscriber (Sub)**: A subscriber is a tenant who subscribes to a publishing object. +- **Subscriber (Sub)**: The subscriber is the tenant that subscribes to the published object. -- **Publish object**: A publish object is a database object created on the publish side and set to publishable, i.e., a database. The data for these objects can be accessed and copied by the subscriber. +- **Publish object**: The publication object is a database object created and set as publishable on the publishing side, that is, the database. The data of these objects can be accessed and copied by subscribers. -- **Subscription objects**: Subscription objects are publishing objects that are copied and stored on the subscription side. The subscription object's data is updated based on the publisher's data. +- **Subscription Object**: The subscription object is the publication object copied and stored on the subscription side. The data of the subscription object will be updated according to the data of the publishing side. -## Publish Subscription Scope Description +## Publish subscription scope description -### Publish/Subscribe to Scope of Application +### Publish/subscribe application scope -- **Publisher (Pub)**and**Sub (Sub)** are both tenants of MatrixOne. +**Publisher (Pub)**and **Subscriber (Sub)**are both tenants of MatrixOne. -### Publishable/Subscribeable Permission Range +### Publishable/subscribeable permission scope -- **Publisher (Pub)**Only the ACCOUNTADMIN or MOADMIN role can create publications and subscriptions. -- **Subscriber (Sub)**Access to subscription data is manipulated by the ACCOUNTADMIN or MOADMIN role. +- **Publisher (Pub)**Only ACCOUNTADMIN or MOADMIN roles can create publications and subscriptions. +- **Subscriber (Sub)**has access to subscription data permissions operated by ACCOUNTADMIN or MOADMIN roles. -### Publish/Subscribe Data Range +### Publish/subscribe data range -- A **publish** can only be associated with a single database. -- Publishing and subscribing is only implemented at the database level; direct table-level publishing and subscribing is not currently supported. -- The **Subscription side** only has read access to the **Subscription library**. -- If **Publisher** adjusts the sharing scope of a publish, those **Subscribers** that are not in the new scope will have invalid access to this **Subscription library** if they have already created a subscription library. -- If the **Publisher** modifies the posting, then the **Subscriber** will see the update without additional action. -- If **Pub** tries to delete a published database, the deletion will not succeed. -- If **Publisher** deletes **Publish** but the corresponding object in the subscription database still exists, accessing this object by **Subscriber (Sub)** will trigger an error and the corresponding **Subscription** will need to be deleted by **Subscriber (Sub)**. -- If the **Publishing side (Pub)** deletes the **Publishing object**, but the corresponding object in the subscription library still exists, then the **Subscribing side (Sub)** accessing this object triggers an error and requires the **Subscribing side (Sub)** to delete the corresponding **Subscription object**. +- A **Publication**can only be associated with a single database. +- Supports database-level and table-level publishing and subscription. +- **Subscriber**only has read permissions on **Subscription Library**. +- If **Publisher (Pub)**adjusts the publishing sharing scope, those **Subscribers (Sub)**that are not within the new scope will have a subscription library created, then the **Subscription library**Access will be invalid. +- If **Publisher (Pub)**modifies the published content, then **Subscriber (Sub)**can see the update without additional operations. +- If **publisher (Pub)**tries to delete an already published database, the deletion will not be successful. +- If **Publisher (Pub)**deletes **Publish**, but the corresponding object in the subscription library still exists, then **Subscriber (Sub)**will trigger an error when accessing this object, which needs to be **Subscriber (Sub)**deletes the corresponding **Subscription**. +- If **publisher (Pub)**deletes **publish object**, but the corresponding object in the subscription library still exists, then **subscriber (Sub)**will trigger an error when accessing this object, which requires The corresponding **subscription object**is deleted by **Subscriber**. -### Publish Subscription Example +## Publish and subscribe example -This chapter will give an example of how three tenants, sys, acc1, and acc2, currently exist in a MatrixOne cluster, operating on the three tenants in order of operation: +Suppose there is a cross-regional retail company. The central warehouse database needs to publish inventory and product price changes, and each branch database subscribes to these changes to ensure that inventory and price information are synchronized in each branch system. -![](https://github.com/matrixorigin/artwork/blob/main/docs/develop/pub-sub/data-share.png?raw=true) +```sql +drop account if exists acc1; +create account acc1 admin_name = 'test_account' identified by '111'; +drop account if exists acc2; +create account acc2 admin_name = 'test_account' identified by '111'; +``` -1. **Publisher**: sys tenant creates database sub1 with table t1 and publishes pub1: +1. Create a central warehouse database and initialize data + + Create a central warehouse database under the `sys` tenant, containing inventory tables and product price lists, and insert some test data to verify the functionality. ```sql - create database sub1; - create table sub1.t1(a int,b int); - create publication pub1 database sub1; - mysql> show publications; - +-------------+----------+---------------------+-------------+-------------+----------+ - | publication | database | create_time | update_time | sub_account | comments | - +-------------+----------+---------------------+-------------+-------------+----------+ - | pub1 | sub1 | 2024-04-23 10:28:15 | NULL | * | | - +-------------+----------+---------------------+-------------+-------------+----------+ - 1 row in set (0.01 sec) + -- Create a central warehouse database + create database central_warehouse_db; + use central_warehouse_db; + + -- Create inventory table + CREATE TABLE inventory ( + product_id INT PRIMARY KEY, + product_name VARCHAR(100), + stock_quantity INT + ); + + -- Create a product price list + CREATE TABLE products ( + product_id INT PRIMARY KEY, + product_name VARCHAR(100), + price DECIMAL(10, 2) + ); + + -- Insert initial data into inventory table + INSERT INTO inventory (product_id, product_name, stock_quantity) VALUES + (1, 'Laptop', 100), + (2, 'Smartphone', 200), + (3, 'Tablet', 150); + + -- Insert initial data into product price list + INSERT INTO products (product_id, product_name, price) VALUES + (1, 'Laptop', 999.99), + (2, 'Smartphone', 599.99), + (3, 'Tablet', 399.99); ``` -2. **Subscribers**: acc1 and acc2 both create the subscription library syssub1, resulting in the shared data table t1: - - ```sql - -- The all option allows you to see all the subscriptions that you have permission to subscribe to, and the unsubscribed sub_time and sub_name are null, so if you don't add all, you can only see the information that you have already subscribed to. - mysql> show subscriptions all; - +----------+-------------+--------------+---------------------+----------+----------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+----------+ - | pub1 | sys | sub1 | 2024-04-23 10:28:15 | NULL | NULL | - +----------+-------------+--------------+---------------------+----------+----------+ - 1 row in set (0.01 sec) - - -- The sql statements for creating a subscription library are the same for both acc1 and acc2, so I won't repeat them here. - create database syssub1 from sys publication pub1; - use syssub1; - - mysql> show subscriptions; - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub1 | sys | sub1 | 2024-04-23 10:28:15 | syssub1 | 2024-04-23 10:35:13 | - +----------+-------------+--------------+---------------------+----------+---------------------+ - 1 row in set (0.00 sec) - - mysql> show tables; - +--------------------+ - | Tables_in_syssub1 | - +--------------------+ - | t1 | - +--------------------+ - 2 rows in set (0.02 sec) - ``` +2. Configure release -3. **Publisher**: sys tenant create data table t2: + Create a database-level publication in the central warehouse database to publish data updates of all tables to tenant `acc1`. If a branch only cares about changes in product prices, it can create a table-level publication to publish only updates to the product price list to tenant `acc2`. ```sql - create table sub1.t2(a text); + create publication db_warehouse_pub database central_warehouse_db account acc1; + create publication tab_products_pub database central_warehouse_db table products account acc2; ``` -4. **Subscribers**: acc1 and acc2 get shared data tables t1 and t2: +3. Subscribe to publications individually in branch tenants ```sql - use syssub1; - mysql> show tables; - +-------------------+ - | Tables_in_syssub1 | - +-------------------+ - | t1 | - | t2 | - +-------------------+ - 2 rows in set (0.01 sec) - ``` + -- Subscribe to db_warehouse_pub in acc1 + create database db_warehouse_sub from sys publication db_warehouse_pub; -5. **Publisher**: sys tenant creates database sub2 with table t1 and publishes pub2 to tenant acc1 - - ```sql - create database sub2; - create table sub2.t1(a float); - create publication pub2 database sub2 account acc1; - ``` - -6. **Subscribers**:acc1 and acc2 both create subscription library syssub2,acc1 gets shared data table t1;acc2 fails to create subscription library syssub2: - - - acc1 - - ```sql - mysql> show subscriptions all; - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub1 | sys | sub1 | 2024-04-23 10:28:15 | syssub1 | 2024-04-23 10:30:43 | - | pub2 | sys | sub2 | 2024-04-23 10:40:54 | NULL | NULL | - +----------+-------------+--------------+---------------------+----------+---------------------+ - 2 rows in set (0.01 sec) - - create database syssub2 from sys publication pub2; - use syssub2; + mysql> show subscriptions; + +------------------+-------------+----------------------+------------+-------------+---------------------+------------------+---------------------+--------+ + | pub_name | pub_account | pub_database | pub_tables | pub_comment | pub_time | sub_name | sub_time | status | + +------------------+-------------+----------------------+------------+-------------+---------------------+------------------+---------------------+--------+ + | db_warehouse_pub | sys | central_warehouse_db | * | | 2024-10-15 11:58:04 | db_warehouse_sub | 2024-10-15 11:59:47 | 0 | + +------------------+-------------+----------------------+------------+-------------+---------------------+------------------+---------------------+--------+ + 1 row in set (0.01 sec) - mysql> show subscriptions all; - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub2 | sys | sub2 | 2024-04-23 10:40:54 | syssub2 | 2024-04-23 10:42:31 | - | pub1 | sys | sub1 | 2024-04-23 10:28:15 | syssub1 | 2024-04-23 10:30:43 | - +----------+-------------+--------------+---------------------+----------+---------------------+ - 2 rows in set (0.01 sec) + use db_warehouse_sub; mysql> show tables; - +--------------------+ - | Tables_in_syssub2 | - +--------------------+ - | t1 | - +--------------------+ - 2 rows in set (0.02 sec) - ``` + +----------------------------+ + | Tables_in_db_warehouse_sub | + +----------------------------+ + | inventory | + | products | + +----------------------------+ + 2 rows in set (0.01 sec) - - acc2 + mysql> select * from inventory; + +------------+--------------+----------------+ + | product_id | product_name | stock_quantity | + +------------+--------------+----------------+ + | 1 | Laptop | 100 | + | 2 | Smartphone | 200 | + | 3 | Tablet | 150 | + +------------+--------------+----------------+ + 3 rows in set (0.01 sec) + + mysql> select * from products; + +------------+--------------+--------+ + | product_id | product_name | price | + +------------+--------------+--------+ + | 1 | Laptop | 999.99 | + | 2 | Smartphone | 599.99 | + | 3 | Tablet | 399.99 | + +------------+--------------+--------+ + 3 rows in set (0.01 sec) + + -- Subscribe to tab_products_pub in acc2 + create database tab_products_sub from sys publication tab_products_pub; - ```sql - -- acc2 看不到 pub2,因为没有订阅权限 - mysql> show subscriptions all; - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub1 | sys | sub1 | 2024-04-23 10:28:15 | syssub1 | 2024-04-23 10:35:13 | - +----------+-------------+--------------+---------------------+----------+---------------------+ + mysql> show subscriptions; + +------------------+-------------+----------------------+------------+-------------+---------------------+------------------+---------------------+--------+ + | pub_name | pub_account | pub_database | pub_tables | pub_comment | pub_time | sub_name | sub_time | status | + +------------------+-------------+----------------------+------------+-------------+---------------------+------------------+---------------------+--------+ + | tab_products_pub | sys | central_warehouse_db | products | | 2024-10-15 11:58:04 | tab_products_sub | 2024-10-15 13:59:22 | 0 | + +------------------+-------------+----------------------+------------+-------------+---------------------+------------------+---------------------+--------+ 1 row in set (0.01 sec) - mysql> create database syssub2 from sys publication pub2; - ERROR 20101 (HY000): internal error: the account acc2 is not allowed to subscribe the publication pub2 - ``` - -7. **Publisher**: sys tenant modifies publishing pub2 to all tenants: - - ```sql - alter publication pub2 account all; - mysql> show publications; - +-------------+----------+---------------------+---------------------+-------------+----------+ - | publication | database | create_time | update_time | sub_account | comments | - +-------------+----------+---------------------+---------------------+-------------+----------+ - | pub2 | sub2 | 2024-04-23 10:40:54 | 2024-04-23 10:47:53 | * | | - | pub1 | sub1 | 2024-04-23 10:28:15 | NULL | * | | - +-------------+----------+---------------------+---------------------+-------------+----------+ - 2 rows in set (0.00 sec) - ``` - -8. **Subscriber**:acc2 Created subscription library syssub2 successfully with shared data table t1: - - ```sql - -- acc2 can now see pub2. - mysql> show subscriptions all; - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub1 | sys | sub1 | 2024-04-23 10:28:15 | syssub1 | 2024-04-23 10:35:13 | - | pub2 | sys | sub2 | 2024-04-23 10:40:54 | NULL | NULL | - +----------+-------------+--------------+---------------------+----------+---------------------+ - 2 rows in set (0.00 sec) - - create database syssub2 from sys publication pub2; - use syssub2; - - mysql> show subscriptions all; - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub2 | sys | sub2 | 2024-04-23 10:40:54 | syssub2 | 2024-04-23 10:50:43 | - | pub1 | sys | sub1 | 2024-04-23 10:28:15 | syssub1 | 2024-04-23 10:35:13 | - +----------+-------------+--------------+---------------------+----------+---------------------+ - 2 rows in set (0.00 sec) + use tab_products_sub; mysql> show tables; - +--------------------+ - | Tables_in_syssub2 | - +--------------------+ - | t1 | - +--------------------+ - 2 rows in set (0.02 sec) - ``` - -9. **Publisher**:sys tenant delete publish pub1: - - ```sql - drop publication pub1; - mysql> show publications; - +-------------+----------+---------------------+---------------------+-------------+----------+ - | publication | database | create_time | update_time | sub_account | comments | - +-------------+----------+---------------------+---------------------+-------------+----------+ - | pub2 | sub2 | 2024-04-23 10:40:54 | 2024-04-23 10:47:53 | * | | - +-------------+----------+---------------------+---------------------+-------------+----------+ - 1 row in set (0.00 sec) - ``` - -10. **Subscribers**:acc1,acc2 Connection to syspub1 failed: + +----------------------------+ + | Tables_in_tab_products_sub | + +----------------------------+ + | products | + +----------------------------+ + 1 row in set (0.01 sec) - ```sql - mysql> use syssub1; - ERROR 20101 (HY000): internal error: there is no publication pub1 + mysql> select * from products; + +------------+--------------+--------+ + | product_id | product_name | price | + +------------+--------------+--------+ + | 1 | Laptop | 999.99 | + | 2 | Smartphone | 599.99 | + | 3 | Tablet | 399.99 | + +------------+--------------+--------+ + 3 rows in set (0.01 sec) ``` -11. **Publisher**: sys tenant creates new database sub1\_new and republishes it as pub1 +4. Data changes - ```sql - create database sub1_new; - use sub1_new; - create table t3(n1 int); - insert into t3 values (1); - create publication pub1 database sub1_new; - mysql> show publications; - +-------------+----------+---------------------+---------------------+-------------+----------+ - | publication | database | create_time | update_time | sub_account | comments | - +-------------+----------+---------------------+---------------------+-------------+----------+ - | pub2 | sub2 | 2024-04-23 10:40:54 | 2024-04-23 10:47:53 | * | | - | pub1 | sub1_new | 2024-04-23 10:59:11 | NULL | * | | - +-------------+----------+---------------------+---------------------+-------------+----------+ - 2 rows in set (0.00 sec) - ``` + On the publishing side of data, the publishing and subscription mechanism will synchronize these changes to the subscribing side. -12. **Subscribers**: acc1, acc2 Connect to syspub1 and see what's new in pub1, meaning if the publisher changes what's published, the subscriber doesn't have to do anything to see the update. - ```sql - use syssub1; - mysql> show subscriptions; - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub2 | sys | sub2 | 2024-04-23 10:40:54 | syssub2 | 2024-04-23 10:42:31 | - | pub1 | sys | sub1_new | 2024-04-23 10:59:11 | syssub1 | 2024-04-23 10:30:43 | - +----------+-------------+--------------+---------------------+----------+---------------------+ + -- Modify the inventory quantity in the inventory table + UPDATE inventory SET stock_quantity = 80 WHERE product_id = 1; + + -- Delete a row in the product price list + DELETE FROM products WHERE product_id = 2; + + mysql> select * from inventory; + +------------+--------------+----------------+ + | product_id | product_name | stock_quantity | + +------------+--------------+----------------+ + | 1 | Laptop | 80 | + | 2 | Smartphone | 200 | + | 3 | Tablet | 150 | + +------------+--------------+----------------+ + 3 rows in set (0.00 sec) + + mysql> select * from products; + +------------+--------------+--------+ + | product_id | product_name | price | + +------------+--------------+--------+ + | 1 | Laptop | 999.99 | + | 3 | Tablet | 399.99 | + +------------+--------------+--------+ 2 rows in set (0.01 sec) - - mysql> show tables; - +-------------------+ - | Tables_in_syssub1 | - +-------------------+ - | t3 | - +-------------------+ - 1 row in set (0.01 sec) - - mysql> select * from t3; - +------+ - | n1 | - +------+ - | 1 | - +------+ - 1 row in set (0.01 sec) ``` -13. **Subscriber**:acc1 Delete subscription: +5. View changes on the subscription side ```sql - -- Remove a subscription by drop database - drop database syssub1; - mysql> show subscriptions; - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub2 | sys | sub2 | 2024-04-23 10:40:54 | syssub2 | 2024-04-23 10:42:31 | - +----------+-------------+--------------+---------------------+----------+---------------------+ - 1 row in set (0.00 sec) + -- Check stock availability in acc1 + mysql> select * from inventory ; + +------------+--------------+----------------+ + | product_id | product_name | stock_quantity | + +------------+--------------+----------------+ + | 1 | Laptop | 80 | + | 2 | Smartphone | 200 | + | 3 | Tablet | 150 | + +------------+--------------+----------------+ + 3 rows in set (0.01 sec) + + -- Check product prices in acc2 + + mysql> select * from products; + +------------+--------------+--------+ + | product_id | product_name | price | + +------------+--------------+--------+ + | 1 | Laptop | 999.99 | + | 3 | Tablet | 399.99 | + +------------+--------------+--------+ + 2 rows in set (0.01 sec) ``` -14. **Publisher**: Before a sys tenant deletes a published database, delete its corresponding publication: +6. Delete publication - ```sql - mysql> drop database sub1_new; - ERROR 20101 (HY000): internal error: can not drop database 'sub1_new' which is publishing - mysql> drop publication pub1; - Query OK, 0 rows affected (0.00 sec) - - mysql> drop database sub1_new; - Query OK, 1 row affected (0.03 sec) - ``` - -15. **Publisher**: sys tenant modifies publication: - - ```sql - alter publication pub2 comment "this is pub2";--alter comments - mysql> show publications; - create database new_sub2; - create table new_sub2.new_t (xxx int); - insert into new_sub2.new_t values (123); - alter publication pub2 database new_sub2;--alter database - mysql> show publications; - +-------------+----------+---------------------+---------------------+-------------+--------------+ - | publication | database | create_time | update_time | sub_account | comments | - +-------------+----------+---------------------+---------------------+-------------+--------------+ - | pub2 | new_sub2 | 2024-04-23 10:40:54 | 2024-04-23 11:04:20 | * | this is pub2 | - +-------------+----------+---------------------+---------------------+-------------+--------------+ - 1 row in set (0.00 sec) - ``` - -16. **Subscribers**: acc1, acc2 View the subscription to see the modified content of the publishing database: + In some cases, when a warehouse is out of use, or data publishing no longer requires synchronization of a certain table, you can use DROP PUBLICATION to release the publishing relationship to prevent unnecessary resource consumption. ```sql - mysql> show subscriptions; - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | - +----------+-------------+--------------+---------------------+----------+---------------------+ - | pub2 | sys | new_sub2 | 2024-04-23 10:40:54 | syssub2 | 2024-04-23 10:42:31 | - +----------+-------------+--------------+---------------------+----------+---------------------+ - 1 row in set (0.00 sec) - - use syssub2; - mysql> show tables; - +-------------------+ - | Tables_in_syssub2 | - +-------------------+ - | new_t | - +-------------------+ - 1 row in set (0.00 sec) - - mysql> select * from new_t; - +------+ - | xxx | - +------+ - | 123 | - +------+ - 1 row in set (0.00 sec) + -- Delete product price list publication + drop publication tab_products_sub; ``` -## Reference Documents +## Reference documentation -### Publisher Reference Documentation +### Publisher reference documentation - [CREATE PUBLICATION](../../Reference/SQL-Reference/Data-Definition-Language/create-publication.md) - [ALTER PUBLICATION](../../Reference/SQL-Reference/Data-Definition-Language/alter-publication.md) @@ -386,7 +263,7 @@ This chapter will give an example of how three tenants, sys, acc1, and acc2, cur - [SHOW PUBLICATIONS](../../Reference/SQL-Reference/Other/SHOW-Statements/show-publications.md) - [SHOW CREATE PUBLICATION](../../Reference/SQL-Reference/Other/SHOW-Statements/show-create-publication.md) -### Subscriber Reference Documents +### Subscriber Reference Documentation - [CREATE...FROM...PUBLICATION...](../../Reference/SQL-Reference/Data-Definition-Language/create-subscription.md) - [SHOW SUBSCRIPTIONS](../../Reference/SQL-Reference/Other/SHOW-Statements/show-subscriptions.md) diff --git a/docs/MatrixOne/Develop/connect-mo/configure-mo-ssl-connection.md b/docs/MatrixOne/Develop/connect-mo/configure-mo-ssl-connection.md index 808028f79..a864b120a 100644 --- a/docs/MatrixOne/Develop/connect-mo/configure-mo-ssl-connection.md +++ b/docs/MatrixOne/Develop/connect-mo/configure-mo-ssl-connection.md @@ -83,7 +83,7 @@ To test the SSL configuration, perform the following steps: Current pager: stdout Using outfile: '' Using delimiter: ; - Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne + Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Protocol version: 10 Connection: 127.0.0.1 via TCP/IP Client characterset: utf8mb4 diff --git a/docs/MatrixOne/Develop/connect-mo/database-client-tools.md b/docs/MatrixOne/Develop/connect-mo/database-client-tools.md index 7a4fdea1f..85b383e01 100644 --- a/docs/MatrixOne/Develop/connect-mo/database-client-tools.md +++ b/docs/MatrixOne/Develop/connect-mo/database-client-tools.md @@ -41,7 +41,7 @@ Make sure you have already [installed and launched MatrixOne](../../Get-Started/ ``` Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1031 - Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne + Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. diff --git a/docs/MatrixOne/Develop/connect-mo/python-connect-to-matrixone.md b/docs/MatrixOne/Develop/connect-mo/python-connect-to-matrixone.md index 23490040b..d3f771208 100644 --- a/docs/MatrixOne/Develop/connect-mo/python-connect-to-matrixone.md +++ b/docs/MatrixOne/Develop/connect-mo/python-connect-to-matrixone.md @@ -69,7 +69,7 @@ The PyMySQL is a pure-Python MySQL client library. ``` > python3 pymysql_connect_matrixone.py - Database version : 8.0.30-MatrixOne-v1.2.4 + Database version : 8.0.30-MatrixOne-v2.0.0 ``` ## Using sqlalchemy connect to MatrixOne diff --git a/docs/MatrixOne/Develop/export-data/select-into-outfile.md b/docs/MatrixOne/Develop/export-data/select-into-outfile.md index adc1b92f9..2af58f773 100644 --- a/docs/MatrixOne/Develop/export-data/select-into-outfile.md +++ b/docs/MatrixOne/Develop/export-data/select-into-outfile.md @@ -1,24 +1,26 @@ -# Export data by SELECT INTO +# SELECT INTO write out -There are two methods to export data with MatrixOne: +MatrixOne supports the following two ways to export data: - `SELECT INTO...OUTFILE` -- `modump` +- `mo-dump` -This document will introduce about how to export data with `SELECT INTO...OUTFILE`. +This document mainly introduces how to use `SELECT INTO...OUTFILE` to export data. -`SELECT...INTO OUTFILE` statement exports a table data into a text file on the server host. +Table data can be exported to a text file or stage on the host using the `SELECT...INTO OUTFILE` syntax. -## Syntax +## Grammar structure -The syntax for this statement combines a regular **SELECT** command with **INTO OUTFILE filename** at the end. The default output format is the same as it is for the LOAD DATA command. So, the following statement exports the **test** table into **/root/test** as a tab-delimited, linefeed-terminated file. +The `SELECT...INTO OUTFILE` syntax is a combination of the `SELECT` syntax and `INTO OUTFILE filename`. The default output format is the same as the `LOAD DATA` command. ``` -mysql> SELECT * FROM TEST - -> INTO OUTFILE '/root/test.csv'; +mysql> SELECT *FROM + -> INTO OUTFILE '|'; ``` -You can change the output format using various options to indicate how to quote and delimit columns and records. Using the following code to export the *TEST* table in a CSV format with CRLF-terminated lines: +You can change the output format using a variety of forms and options to represent how columns and records are referenced and separated. + +Use the following code to export the *TEST*table in *.csv*format. The following lines of code are displayed with carriage returns and line feeds: ``` mysql> SELECT * FROM TEST INTO OUTFILE '/root/test.csv' @@ -26,29 +28,32 @@ mysql> SELECT * FROM TEST INTO OUTFILE '/root/test.csv' -> LINES TERMINATED BY '\r\n'; ``` -The `SELECT ... INTO OUTFILE` has the following properties − +**`SELECT ... INTO OUTFILE` features are as follows**: + +- The exported file is created directly by the MatrixOne service, so the `filename` in the command line should point to the location of the server host where you need the file to be stored. MatrixOne does not currently support exporting files to the client file system. + +- `SELECT ... INTO OUTFILE` is used to export the retrieved data to a file in a format, that is, the file to be exported is directly created by the MatrixOne service, and the exported file can only be located on the server host where MatrixOne is located, so You must have a username and password to log in to the server host where MatrixOne is located, and you must have permission to retrieve files from MatrixOne. + +- You must have permission to execute `SELECT`. -- The output file is created directly by the MatrixOne server, so the filename should indicate where you want the file to be written on the server host. MatrixOne doesn't support export the file to a client-side file system. -- You must have the privilege to execute the `SELECT ... INTO` statement. -- The output file must not already exist. This prevents MatrixOne from clobbering files that may be important. -- You should have a login account on the server host or some way to retrieve the file from that host. Otherwise, the `SELECT ... INTO OUTFILE` command will most likely be of no value to you. +- Check that there are no files with the same name in the directory where the files need to be exported, otherwise they will be overwritten by the newly exported files. ## Example -### Before you start +### Preparation before starting -Make sure you have already [Deployed standalone MatrixOne](../../Get-Started/install-standalone-matrixone.md). +[Stand-alone deployment of MatrixOne](../../Get-Started/install-standalone-matrixone.md) has been completed. !!! note - If you install MatrixOne by `docker`, the directory is inside the docker image by default. To work with local directory, you need to bind a local directory to the container. In the following example, the local file system path `${local_data_path}/mo-data` is binded to the MatrixOne docker image, with a mapping to the `/mo-data` path. For more information, see [Docker Mount Volume tutorial](https://www.freecodecamp.org/news/docker-mount-volume-guide-how-to-mount-a-local-directory/). + If you installed MatrixOne via `docker`, the export directory is located in the docker image by default. If you need to mount a local directory, see the following code example: the local file system path *${local_data_path}/mo-data*is mounted into the MatrixOne Docker image and mapped to the */mo-data*path. For more information, see the [Docker Mount Volume tutorial](https://www.freecodecamp.org/news/docker-mount-volume-guide-how-to-mount-a-local-directory/). ``` -sudo docker run --name --privileged -d -p 6001:6001 -v ${local_data_path}/mo-data:/mo-data:rw matrixorigin/matrixone:1.2.4 +sudo docker run --name --privileged -d -p 6001:6001 -v ${local_data_path}/mo-data:/mo-data:rw matrixorigin/matrixone:2.0.0 ``` ### Steps -1. Create tables in MatrixOne: +1. Create a new data table in MatrixOne: ```sql create database aaa; @@ -65,21 +70,45 @@ sudo docker run --name --privileged -d -p 6001:6001 -v ${local_data_path} +------+-----------+------+ ``` -2. For installation with source code or binary file, export the table to your local directory, for example, *~/tmp/export_demo/export_datatable.txt*. +2.Data export + +- Export to local + + For installing and building MatrixOne using source code or binary files, export the table to a local directory, such as *~/tmp/export_demo/export_datatable.txt*. The command example is as follows: ``` select * from user into outfile '~/tmp/export_demo/export_datatable.txt' ``` - For installation with docker, export the your mounted directory path of container as the following example. The directory `mo-data` refers to the local path of `~/tmp/docker_export_demo/mo-data`. + Use Docker to install and start MatrixOne, and export to the directory path of your mounted container, as shown in the example below. The directory *mo-data*refers to the local path *~/tmp/docker_export_demo/mo-data*. ``` select * from user into outfile 'mo-data/export_datatable.txt'; ``` -3. Check the table in your directory `export_datatable.txt`, the result is as below: + - export to satge + + ```sql + create stage stage_fs url = 'file:///Users/admin/test'; + select * from user into outfile 'stage://stage_fs/user.csv'; + ``` + +3. Check the export status: + - Export to local + ``` + (base) admin@192 test % cat export_datatable.txt + id,user_name,sex + 1,"weder","man" + 2,"tom","man" + 3,"wederTom","man" + ``` + + - Export to stage + + ```bash + (base) admin@192 test % cat user.csv id,user_name,sex 1,"weder","man" 2,"tom","man" diff --git a/docs/MatrixOne/Develop/import-data/bulk-load/1.1-load-s3.md b/docs/MatrixOne/Develop/import-data/bulk-load/1.1-load-s3.md deleted file mode 100644 index be1a32007..000000000 --- a/docs/MatrixOne/Develop/import-data/bulk-load/1.1-load-s3.md +++ /dev/null @@ -1,287 +0,0 @@ -# Load data from S3 - -## Overview - -S3 (Simple Storage Service) object storage refers to Amazon's Simple Storage Service. You can also store almost any type and size of data with S3-compatible object storage, including data lakes, cloud-native applications, and mobile apps. If you are unfamiliar with S3 object service, you may look up some basic introductions in [AWS](https://docs.aws.amazon.com/s3/index.html). - -AWS S3 has been remarkably successful for over a decade, so it became the de facto standard for object storage. Thus almost every mainstream public cloud vendors provide an S3-compatible object storage service. - -MatrixOne supports loading files from S3-compatible object storage services into databases. MatrixOne supports AWS and mainstream cloud vendors in China (Alibaba Cloud, Tencent Cloud). - -In MatrixOne, there are two methods to import the data from S3-compatible object storage: - -* Use `Load data` with an s3option to load the file into MatrixOne. This method will load the data into MatrixOne, and all next queries will happen inside MatrixOne. -* Create an `external table` with an s3option mapping to an S3 file, and query this external table directly. This method allows data access through an S3-compatible object storage service; each query's networking latency will be counted. - -When importing data from public clouds like AWS S3 or Alibaba Cloud OSS, the appropriate access permissions are required. Generally, there are two methods to choose from: role-based access and key-based access. - -- **Role-Based Access**: This involves creating a specific role within the public cloud account where the data is stored. This role is authorized to grant MatrixOne applications the necessary data access permissions. This approach is more secure and offers greater ease in managing and adjusting data access permissions. - -- **Key-Based Access**: Data can be accessed using the `Access Key ID` and `Secret Access Key` of a user with the required data access permissions. While this method is relatively straightforward, it is less secure because exposing the `Access Key ID` and `Secret Access Key` could lead to severe consequences. - -## Method 1: LOAD DATA - -### Syntax - -```sql -LOAD DATA - | URL s3options {"endpoint"='', "access_key_id"='', "secret_access_key"='', "bucket"='', "role_arn"='xxxx', "external_id"='yyy', "filepath"='', "region"='', "compression"=''} - INTO TABLE tbl_name - [{FIELDS | COLUMNS} - [TERMINATED BY 'string'] - [[OPTIONALLY] ENCLOSED BY 'char'] - [ESCAPED BY 'char'] - ] - [IGNORE number {LINES | ROWS}] - [PARALLEL {'TRUE' | 'FALSE'}] -``` - -**Parameter Description** - -|Parameter|Description| -|:-:|:-:| -|endpoint| The URL that can connect to the object storage service. For example: s3.us-west-2.amazonaws.com
Note: LOAD DATA supports only URLs for AWS object storage services and Alibaba Cloud external object storage services.| -|access_key_id| Access key ID used for authentication.| -|secret_access_key| Secret access key associated with the access key ID.| -|bucket| Specifies the bucket in S3 storage.| -|role_arn| The Amazon Resource Name (ARN) of an AWS role, typically used for cross-account access.
__Note:__ It is recommended to use role-based access. If you choose role-based access, you do not need to fill in `access_key_id` and `secret_access_key`. Only the `role_arn` parameter needs to be provided.| -|external_id| An external ID used in conjunction with the role ARN. | -|filepath| relative file path. regex expression is supported as /files/*.csv. | -|region| object storage service region| -|compression| Compressed format of S3 files. If empty or "none", it indicates uncompressed files. Supported fields or Compressed format are "auto", "none", "gzip", "bz2", and "lz4".| - -The other paramaters are identical to a ordinary LOAD DATA, see [LOAD DATA](../../../Reference/SQL-Reference/Data-Manipulation-Language/load-data.md) for more details. - -#### Statement Examples - -- **Role-Based Access** - -```sql -# LOAD a csv file from AWS S3 us-east-1 region, test-load-mo bucket, without compression -LOAD DATA URL s3option{"endpoint"='s3.us-east-1.amazonaws.com', "bucket"='test-load-mo', "role_arn"='xxxx', "filepath"='test.csv', "region"='us-east-1', "compression"='none'} INTO TABLE t1 FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n'; - -# LOAD all csv files from Alibaba Cloud OSS Shanghai region, test-load-data bucket, without compression -LOAD DATA URL s3option{"endpoint"='oss-cn-shanghai.aliyuncs.com', "bucket"='test-load-data', "role_arn"='xxxx', "filepath"='/test/*.csv', "region"='oss-cn-shanghai', "compression"='none'} INTO TABLE t1 FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n'; -``` - -- **Key-Based Access** - -```sql -# LOAD a csv file from AWS S3 us-east-1 region, test-load-mo bucket, without compression -LOAD DATA URL s3option{"endpoint"='s3.us-east-1.amazonaws.com', "access_key_id"='XXXXXX', "secret_access_key"='XXXXXX', "bucket"='test-load-mo', "filepath"='test.csv', "region"='us-east-1', "compression"='none'} INTO TABLE t1 FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n'; - -# LOAD all csv files from Alibaba Cloud OSS Shanghai region, test-load-data bucket, without compression -LOAD DATA URL s3option{"endpoint"='oss-cn-shanghai.aliyuncs.com', "access_key_id"='XXXXXX', "secret_access_key"='XXXXXX', "bucket"='test-load-data', "filepath"='/test/*.csv', "region"='oss-cn-shanghai', "compression"='none'} INTO TABLE t1 FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n'; - -# LOAD a csv file from Tencent Cloud COS Shanghai region, test-1252279971 bucket, without bz2 compression -LOAD DATA URL s3option{"endpoint"='cos.ap-shanghai.myqcloud.com', "access_key_id"='XXXXXX', "secret_access_key"='XXXXXX', "bucket"='test-1252279971', "filepath"='test.csv.bz2', "region"='ap-shanghai', "compression"='bz2'} INTO TABLE t1 FIELDS TERMINATED BY ',' ENCLOSED BY '\"' LINES TERMINATED BY '\n'; -``` - -!!! note - MatrixOne provides security assurance for S3 authentication information, such as `access_key_id` and `secret_access_key` sensitive information will be hidden in the system table (statement_info) records to ensure your account security. - -### Tutorial: Load a file from AWS S3 - -In this tutorial, we will walk you through the process of loading a **.csv** file from AWS S3; we assume that you already have an AWS account and already have your data file ready in your S3 service. If you do not already have that, please sign up and upload your data file first; you may check on the AWS S3 [official tutorial](https://docs.aws.amazon.com/AmazonS3/latest/userguide/GetStartedWithS3.html). The process for Alibaba Cloud OSS and Tencent Cloud COS is similar to AWS S3. - -!!! note - This code example does not show account information such as access_key_id and secret_access_key because of account privacy. - You can read this document to understand the main steps; specific data and account information will not be shown. - -1. Enter into **AWS S3 > buckets > Create bucket**, create a bucket **test-loading** with a public access and upload the file [*char_varchar_1.csv*](https://github.com/matrixorigin/matrixone/blob/main/test/distributed/resources/load_data/char_varchar_1.csv). - - - - ![public block](https://github.com/matrixorigin/artwork/blob/main/docs/develop/load_S3/create_bucket_public_block.png?raw=true) - -2. Get or create your AWS api key. Enter into **Your Account Name > Security Credentials**. - - - Method 1 - Role-Based Access (**Recommended**): To obtain your ARN, navigate to Security Credentials: - - ![Access Key](https://github.com/matrixorigin/artwork/blob/main/docs/develop/load_S3/arn.png?raw=true) - - - Method 2 - Key-Based Access (**Not Recommended**): To get your existing Access Key or create a new one. If you can't access your AWS Access key, you can contact your AWS administrator. - - - - Within **Security Credentials > Create access key**, you can obtain the Access key and Secret access key either from the downloaded credentials or from this web page. - - ![Access Key](https://github.com/matrixorigin/artwork/blob/main/docs/develop/load_S3/access_key.png?raw=true) - - ![Retrieve Access Key](https://github.com/matrixorigin/artwork/blob/main/docs/develop/load_S3/retrieve_access_key.png?raw=true) - -3. Launch the MySQL Client, create tables in MatrixOne, for example: - - ```sql - create database db; - use db; - drop table if exists t1; - create table t1(col1 char(225), col2 varchar(225), col3 text, col4 varchar(225)); - ``` - -4. Import the file into MatrixOne: - - - Method 1 - Role-Based Access (**Recommended**): - - ``` - LOAD DATA URL s3option{"endpoint"='s3.us-east-1.amazonaws.com', "bucket"='test-loading', "role_arn"='xxxx", "filepath"='char_varchar_1.csv', "region"='us-east-1', "compression"='none'} INTO TABLE t1; - ``` - - - Method 2 - Key-Based Access (**Not Recommended**) - - ``` - LOAD DATA URL s3option{"endpoint"='s3.us-east-1.amazonaws.com', "access_key_id"='XXXXXX', "secret_access_key"='XXXXXX', "bucket"='test-loading', "filepath"='char_varchar_1.csv', "region"='us-east-1', "compression"='none'} INTO TABLE t1; - ``` - -5. After the import is successful, you can run SQL statements to check the result of imported data: - - ```sql - mysql> select * from t1; - +-----------+-----------+-----------+-----------+ - | col1 | col2 | col3 | col4 | - +-----------+-----------+-----------+-----------+ - | a | b | c | d | - | a | b | c | d | - | 'a' | 'b' | 'c' | 'd' | - | 'a' | 'b' | 'c' | 'd' | - | aa,aa | bb,bb | cc,cc | dd,dd | - | aa, | bb, | cc, | dd, | - | aa,,,aa | bb,,,bb | cc,,,cc | dd,,,dd | - | aa',',,aa | bb',',,bb | cc',',,cc | dd',',,dd | - | aa"aa | bb"bb | cc"cc | dd"dd | - | aa"aa | bb"bb | cc"cc | dd"dd | - | aa"aa | bb"bb | cc"cc | dd"dd | - | aa""aa | bb""bb | cc""cc | dd""dd | - | aa""aa | bb""bb | cc""cc | dd""dd | - | aa",aa | bb",bb | cc",cc | dd",dd | - | aa"",aa | bb"",bb | cc"",cc | dd"",dd | - | | | | | - | | | | | - | NULL | NULL | NULL | NULL | - | | | | | - | " | " | " | " | - | "" | "" | "" | "" | - +-----------+-----------+-----------+-----------+ - 21 rows in set (0.03 sec) - ``` - -## Method 2: Specify S3 file to an external table - -### Syntax - -```sql -create external table t(...) URL s3option{"endpoint"='', "access_key_id"='', "secret_access_key"='', "bucket"='', "filepath"='', "region"='', "compression"=''} -[{FIELDS | COLUMNS} - [TERMINATED BY 'string'] - [[OPTIONALLY] ENCLOSED BY 'char'] -] -[IGNORE number {LINES | ROWS}]; -``` - -!!! note - MatrixOne only supports `select` on external tables. `Delete`, `insert`, and `update` are not supported. - -**Parameter Description** - -|Parameter|Description| -|:-:|:-:| -|endpoint|A endpoint is a URL that can conncect to object storage service. For example: s3.us-west-2.amazonaws.com| -|access_key_id| Access key ID | -|secret_access_key| Secret access key | -|bucket| S3 Bucket to access | -|filepath| relative file path. regex expression is supported as /files/*.csv. | -|region| object storage service region| -|compression| Compressed format of S3 files. If empty or "none", it indicates uncompressed files. Supported fields or Compressed format are "auto", "none", "gzip", "bz2", and "lz4".| - -The other paramaters are identical to a ordinary LOAD DATA, see [LOAD DATA](../../../Reference/SQL-Reference/Data-Manipulation-Language/load-data.md) for more details. - -For more information about External Table, see [CREATE EXTERNAL TABLE](../../../Reference/SQL-Reference/Data-Definition-Language/create-external-table.md). - -**Statement Examples**: - -```sql -## Create a external table for a .csv file from AWS S3 -create external table t1(col1 char(225)) url s3option{"endpoint"='s3.us-east-1.amazonaws.com', "access_key_id"='XXXXXX', "secret_access_key"='XXXXXX', "bucket"='test-loading', "filepath"='test.csv', "region"='us-east-1', "compression"='none'} fields terminated by ',' enclosed by '\"' lines terminated by '\n'; - -## Create a external table for a .csv file compressed with BZIP2 from Tencent Cloud -create external table t1(col1 char(225)) url s3option{"endpoint"='cos.ap-shanghai.myqcloud.com', "access_key_id"='XXXXXX', "secret_access_key"='XXXXXX', "bucket"='test-1252279971', "filepath"='test.csv.bz2', "region"='ap-shanghai', "compression"='bz2'} fields terminated by ',' enclosed by '\"' lines terminated by '\n' ignore 1 lines; -``` - -### Tutorial: Create an external table with S3 file - -This tutorial will walk you through the whole process of creating an external table with a **.csv** file from AWS S3. - -!!! note - This code example does not show account information such as access_key_id and secret_access_key because of account privacy. - You can read this document to understand the main steps; specific data and account information will not be shown. - -1. Download the [data file](https://github.com/matrixorigin/matrixone/blob/main/test/distributed/resources/load_data/char_varchar_1.csv). Enter into **AWS S3 > buckets**, create a bucket **test-loading** with a public access and upload the file *char_varchar_1.csv*. - - - - ![public block](https://github.com/matrixorigin/artwork/blob/main/docs/develop/load_S3/create_bucket_public_block.png?raw=true) - -2. Get or create your AWS api key. Enter into **Your Account Name > Security Credentials**, get your existing Access Key or create a new one. - - - - ![Access Key](https://github.com/matrixorigin/artwork/blob/main/docs/develop/load_S3/access_key.png?raw=true) - - You can get the access key id and secret access key from the downloaded credentials or this webpage. - - ![Retrieve Access Key](https://github.com/matrixorigin/artwork/blob/main/docs/develop/load_S3/retrieve_access_key.png?raw=true) - -3. Launch the MySQL Client, and specify the S3 file to an external table: - - ```sql - create database db; - use db; - drop table if exists t1; - create external table t1(col1 char(225), col2 varchar(225), col3 text, col4 varchar(225)) url s3option{"endpoint"='s3.us-east-1.amazonaws.com', "access_key_id"='XXXXXX', "secret_access_key"='XXXXXX', "bucket"='test-loading', "filepath"='char_varchar_1.csv', "region"='us-east-1', "compression"='none'} fields terminated by ',' enclosed by '\"' lines terminated by '\n'; - ``` - -4. After the import is successful, you can run SQL statements to check the result of the imported data. You can see that the query speed is significantly slower than querying from a local table. - - ```sql - select * from t1; - +-----------+-----------+-----------+-----------+ - | col1 | col2 | col3 | col4 | - +-----------+-----------+-----------+-----------+ - | a | b | c | d | - | a | b | c | d | - | 'a' | 'b' | 'c' | 'd' | - | 'a' | 'b' | 'c' | 'd' | - | aa,aa | bb,bb | cc,cc | dd,dd | - | aa, | bb, | cc, | dd, | - | aa,,,aa | bb,,,bb | cc,,,cc | dd,,,dd | - | aa',',,aa | bb',',,bb | cc',',,cc | dd',',,dd | - | aa"aa | bb"bb | cc"cc | dd"dd | - | aa"aa | bb"bb | cc"cc | dd"dd | - | aa"aa | bb"bb | cc"cc | dd"dd | - | aa""aa | bb""bb | cc""cc | dd""dd | - | aa""aa | bb""bb | cc""cc | dd""dd | - | aa",aa | bb",bb | cc",cc | dd",dd | - | aa"",aa | bb"",bb | cc"",cc | dd"",dd | - | | | | | - | | | | | - | NULL | NULL | NULL | NULL | - | | | | | - | " | " | " | " | - | "" | "" | "" | "" | - +-----------+-----------+-----------+-----------+ - 21 rows in set (1.32 sec) - ``` - -5. (Optional)If you need to import external table data into a data table in MatrixOne, you can use the following SQL statement: - - Create a new table t2 in MatrixOne: - - ```sql - create table t2(col1 char(225), col2 varchar(225), col3 text, col4 varchar(225)); - ``` - - Import the external table *t1* to *t2*: - - ```sql - insert into t2 select * from t1; - ``` diff --git a/docs/MatrixOne/Develop/import-data/bulk-load/load-csv.md b/docs/MatrixOne/Develop/import-data/bulk-load/load-csv.md index f9cce5b22..b2f6ef17b 100644 --- a/docs/MatrixOne/Develop/import-data/bulk-load/load-csv.md +++ b/docs/MatrixOne/Develop/import-data/bulk-load/load-csv.md @@ -7,37 +7,45 @@ This document will guide you on how to import large amounts of *.csv* format dat - Scenario 1: The data file is in the same machine with the MatrixOne server. ``` -LOAD DATA -INFILE 'file_name' -INTO TABLE tbl_name -[{FIELDS | COLUMNS} -[TERMINATED BY 'string'] -[[OPTIONALLY] ENCLOSED BY 'char'] -] -[LINES -[STARTING BY 'string'] -[TERMINATED BY 'string'] -] -[IGNORE number {LINES | ROWS}] -[PARALLEL {'TRUE' | 'FALSE'}] +> LOAD DATA + INFILE '|' + INTO TABLE tbl_name + [CHARACTER SET charset_name] + [{FIELDS | COLUMNS} + [TERMINATED BY 'string'] + [[OPTIONALLY] ENCLOSED BY 'char'] + [ENCASPED BY 'char'] + ] + [LINES + [STARTING BY 'string'] + [TERMINATED BY 'string'] + ] + [IGNORE number {LINES | ROWS}] + [SET column_name_1=nullif(column_name_1, expr1), column_name_2=nullif(column_name_2, expr2)...] + [PARALLEL {'TRUE' | 'FALSE'}] + [STRICT {'TRUE' | 'FALSE'}] ``` - Scenario 2: The data file is in separate machines with the MatrixOne server. ``` -LOAD DATA LOCAL -INFILE 'file_name' -INTO TABLE tbl_name -[{FIELDS | COLUMNS} -[TERMINATED BY 'string'] -[[OPTIONALLY] ENCLOSED BY 'char'] -] -[LINES -[STARTING BY 'string'] -[TERMINATED BY 'string'] -] -[IGNORE number {LINES | ROWS}] -[PARALLEL {'TRUE' | 'FALSE'}] +> LOAD DATA LOCAL + INFILE '|' + INTO TABLE tbl_name + [CHARACTER SET charset_name] + [{FIELDS | COLUMNS} + [TERMINATED BY 'string'] + [[OPTIONALLY] ENCLOSED BY 'char'] + [ENCASPED BY 'char'] + ] + [LINES + [STARTING BY 'string'] + [TERMINATED BY 'string'] + ] + [IGNORE number {LINES | ROWS}] + [SET column_name_1=nullif(column_name_1, expr1), column_name_2=nullif(column_name_2, expr2)...] + [PARALLEL {'TRUE' | 'FALSE'}] + [STRICT {'TRUE' | 'FALSE'}] ``` ## Before you start @@ -96,7 +104,7 @@ __Note__: A `csv`(comma-separated values) file is a delimited text file that use ### Example using `Load data` with `docker` version If you install MatrixOne by `docker`, the file system is inside the docker image by default. To work with local directory, you need to bind a local directory to the container. In the following example, the local file system path `~/tmp/docker_loaddata_demo/` is binded to the MatrixOne docker image, with a mapping to the `/ssb-dbgen-path` path inside the docker. -We will walk you through the whole process of loading data with MatrixOne 1.2.4 docker version in this example. +We will walk you through the whole process of loading data with MatrixOne 2.0.0 docker version in this example. 1. Download the dataset file and store the data in *~/tmp/docker_loaddata_demo/*: @@ -114,7 +122,7 @@ We will walk you through the whole process of loading data with MatrixOne 1.2.4 3. Use Docker to launch MatrixOne, and mount the directory *~/tmp/docker_loaddata_demo/* that stores data files to a directory in the container. The container directory is */sb-dbgen-path* as an example: ``` - sudo docker run --name matrixone --privileged -d -p 6001:6001 -v ~/tmp/docker_loaddata_demo/:/ssb-dbgen-path:rw matrixorigin/matrixone:1.2.4 + sudo docker run --name matrixone --privileged -d -p 6001:6001 -v ~/tmp/docker_loaddata_demo/:/ssb-dbgen-path:rw matrixorigin/matrixone:2.0.0 ``` 4. Connect to MatrixOne server: diff --git a/docs/MatrixOne/Develop/import-data/bulk-load/using-source.md b/docs/MatrixOne/Develop/import-data/bulk-load/using-source.md index 63d3e3232..5c6ec174b 100644 --- a/docs/MatrixOne/Develop/import-data/bulk-load/using-source.md +++ b/docs/MatrixOne/Develop/import-data/bulk-load/using-source.md @@ -66,6 +66,6 @@ select * from tool; ## Constraints -MatrixOne v24.1.2.4 version already supports MySQL table creation statements, so you can smoothly migrate MySQL tables to MatrixOne. However, it should be noted that during the migration process, some keywords incompatible with MySQL, such as `engine=`, will be automatically ignored in MatrixOne and will not affect the migration of the table structure. +MatrixOne v24.2.0.0 version already supports MySQL table creation statements, so you can smoothly migrate MySQL tables to MatrixOne. However, it should be noted that during the migration process, some keywords incompatible with MySQL, such as `engine=`, will be automatically ignored in MatrixOne and will not affect the migration of the table structure. However, it should be noted that although MatrixOne supports MySQL table creation statements, manual modification is still required if the migrated table contains incompatible data types, triggers, functions, or stored procedures. For more detailed compatibility information, see [MySQL Compatibility](../../../Overview/feature/mysql-compatibility.md). diff --git a/docs/MatrixOne/FAQs/product-faqs.md b/docs/MatrixOne/FAQs/product-faqs.md index 0432d67a9..db712ac3c 100644 --- a/docs/MatrixOne/FAQs/product-faqs.md +++ b/docs/MatrixOne/FAQs/product-faqs.md @@ -29,7 +29,7 @@ Both MatrixOne 0.8.0 and above can be upgraded directly from a lower version to Is MatrixOne stable **now? Which version is recommended?** -MatrixOne is now available in version v24.1.2.4. We've done a lot of optimization work on stability, and it's ready to be used in the production business. +MatrixOne is now available in version v24.2.0.0. We've done a lot of optimization work on stability, and it's ready to be used in the production business. **Is there a cloud version of MatrixOne? Want a quick test to see**. diff --git a/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method1.md b/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method1.md index 0f2f9298e..98d4b7505 100644 --- a/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method1.md +++ b/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method1.md @@ -184,7 +184,7 @@ GCC_VERSION="8.5.0" CLANG_VERSION="13.0" GO_VERSION="1.22" MO_GIT_URL="https://github.com/matrixorigin/matrixone.git" -MO_DEFAULT_VERSION="v1.2.4" +MO_DEFAULT_VERSION="v2.0.0" GOPROXY="https://goproxy.cn,direct" STOP_INTERVAL="5" START_INTERVAL="2" @@ -200,7 +200,7 @@ Generally, the parameters that need to be adjusted are as follows: ```` mo_ctl set_conf MO_PATH="yourpath" # Set custom MatrixOne download path mo_ctl set_conf MO_GIT_URL="https://githubfast.com/matrixorigin/matrixone.git" # For the problem of slow downloading from the original GitHub address, set image download address -mo_ctl set_conf MO_DEFAULT_VERSION="v1.2.4" # Set the version of MatrixOne downloaded +mo_ctl set_conf MO_DEFAULT_VERSION="v2.0.0" # Set the version of MatrixOne downloaded mo_ctl set_conf MO_DEPLOY_MODE=git # Deployment Configuration ```` @@ -219,7 +219,7 @@ Depending on your needs, choose whether you want to keep your code up to date, o === "Get the MatrixOne(Stable Version) code to build" ``` - mo_ctl deploy v1.2.4 + mo_ctl deploy v2.0.0 ``` ## Step 4: Launch MatrixOne server @@ -256,7 +256,7 @@ root@VM-16-2-debian:~# mo_ctl connect mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 15 -Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne +Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Copyright (c) 2000, 2023, Oracle and/or its affiliates. diff --git a/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method2.md b/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method2.md index a7794fee5..2806ee856 100644 --- a/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method2.md +++ b/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method2.md @@ -96,16 +96,16 @@ The Debian11.1 version does not have MySQL Client installed by default, so it ne ```bash mkdir -p /root/matrixone & cd /root/ - wget https://github.com/matrixorigin/matrixone/releases/download/v1.2.4/mo-v1.2.4-linux-x86_64.zip - unzip -d matrixone/ mo-v1.2.4-linux-x86_64.zip + wget https://github.com/matrixorigin/matrixone/releases/download/v2.0.0/mo-v2.0.0-linux-x86_64.zip + unzip -d matrixone/ mo-v2.0.0-linux-x86_64.zip ``` Binary for ARM architecture system: ```bash mkdir -p /root/matrixone & cd /root/ - wget https://github.com/matrixorigin/matrixone/releases/download/v1.2.4/mo-v1.2.4-linux-arm64.zip - unzip -d matrixone/ mo-v1.2.4-linux-arm64.zip + wget https://github.com/matrixorigin/matrixone/releases/download/v2.0.0/mo-v2.0.0-linux-arm64.zip + unzip -d matrixone/ mo-v2.0.0-linux-arm64.zip ``` === "**Downloading method 2: Using `curl` to install binary packages**" @@ -114,21 +114,21 @@ The Debian11.1 version does not have MySQL Client installed by default, so it ne ```bash mkdir -p /root/matrixone & cd /root/ - curl -OL https://github.com/matrixorigin/matrixone/releases/download/v1.2.4/mo-v1.2.4-linux-x86_64.zip - unzip -d matrixone/ mo-v1.2.4-linux-x86_64.zip + curl -OL https://github.com/matrixorigin/matrixone/releases/download/v2.0.0/mo-v2.0.0-linux-x86_64.zip + unzip -d matrixone/ mo-v2.0.0-linux-x86_64.zip ``` Binary for ARM architecture system: ```bash mkdir -p /root/matrixone & cd /root/ - curl -OL https://github.com/matrixorigin/matrixone/releases/download/v1.2.4/mo-v1.2.4-linux-arm64.zip - unzip -d matrixone/ mo-v1.2.4-linux-arm64.zip + curl -OL https://github.com/matrixorigin/matrixone/releases/download/v2.0.0/mo-v2.0.0-linux-arm64.zip + unzip -d matrixone/ mo-v2.0.0-linux-arm64.zip ``` === "**Downloading method 3: Go to the page and download**" - If you want a more intuitive way to download the page, go to the [version 1.2.4](https://github.com/matrixorigin/matrixone/releases/tag/v1.2.4), pull down to find the **Assets** column, and click the installation package *mo-v1.2.4-linux-x86_64.zip* or *mo-v1.2.4-linux-arm64.zip* can be downloaded. + If you want a more intuitive way to download the page, go to the [version 2.0.0](https://github.com/matrixorigin/matrixone/releases/tag/v2.0.0), pull down to find the **Assets** column, and click the installation package *mo-v2.0.0-linux-x86_64.zip* or *mo-v2.0.0-linux-arm64.zip* can be downloaded. ## Step 3: Install the mo_ctl tool @@ -147,8 +147,8 @@ wget https://raw.githubusercontent.com/matrixorigin/mo_ctl_standalone/main/insta The parameters that need to be adjusted are as follows: ```` -mo_ctl set_conf MO_PATH="/yourpath/mo-v1.2.4-xx-xx" # Set the MO_PATH to the directory where the binary files are extracted -mo_ctl set_conf MO_CONF_FILE="/yourpath/mo-v1.2.4-xx-xx/etc/launch/launch.toml" # Set the MO_CONF_FILE path +mo_ctl set_conf MO_PATH="/yourpath/mo-v2.0.0-xx-xx" # Set the MO_PATH to the directory where the binary files are extracted +mo_ctl set_conf MO_CONF_FILE="/yourpath/mo-v2.0.0-xx-xx/etc/launch/launch.toml" # Set the MO_CONF_FILE path mo_ctl set_conf MO_DEPLOY_MODE=binary #Deployment Configuration ```` @@ -156,17 +156,17 @@ mo_ctl set_conf MO_DEPLOY_MODE=binary #Deployment Configuration Launch the MatrixOne service through the `mo_ctl start` command. -If the operation is regular, the following log will appear. The relevant operation logs of MatrixOne will be in `/yourpath/mo-v1.2.4-xx-xx/matrixone/logs/`. +If the operation is regular, the following log will appear. The relevant operation logs of MatrixOne will be in `/yourpath/mo-v2.0.0-xx-xx/matrixone/logs/`. ``` root@VM-16-2-debian:~# mo_ctl start 2024-03-07 14:34:04.942 UTC+0800 [INFO] No mo-service is running 2024-03-07 14:34:04.998 UTC+0800 [INFO] Get conf succeeded: MO_DEPLOY_MODE="binary" 2024-03-07 14:34:05.024 UTC+0800 [INFO] GO memory limit(Mi): 14745 -2024-03-07 14:34:05.072 UTC+0800 [INFO] Starting mo-service: cd /Users/admin/mo-v1.2.4-linux-arm64/ && GOMEMLIMIT=14745MiB /Users/admin/mo-v1.2.4-linux-arm64/mo-service -daemon -debug-http :9876 -launch /Users/admin/mo-v1.2.4-linux-arm64/etc/launch/launch.toml >/Users/admin/mo-v1.2.4-linux-arm64/matrixone/logs/stdout-20240307_143405.log 2>/Users/admin/mo-v1.2.4-linux-arm64/matrixone/logs/stderr-20240307_143405.log +2024-03-07 14:34:05.072 UTC+0800 [INFO] Starting mo-service: cd /Users/admin/mo-v2.0.0-linux-arm64/ && GOMEMLIMIT=14745MiB /Users/admin/mo-v2.0.0-linux-arm64/mo-service -daemon -debug-http :9876 -launch /Users/admin/mo-v2.0.0-linux-arm64/etc/launch/launch.toml >/Users/admin/mo-v2.0.0-linux-arm64/matrixone/logs/stdout-20240307_143405.log 2>/Users/admin/mo-v2.0.0-linux-arm64/matrixone/logs/stderr-20240307_143405.log 2024-03-07 14:34:05.137 UTC+0800 [INFO] Wait for 2 seconds 2024-03-07 14:34:07.261 UTC+0800 [INFO] At least one mo-service is running. Process info: - 501 27145 1 0 2:34下午 ?? 0:00.18 /Users/admin/mo-v1.2.4-linux-arm64/mo-service -daemon -debug-http :9876 -launch /Users/admin/mo-v1.2.4-linux-arm64/etc/launch/launch.toml + 501 27145 1 0 2:34下午 ?? 0:00.18 /Users/admin/mo-v2.0.0-linux-arm64/mo-service -daemon -debug-http :9876 -launch /Users/admin/mo-v2.0.0-linux-arm64/etc/launch/launch.toml 2024-03-07 14:34:07.284 UTC+0800 [INFO] List of pid(s): 27145 2024-03-07 14:34:07.308 UTC+0800 [INFO] Start succeeded @@ -188,7 +188,7 @@ root@VM-16-2-debian:~# mo_ctl connect 2024-03-07 14:34:59.942 UTC+0800 [INFO] Ok, connecting for user ... Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 426 -Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne +Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. diff --git a/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method3.md b/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method3.md index 3a168ae96..0041d8f84 100644 --- a/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method3.md +++ b/docs/MatrixOne/Get-Started/install-on-linux/install-on-linux-method3.md @@ -67,18 +67,18 @@ Create and run the container of MatrixOne It will pull the image from Docker Hub if not exists. You can choose to pull the stable version image or the develop version image. -=== "Stable Version Image(1.2.4 version)" +=== "Stable Version Image(2.0.0 version)" ```bash - docker pull matrixorigin/matrixone:1.2.4 - docker run -d -p 6001:6001 --name matrixone matrixorigin/matrixone:1.2.4 + docker pull matrixorigin/matrixone:2.0.0 + docker run -d -p 6001:6001 --name matrixone matrixorigin/matrixone:2.0.0 ``` If you are using the network in mainland China, you can pull the MatrixOne stable version image on Alibaba Cloud: ```bash - docker pull registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:1.2.4 - docker run -d -p 6001:6001 --name matrixone registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:1.2.4 + docker pull registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:2.0.0 + docker run -d -p 6001:6001 --name matrixone registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:2.0.0 ``` === "Develop Version Image" @@ -102,7 +102,7 @@ It will pull the image from Docker Hub if not exists. You can choose to pull the If your Docker version is lower than 20.10.18 or the Docker client and server versions are inconsistent, upgrading to the latest stable version before attempting is recommended. If you choose to proceed with the current versions, you need to add the parameter `--privileged=true` to the `docker run` command, as shown below: ```bash -docker run -d -p 6001:6001 --name matrixone --privileged=true matrixorigin/matrixone:1.2.4 +docker run -d -p 6001:6001 --name matrixone --privileged=true matrixorigin/matrixone:2.0.0 ``` !!! note @@ -160,7 +160,7 @@ Depending on your needs, choose whether you want to keep your code up to date, o === "Get the MatrixOne(Stable Version) code to build" ``` - mo_ctl deploy v1.2.4 + mo_ctl deploy v2.0.0 ``` - Launch MatrixOne server @@ -198,7 +198,7 @@ root@VM-16-2-debian:~# mo_ctl connect mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 15 -Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne +Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Copyright (c) 2000, 2023, Oracle and/or its affiliates. diff --git a/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method1.md b/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method1.md index fd49c1ff2..7df8c6c34 100644 --- a/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method1.md +++ b/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method1.md @@ -170,7 +170,7 @@ GCC_VERSION="8.5.0" CLANG_VERSION="13.0" GO_VERSION="1.22" MO_GIT_URL="https://github.com/matrixorigin/matrixone.git" -MO_DEFAULT_VERSION="v1.2.4" +MO_DEFAULT_VERSION="v2.0.0" GOPROXY="https://goproxy.cn,direct" STOP_INTERVAL="5" START_INTERVAL="2" @@ -205,7 +205,7 @@ Depending on your needs, choose whether you want to keep your code up to date, o === "Get the MatrixOne(Stable Version) code to build" ``` - mo_ctl deploy v1.2.4 + mo_ctl deploy v2.0.0 ``` ## Step 4: Launch MatrixOne server @@ -242,7 +242,7 @@ This command will invoke the MySQL Client tool to connect to the MatrixOne servi mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 15 -Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne +Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Copyright (c) 2000, 2023, Oracle and/or its affiliates. diff --git a/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method2.md b/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method2.md index 3b5c8f3e2..e9caac15f 100644 --- a/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method2.md +++ b/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method2.md @@ -98,16 +98,16 @@ __Tips__: It is recommended that you download and install one of these two tools ```bash mkdir -p /User/username/mo/matrixone & cd /User/username/mo - wget https://github.com/matrixorigin/matrixone/releases/download/v1.2.4/mo-v1.2.4-darwin-x86_64.zip - unzip -d matrixone/ mo-v1.2.4-darwin-x86_64.zip + wget https://github.com/matrixorigin/matrixone/releases/download/v2.0.0/mo-v2.0.0-darwin-x86_64.zip + unzip -d matrixone/ mo-v2.0.0-darwin-x86_64.zip ``` Binary for ARM architecture system: ```bash mkdir -p /User/username/mo/matrixone & cd /User/username/mo - wget https://github.com/matrixorigin/matrixone/releases/download/v1.2.4/mo-v1.2.4-darwin-arm64.zip - unzip -d matrixone/ mo-v1.2.4-darwin-arm64.zip + wget https://github.com/matrixorigin/matrixone/releases/download/v2.0.0/mo-v2.0.0-darwin-arm64.zip + unzip -d matrixone/ mo-v2.0.0-darwin-arm64.zip ``` === "**Downloading method 2: Using `curl` to install binary packages**" @@ -116,21 +116,21 @@ __Tips__: It is recommended that you download and install one of these two tools ```bash mkdir -p /User/username/mo/matrixone & cd /User/username/mo - curl -OL https://github.com/matrixorigin/matrixone/releases/download/v1.2.4/mo-v1.2.4-darwin-x86_64.zip - unzip -d matrixone/ mo-v1.2.4-darwin-x86_64.zip + curl -OL https://github.com/matrixorigin/matrixone/releases/download/v2.0.0/mo-v2.0.0-darwin-x86_64.zip + unzip -d matrixone/ mo-v2.0.0-darwin-x86_64.zip ``` Binary for ARM architecture system: ```bash mkdir -p /User/username/mo/matrixone & cd /User/username/mo - curl -OL https://github.com/matrixorigin/matrixone/releases/download/v1.2.4/mo-v1.2.4-darwin-arm64.zip + curl -OL https://github.com/matrixorigin/matrixone/releases/download/v2.0.0/mo-v2.0.0-darwin-arm64.zip unzip -d matrixone/ mo-v-darwin-arm64.zip ``` === "**Downloading method 3: Go to the page and download**" - If you want a more intuitive way to download the page, go to the [version 1.2.4](https://github.com/matrixorigin/matrixone/releases/tag/v1.2.4), pull down to find the **Assets** column, and click the installation package *mo-v1.2.4-darwin-x86_64.zip* or *mo-v1.2.4-darwin-arm64.zip* can be downloaded. + If you want a more intuitive way to download the page, go to the [version 2.0.0](https://github.com/matrixorigin/matrixone/releases/tag/v2.0.0), pull down to find the **Assets** column, and click the installation package *mo-v2.0.0-darwin-x86_64.zip* or *mo-v2.0.0-darwin-arm64.zip* can be downloaded. ## Step 3: Install the mo_ctl tool @@ -149,8 +149,8 @@ wget https://raw.githubusercontent.com/matrixorigin/mo_ctl_standalone/main/insta The parameters that need to be adjusted are as follows: ```` -mo_ctl set_conf MO_PATH="/yourpath/mo-v1.2.4-xx-xx" # Set the MO_PATH to the directory where the binary files are extracted -mo_ctl set_conf MO_CONF_FILE="/yourpath/mo-v1.2.4-xx-xx/etc/launch/launch.toml" # Set the MO_CONF_FILE path +mo_ctl set_conf MO_PATH="/yourpath/mo-v2.0.0-xx-xx" # Set the MO_PATH to the directory where the binary files are extracted +mo_ctl set_conf MO_CONF_FILE="/yourpath/mo-v2.0.0-xx-xx/etc/launch/launch.toml" # Set the MO_CONF_FILE path mo_ctl set_conf MO_DEPLOY_MODE=binary #Deployment Configuration ```` @@ -158,17 +158,17 @@ mo_ctl set_conf MO_DEPLOY_MODE=binary #Deployment Configuration Launch the MatrixOne service through the `mo_ctl start` command. -If the operation is regular, the following log will appear. The relevant operation logs of MatrixOne will be in `/yourpath/mo-v1.2.4-xx-xx/matrixone/logs/ . +If the operation is regular, the following log will appear. The relevant operation logs of MatrixOne will be in `/yourpath/mo-v2.0.0-xx-xx/matrixone/logs/ . ``` > mo_ctl start 2024-03-07 14:34:04.942 UTC+0800 [INFO] No mo-service is running 2024-03-07 14:34:04.998 UTC+0800 [INFO] Get conf succeeded: MO_DEPLOY_MODE="binary" 2024-03-07 14:34:05.024 UTC+0800 [INFO] GO memory limit(Mi): 14745 -2024-03-07 14:34:05.072 UTC+0800 [INFO] Starting mo-service: cd /Users/admin/mo-v1.2.4-darwin-arm64/ && GOMEMLIMIT=14745MiB /Users/admin/mo-v1.2.4-darwin-arm64/mo-service -daemon -debug-http :9876 -launch /Users/admin/mo-v1.2.4-darwin-arm64/etc/launch/launch.toml >/Users/admin/mo-v1.2.4-darwin-arm64/matrixone/logs/stdout-20240307_143405.log 2>/Users/admin/mo-v1.2.4-darwin-arm64/matrixone/logs/stderr-20240307_143405.log +2024-03-07 14:34:05.072 UTC+0800 [INFO] Starting mo-service: cd /Users/admin/mo-v2.0.0-darwin-arm64/ && GOMEMLIMIT=14745MiB /Users/admin/mo-v2.0.0-darwin-arm64/mo-service -daemon -debug-http :9876 -launch /Users/admin/mo-v2.0.0-darwin-arm64/etc/launch/launch.toml >/Users/admin/mo-v2.0.0-darwin-arm64/matrixone/logs/stdout-20240307_143405.log 2>/Users/admin/mo-v2.0.0-darwin-arm64/matrixone/logs/stderr-20240307_143405.log 2024-03-07 14:34:05.137 UTC+0800 [INFO] Wait for 2 seconds 2024-03-07 14:34:07.261 UTC+0800 [INFO] At least one mo-service is running. Process info: - 501 27145 1 0 2:34下午 ?? 0:00.18 /Users/admin/mo-v1.2.4-darwin-arm64/mo-service -daemon -debug-http :9876 -launch /Users/admin/mo-v1.2.4-darwin-arm64/etc/launch/launch.toml + 501 27145 1 0 2:34下午 ?? 0:00.18 /Users/admin/mo-v2.0.0-darwin-arm64/mo-service -daemon -debug-http :9876 -launch /Users/admin/mo-v2.0.0-darwin-arm64/etc/launch/launch.toml 2024-03-07 14:34:07.284 UTC+0800 [INFO] List of pid(s): 27145 2024-03-07 14:34:07.308 UTC+0800 [INFO] Start succeeded @@ -190,7 +190,7 @@ This command will invoke the MySQL Client tool to connect to the MatrixOne servi 2024-03-07 14:34:59.942 UTC+0800 [INFO] Ok, connecting for user ... Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 426 -Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne +Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. diff --git a/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method3.md b/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method3.md index f88dd3453..bf8b3af0e 100644 --- a/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method3.md +++ b/docs/MatrixOne/Get-Started/install-on-macos/install-on-macos-method3.md @@ -59,18 +59,18 @@ Create and run the container of MatrixOne It will pull the image from Docker Hub if not exists. You can choose to pull the stable version image or the develop version image. -=== "Stable Version Image(1.2.4 version)" +=== "Stable Version Image(2.0.0 version)" ```bash - docker pull matrixorigin/matrixone:1.2.4 - docker run -d -p 6001:6001 --name matrixone matrixorigin/matrixone:1.2.4 + docker pull matrixorigin/matrixone:2.0.0 + docker run -d -p 6001:6001 --name matrixone matrixorigin/matrixone:2.0.0 ``` If you are using the network in mainland China, you can pull the MatrixOne stable version image on Alibaba Cloud: ```bash - docker pull registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:1.2.4 - docker run -d -p 6001:6001 --name matrixone registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:1.2.4 + docker pull registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:2.0.0 + docker run -d -p 6001:6001 --name matrixone registry.cn-shanghai.aliyuncs.com/matrixorigin/matrixone:2.0.0 ``` === "Develop Version Image" @@ -94,7 +94,7 @@ It will pull the image from Docker Hub if not exists. You can choose to pull the If your Docker version is lower than 20.10.18 or the Docker client and server versions are inconsistent, upgrading to the latest stable version before attempting is recommended. If you choose to proceed with the current versions, you need to add the parameter `--privileged=true` to the `docker run` command, as shown below: ```bash -docker run -d -p 6001:6001 --name matrixone --privileged=true matrixorigin/matrixone:1.2.4 +docker run -d -p 6001:6001 --name matrixone --privileged=true matrixorigin/matrixone:2.0.0 ``` !!! note @@ -120,7 +120,7 @@ The parameters that need to be adjusted are as follows: ``` mo_ctl set_conf MO_CONTAINER_DATA_HOST_PATH="/yourpath/mo/" # Set the data directory for host -mo_ctl set_conf MO_CONTAINER_IMAGE="matrixorigin/matrixone:1.2.4" # Set image +mo_ctl set_conf MO_CONTAINER_IMAGE="matrixorigin/matrixone:2.0.0" # Set image mo_ctl set_conf MO_DEPLOY_MODE=docker # Deployment Configuration ``` @@ -152,7 +152,7 @@ Depending on your needs, choose whether you want to keep your code up to date, o === "Get the MatrixOne(Stable Version) code to build" ``` - mo_ctl deploy v1.2.4 + mo_ctl deploy v2.0.0 ``` - Launch MatrixOne server @@ -190,7 +190,7 @@ root@VM-16-2-debian:~# mo_ctl connect mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 15 -Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne +Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Copyright (c) 2000, 2023, Oracle and/or its affiliates. diff --git a/docs/MatrixOne/Maintain/backup-restore/active-standby.md b/docs/MatrixOne/Maintain/backup-restore/active-standby.md new file mode 100644 index 000000000..30f00747f --- /dev/null +++ b/docs/MatrixOne/Maintain/backup-restore/active-standby.md @@ -0,0 +1,183 @@ +# MatrixOne active and backup disaster recovery function + +MatrixOne supports the cold backup function of the active and standby clusters based on log replication. It ensures the consistency and high availability of the active and standby data by synchronizing the transaction logs of the primary database to the standby database in real time. When the primary database fails, the standby database can quickly take over the business to ensure uninterrupted operations; after the fault is restored, the system can synchronize the data in the standby database back to the primary database to achieve seamless switchback. This solution significantly reduces downtime, improves data reliability and service continuity, and is suitable for scenarios such as finance and e-commerce that have high requirements for high availability. + +## Operation steps + +### Main cluster configuration + +#### Modify configuration file + +Go to the /your_matrixone_path/etc/launch directory and add the standby configuration of fileservice. Both the normal copy and the non-voting copy need to be configured. + +- Added log1.toml, the log node where the normal copy is located + +```shell +service-type = "LOG" +data-dir = "./mo-data" + +[log] +level = "info" + +[malloc] +check-fraction = 65536 +enable-metrics=true +[[fileservice]] +name = "STANDBY" +backend = "DISK" +data-dir = "mo-data/standby" + +#For a normal logservice copy, add a configuration item to the bootstrap configuration group to start synchronization of the standby cluster. +[logservice.BootstrapConfig] +standby-enabled = true +``` + +- Added log2.toml, the log node where the non-voting copy is located + +```shell +#You need to start at least one more logservice instance as a node running a non-voting replica, and configure locality for this instance: (127.0.0.1 in the configuration needs to be replaced with the actual IP address) +service-type = "LOG" +data-dir = "./mo-data" + +[log] +level = "info" + +[malloc] +check-fraction = 65536 +enable-metrics=true + +[[fileservice]] +name = "STANDBY" +backend = "DISK" +data-dir = "mo-data/standby" + +[logservice] +deployment-id=1 +uuid = "4c4dccb4-4d3c-41f8-b482-5251dc7a41bd" #UUID of the new node +raft-address = "127.0.0.1:32010" # The address of raft service +logservice-address = "127.0.0.1:32011" # The address of the logservice service +logservice-listen-address = "0.0.0.0:32011" +gossip-address = "127.0.0.1:32012" # The address of the gossip service +gossip-seed-addresses = [ # The gossip seed addresses of normal replicas +"127.0.0.1:32002", +"127.0.0.1:32012" +] +locality = "region:west" # Configure locality to run non-voting replicas +[logservice.BootstrapConfig] +bootstrap-cluster = false # Turn off bootstrap operation +standby-enabled = true # Start standy function + +[hakeeper-client] +discovery-address="127.0.0.1:32001" # 32001 is the default HAKeeper address +``` + +- Modify launch.toml + +```shell +logservices = [ + "./etc/launch/log1.toml", + "./etc/launch/log2.toml" + ] + + tnservices = [ + "./etc/launch/tn.toml" + ] + + cnservices = [ + "./etc/launch/cn.toml" + ] +``` + +#### Start non-voting replica + +After the MO cluster is started, execute the SQL command to dynamically increase or decrease the non-voting replicas: + +- Set non-voting locality, which needs to be the same as in the configuration file: + + ```bash + mysql> set logservice settings non_voting_locality="region:west"; + Query OK, 0 rows affected (0.01 sec) + ``` + +- Set the non-voting replica num. The following command sets the number of non-voting replicas to 1: + + ``` + mysql> set logservice settings non_voting_replica_num=1; + Query OK, 0 rows affected (0.02 sec) + ``` + +After executing the command, wait a moment and query the replica status through two SQL commands: + +- show logservice stores; + + ```sql + mysql> show logservice stores; + +-----------------------------------------+------+-------------+---------------------------+-------------+-----------------+-----------------+-----------------+ + | store_id | tick | replica_num | replicas | locality | raft_address | service_address | gossip_address | + +----------------------------------------+------+-------------+---------------------------+-------------+-----------------+-----------------+-----------------+ + | 4c4dccb4-4d3c-41f8-b482-5251dc7a41bd | 975 | 3 | 0:262148;1:262149;3:262150 | region:west | 127.0.0.1:32010 | 127.0.0.1:32011 | 127.0.0.1: 32012 | + | 7c4dccb4-4d3c-41f8-b482-5251dc7a41bf | 974 | 3 | 0:131072;1:262145;3:262146 | + +-----------------------------------------+------+-------------+---------------------------+-------------+-----------------+-----------------+-----------------+ + 2 rows in set (0.02 sec) + ``` + +- show logservice replicas; + + ```sql + mysql> show logservice replicas; + +----------+------------+--------------+-----------------------+------+-------+---------------------------------------+ + | shard_id | replica_id | replica_role | replica_is_non_voting | term | epoch | store_info | + +----------+------------+---------------+-----------------------+------+-------+---------------------------------------+ + | 0 | 131072 | Leader | false | 2 | 1059 | 7c4dccb4-4d3c-41f8-b482-5251dc7a41bf | + | 0 | 262148 | Follower | true | 2 | 1059 | 4c4dccb4-4d3c-41f8-b482-5251dc7a41bd | + | 1 | 262145 | Leader | false | 2 | 120 | 7c4dccb4-4d3c-41f8-b482-5251dc7a41bf | + | 1 | 262149 | Follower | true | 2 | 120 | 4c4dccb4-4d3c-41f8-b482-5251dc7a41bd | + | 3 | 262146 | Leader | false | 2 | 12 | 7c4dccb4-4d3c-41f8-b482-5251dc7a41bf | + | 3 | 262150 | Follower | true | 2 | 12 | 4c4dccb4-4d3c-41f8-b482-5251dc7a41bd | + +----------+------------+--------------+-----------------------+------+-------+---------------------------------------+ + 6 rows in set (0.01 sec) + ``` + +### Copy data files to the standby cluster + +1. Stop the main cluster, including the non-voting logservice service of the main cluster. +2. Copy the standby files to the mo-data/shared directory of the standby machine. + + ``` shell + cp -r /your_matrixone_pathe/mo-data/standby/*@:/your_matrixone_path/mo-data/shared/ + ``` + +3. Copy the data in the non-voting copy of the primary cluster to the logservice-data/directory of the standby machine. + + ``` shell + cp -r /your_matrixone_pathe/mo-data/logservice-data//@:/your_matrixone_pathe/mo-data/logservice-data/ + ``` + + Where is the uuid configured in the configuration file of the non-voting logservice instance. + +### Start the standby cluster + +#### Data synchronization + +The standby cluster uses the logtail data synchronization tool `mo_ctl` to synchronize the data in the non-voting copy of the primary cluster to the logservice of the standby cluster. The command is as follows: + +```shell +mo_ctl data-sync start --logservice-address=127.0.0.1:32001 --log-data-dir=/your_matrixone_path/mo-data/logservice-data//
/00000000000000000001/tandb +``` + +Where is the uuid configured in the configuration file of the non-voting logservice instance. + +!!! note + mo_ctl is a tool for MatrixOne distributed enterprise-level management. You need to contact your account manager to obtain it. + +#### Stop the logservice service + +```shell +kill `ps -ef | grep mo-service | grep -v grep | awk '{print $2}'` +``` + +#### Start TN, CN and Log services + +``` +nohup ./mo-service -launch etc/launch/launch.to ml >mo.log 2>&1 & +``` \ No newline at end of file diff --git a/docs/MatrixOne/Maintain/backup-restore/backup-restore-overview.md b/docs/MatrixOne/Maintain/backup-restore/backup-restore-overview.md index a04943f99..5d0828db0 100644 --- a/docs/MatrixOne/Maintain/backup-restore/backup-restore-overview.md +++ b/docs/MatrixOne/Maintain/backup-restore/backup-restore-overview.md @@ -1,97 +1,101 @@ # MatrixOne Backup and Recovery Overview -Database backup and recovery is one of the core operations of any database management system and an important guarantee of data security and availability. MatrixOne also provides flexible and robust database backup and recovery capabilities to ensure the integrity and continuity of user data. This article describes the importance of MatrixOne database backup and recovery, backup strategies, and backup methods. +Database backup and recovery is one of the core operations of any database management system and an important guarantee for data security and availability. MatrixOne also provides flexible and powerful database backup and recovery functions to ensure the integrity and continuity of user data. This article will introduce the importance, backup strategies, and backup methods of MatrixOne database backup and recovery. -## Backup and recovery strategies +## Backup and recovery strategy -Under different disaster recovery scenarios, a database backup can be used to restore health. +Database backups can be used to restore operational status in various disaster recovery scenarios. -Here are backup recovery methods for different scenarios: +The following are backup and recovery methods for different situations: -1. **Operating system crash**: - - - If there is a physical backup: After a crash, restore the entire database state through a physical backup, restore the backup to a normal hardware environment, and apply redo logs to ensure data consistency. - - If there is a logical backup: Rebuild the database schema and data on the new server through a logical backup. Install the database software and execute the SQL statements in the logical backup to create tables, indexes, etc. before importing the data. +1. **Operating system crashes**: + - If there is a physical backup: After a crash, restore the entire database state through physical backup, restore the backup to a normal hardware environment, and apply redo logs to ensure data consistency. + - If there is a logical backup: Rebuild the database schema and data on the new server through logical backup. First install the database software, execute the SQL statements in the logical backup to create tables, indexes, etc., and then import the data. + 2. **Power supply (detection) failed**: - - If there is a physical backup: After failure, the database can be restored through a physical backup, restoring the backup to a normal hardware environment, and applying redo logs to ensure data consistency. - - If there is a logical backup: Again, rebuild the database on the new server through a logical backup. + - If there is a physical backup: After failure, the database can be restored through the physical backup, the backup can be restored to the normal hardware environment, and redo logs can be used to ensure data consistency. + - If there is a logical backup: Again, rebuild the database on the new server via the logical backup. 3. **File system crash**: - - If there is a physical backup: Restore the database using a physical backup, restore the backup to a normal hardware environment, and apply redo logs to ensure data consistency. + - If there is a physical backup: Use the physical backup to restore the database, restore the backup to a normal hardware environment, and apply redo logs to ensure data consistency. - If there is a logical backup: After a crash, rebuild the database schema and data on the new server. -4. **Hardware issues (hard drives, motherboards, etc.):** +4. **Hardware problems (hard disk, motherboard, etc.)**: - - If there is a physical backup: restore the database through a physical backup, restore the backup to a new hardware environment, and apply redo logs to ensure data consistency. - - If there is a logical backup: Use the logical backup to rebuild the database in the new hardware environment. + - If there is a physical backup: restore the database through the physical backup, restore the backup to the new hardware environment, and apply redo logs to ensure data consistency. + - If there is a logical backup: Use the logical backup to rebuild the database on the new hardware environment. -The following strategies can be followed for backup recovery: +The following strategies can be followed for backup and recovery: -1. **Frequency of backups**: Determine the frequency of backups, usually divided into full and incremental backups. Full backups take up more storage and time, but restore faster, while incremental backups are more economical. +1. **Backup Frequency**: Determine the frequency of backup, usually divided into full backup and incremental backup. Full backup will take up more storage space and time, but the recovery speed is faster, while incremental backup is more economical. -2. **Backup storage**: Choose a secure backup storage location to ensure backup data is not vulnerable to corruption or loss. Offline storage media or cloud storage is often used to house backups. +2. **Backup Storage**: Choose a safe backup storage location to ensure that the backup data is not easily damaged or lost. Offline storage media or cloud storage are usually used to store backups. -3. **Backup Retention Period**: Determine the retention period for backup data for retrieval and recovery of historical data when needed. Develop appropriate data retention policies based on regulations and business needs. +3. **Backup retention period**: Determine the retention period of backup data so that historical data can be retrieved and restored when needed. Develop appropriate data retention policies based on regulations and business needs. -Whatever the recovery scenario, the following principles should be followed: +Regardless of the recovery situation, the following principles should be followed: -1. Consider stopping database operation to prevent data changes. -2. Select the appropriate backup for recovery based on the backup type. -3. Restore the backup file. -4. Consider applying the appropriate redo logs to ensure data consistency. -5. Start the database and perform the necessary tests. +1. Consider stopping the database to prevent data changes. +2. Select the appropriate backup according to the backup type to restore. +3. Restore the backup files. +4. Consider applying corresponding redo logs to ensure data consistency. +5. Start the database and perform necessary tests. ## Database backup method -MatrixOne offers a variety of backup methods that take into account factors such as database requirements, performance, resources, and recovery time. +MatrixOne provides a variety of backup methods, and you can select the appropriate backup method based on factors such as database requirements, performance, resources, and recovery time. -The MatrixOne database provides multiple backup tools to meet different scenarios and needs: +MatrixOne database provides a variety of backup tools to meet different scenarios and needs: -1. **mo-dump**: Used to export data and patterns from a database. It generates recoverable SQL scripts for logical backups. +1. **mo-dump**: used to export data and schema in the database. It generates recoverable SQL scripts for logical backups. -2. **mo-backup**: for physical backup and recovery. `mo-backup` is a physical backup and recovery tool for MatrixOne enterprise-class services that helps you protect your MatrixOne databases and perform reliable recovery operations when needed. +2. **mo-backup**: used for physical backup and recovery. `mo-backup` is the physical backup and recovery tool for MatrixOne enterprise-class services, helping you protect its MatrixOne database and perform reliable recovery operations when needed. - !!! note - **mo-backup** Physical backup and recovery tool for enterprise level services, you need to contact your MatrixOne account manager for the tool download path. +!!! note + **mo-backup**is a physical backup and recovery tool for enterprise-level services. You need to contact your MatrixOne account manager to obtain the tool download path. -### Logical Backup and Recovery +### Logical backup and recovery -#### Backup with `SELECT INTO OUTFILE` +#### Backup using `SELECT INTO OUTFILE` -Use the `SELECT ... INTO OUTFILE` command to export the retrieved data to a file in format, created by the MatrixOne service and only on the MatrixOne server host. Export to client file system is not supported, export directory do not rename files to avoid overwriting new files. +Use the `SELECT ... INTO OUTFILE` command to export the retrieved data into a formatted file created by the MatrixOne service, only on the MatrixOne server host. Exporting to the client file system is not supported. Do not rename files in the export directory to avoid overwriting with new files. -For operational steps and examples, see [`SELECT INTO...OUTFILE`](../../Develop/export-data/select-into-outfile.md) +For operation steps and examples, see [`SELECT INTO...OUTFILE`](../../Develop/export-data/select-into-outfile.md) -#### Backup with `mo-dump` +#### Use `mo-dump` to backup -MatrixOne supports logical backups using the `mo-dump` tool to generate SQL statements that can be used to recreate database objects and data. +MatrixOne supports logical backup using the `mo-dump` tool, which generates SQL statements that can be used to recreate database objects and data. -For operational steps and examples, see the [`mo-dump tool Write`](../../Develop/export-data/modump.md) Out +For operation steps and examples, see [`mo-dump tool written`](../../Develop/export-data/modump.md) -#### Bulk Import Recovery Using the Command Line +#### Use command line to batch import recovery -MatrixOne supports inserting large numbers of rows into database tables using the `LOAD DATA` command. It also supports importing table structures and data into the entire database using the `SOURCE` command. +MatrixOne supports using the `LOAD DATA` command to insert a large number of rows into a database table, and also supports using the `SOURCE` command to import table structures and data into the entire database. -For more information, see [Batch Import](../../Develop/import-data/bulk-load/bulk-load-overview.md) +For more information, refer to [Bulk Import](../../Develop/import-data/bulk-load/bulk-load-overview.md) -### Physical Backup and Recovery +### Physical backup and recovery -#### Backup and restore with `mo_br` +#### Backup and restore using `mo_br` -MatrixOne supports regular physical and snapshot backups using the `mo_br` tool. +MatrixOne supports regular physical backups, snapshot backups, and PITR backups using the `mo_br` tool. -See the [`mo-br User Guide`](../backup-restore/mobr-backup-restore/mobr.md) for steps and examples +For operation steps and examples, see [`mo-br Usage Guide`](../backup-restore/mobr-backup-restore/mobr.md) -#### Using SQL Backup and Recovery +#### Backup and restore using SQL -MatrixOne supports snapshot backup and recovery using SQL. +MatrixOne supports snapshots, PITR backup and recovery using SQL. -Refer to the documentation for methods of snapshot backup and recovery using SQL: +For methods of using SQL for snapshots, PITR backup and recovery, please refer to the documentation: - [CREATE SNAPSHOT](../../Reference/SQL-Reference/Data-Definition-Language/create-snapshot.md) - [DROP SNAPSHOT](../../Reference/SQL-Reference/Data-Definition-Language/drop-snapshot.md) - [SHOW SNAPSHOTS](../../Reference/SQL-Reference/Data-Definition-Language/create-snapshot.md) -- [RESTORE ACCOUNT](../../Reference/SQL-Reference/Data-Definition-Language/restore-account.md) +- [RESTORE ... FROM SNAPSHOT](../../Reference/SQL-Reference/Data-Definition-Language/restore-snapshot.md) +- [CREATE PITR](../../Reference/SQL-Reference/Data-Definition-Language/create-pitr.md) +- [DROP PITR](../../Reference/SQL-Reference/Data-Definition-Language/drop-pitr.md) +- [SHOW PITR](../../Reference/SQL-Reference/Data-Definition-Language/create-pitr.md) +- [RESTORE ... FROM PITR](../../Reference/SQL-Reference/Data-Definition-Language/restore-pitr.md) diff --git a/docs/MatrixOne/Maintain/backup-restore/key-concepts.md b/docs/MatrixOne/Maintain/backup-restore/key-concepts.md index ce06c6fe6..767c2a13d 100644 --- a/docs/MatrixOne/Maintain/backup-restore/key-concepts.md +++ b/docs/MatrixOne/Maintain/backup-restore/key-concepts.md @@ -1,52 +1,51 @@ -# Backup and Recovery Related Concepts +# Concepts related to backup and recovery -## Physical, snapshot and logical backups +## Physical backup, snapshot backup and logical backup ### Physical backup -Physical backup is the process of copying database files directly to backup media such as tape, hard drives, etc. This method copies all physical blocks of the database to backup media, including data files, control files, and redo log files. Backed-up data is binary data that is actually stored on disk, and recovery operations are usually quick. +Physical backup is the process of copying database files directly to backup media (such as tape, hard disk, etc.). This method copies all physical data blocks of the database to the backup media, including data files, control files, redo log files, etc. The backed up data is binary data actually stored on disk, and recovery operations are usually fast. -### snapshot backup +### Snapshot backup -Database snapshot backup is a form of physical backup, but unlike traditional physical backup, it creates an instant copy of data by capturing a read-only static view of the database at a specific point in time. This backup method utilizes an incremental storage mechanism that records only blocks of data that have changed since the last snapshot, making efficient use of storage space. Snapshot backups support fast recovery because they provide a complete, consistent view of the database for data protection, report generation, analysis, and other scenarios that require data consistency. In addition, they typically rely on the snapshot capabilities of the underlying storage system to provide a secure copy of data access to the database without affecting its proper operation. +Database snapshot backup is a form of physical backup, but it is different from traditional physical backup in that it creates an instant copy of the data by capturing a read-only static view of the database at a specific point in time. This backup method uses the incremental storage mechanism to record only the data blocks that have changed since the last snapshot, thus using storage space efficiently. Snapshot backups enable fast recovery because they provide a complete consistent view of the database, making them suitable for data protection, report generation, analytics, and other scenarios that require data consistency. In addition, they usually rely on the snapshot function of the underlying storage system to provide the database with a secure copy of data access without affecting the normal operation of the database. ### Logical backup -A logical backup is a backup of logical objects (such as tables, indexes, stored procedures, etc.) in a database through SQL statements. This backup method exports definitions and data for logical objects to a backup file, but does not involve binary data for database files. While recovery is slower, backup data is often easier to read and modify. +Logical backup is to back up logical objects (such as tables, indexes, stored procedures, etc.) in the database through SQL statements. This backup method exports the definition and data of the logical objects to the backup file, but does not involve the binary data of the database file. Although recovery is slower, backup data is usually easier to read and modify. -### The difference +### the difference -Data physical backup, logical backup, and snapshot backup are three different data protection strategies: physical backup creates a full copy of the database by directly replicating the database's storage files for rapid recovery and large-scale data migration; logical backup exports the logical structure of the database, such as SQL statements, to store data and structure in text to facilitate data migration across platforms and versions; and snapshot backup is a read-only view of the database at a given point in time, using incremental storage technology to record changes for rapid recovery to a specific point in time state, often dependent on storage system support. +Data physical backup, logical backup and snapshot backup are three different data protection strategies: physical backup creates a complete copy of the database by directly copying the storage files of the database, which is suitable for quick recovery and large-scale data migration; logical backup exports the database Logical structures, such as SQL statements, store data and structures in text form to facilitate data migration across platforms and versions; snapshot backup is a read-only view of the database at a certain moment, using incremental storage technology to record changes, and is suitable for rapid Restoring to a state at a specific point in time usually relies on the support of the storage system. -## Full, incremental, and differential backups +## Full backup, incremental backup and differential backup ### Full backup -Full backup backs up the entire data set to the storage device, including all files and folders. Although often time consuming and requiring large storage space, data can be fully restored without additional backup files. +A full backup backs up the entire data set to the storage device, including all files and folders. Although usually time-consuming and requiring large storage space, data can be completely restored without the need for additional backup files. -### incremental backup +### Incremental backup -Incremental backups back up only recently changed or added files or blocks. Back up only changes since the last backup, usually with less backup time and storage. Usually performed between regular full backups to ensure backup data is kept up to date. All incremental and up-to-date full backups are used to restore data. +Incremental backup only backs up recently changed or newly added files or data blocks. Only the changes since the last backup are backed up, usually with less backup time and less storage space. Typically performed between regular full backups to ensure backup data remains up to date. All incremental backups and the latest full backup are required to restore data. ### Differential backup -Differential backups are backups of data that has changed since the last full backup, so backup data is larger and takes longer than incremental backups. When restoring data, simply restore the most recent full backup and then the most recent differential backup. The backup and recovery process is relatively simple due to shorter recovery times without relying on previously backed up data. +Differential backup backs up data that has changed since the last full backup, so the backup data volume is larger than incremental backup, and the backup time is longer. When restoring data, just restore the most recent full backup first and then the most recent differential backup. Since it does not rely on previous backup data, the recovery time is short and the backup and recovery process is relatively simple. -### Differences between full, incremental, and differential backups +### The difference between full backup, incremental backup and differential backup -- Full backup provides full data backup, but requires more time and storage. -- Incremental backups are suitable for environments with less data change, saving backup storage space and time, but with long recovery times and reliance on front backup data. -- Differential backup is suitable for environments with more data changes, larger backup data, shorter recovery time than incremental backup, and a relatively simple backup recovery process. +- Full backup provides complete data backup, but requires more time and storage space. +- Incremental backup is suitable for environments with less data changes, saving backup storage space and time, but recovery time is long and requires previous backup data. +- Differential backup is suitable for environments with a lot of data changes. The backup data is larger, the recovery time is shorter than incremental backup, and the backup and recovery process is relatively simple. -## Recovery +## recover -### Physical Recovery +### Physical recovery -Physical recovery is database recovery using physical backups. Typically used for critical failures such as hard disk failures, operating system failures, or file system failures. In physical recovery, use professional recovery tools or software to read the actual data blocks in a backup file or storage medium and try to repair the damaged blocks. +Physical recovery is the use of physical backup for database recovery. Typically used for serious failures such as hard drive failure, operating system failure, or file system failure. In physical recovery, professional recovery tools or software are used to read the actual data blocks in the backup file or storage media and try to repair the damaged blocks. +The advantage of physical recovery is to quickly restore the database because there is no need to execute SQL statements or other high-level operations and directly process data blocks. In addition, physical recovery can restore all database objects, including tables, indexes, stored procedures, etc. -Physical recovery has the advantage of quickly restoring the database by processing blocks directly without SQL statements or other high-level operations. In addition, physical recovery restores all database objects, including tables, indexes, stored procedures, etc. +### Complete recovery and incomplete recovery -### Full versus incomplete recovery - -- Full Recovery: Apply all redo logs from the backup set and restore the data to a committed state for all logs in the backup. -- Incomplete Recovery: Apply a backup set of partial redo logs or add-ons to restore the database to a point in time that the backup redo logs contain. +- Full recovery: Apply all redo logs in the backup set and restore the data to the state where all logs in the backup have been committed. +- Incomplete recovery: Apply partial redo logs or additional backups of the backup set to restore the database to a certain point in time included in the backup redo logs. \ No newline at end of file diff --git a/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-backup-restore.md b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-backup-restore.md new file mode 100644 index 000000000..5c9ff2821 --- /dev/null +++ b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-backup-restore.md @@ -0,0 +1,666 @@ +# mo_br backup and recovery + +For enterprises, a large amount of data is generated every day, so database backup is very important. In the event of a system crash, hardware failure, or user misoperation, you can restore data and restart the system without causing data loss. + +In addition, data backup also serves as a safeguard before upgrading the MatrixOne installation, and data backup can also be used to transfer the MatrixOne installation to another system. + +MatrixOne supports physical backup via the `mo_br` utility. `mo_br` is a command line utility for generating physical backups of MatrixOne databases. It generates SQL statements that can be used to recreate database objects and data. + +We will use a simple example to describe how to use the `mo_br` utility to complete the data backup and restore process. + +## Create a backup + +### Grammar structure + +``` +mo_br backup + --host + --port + --user + --password + --backup_dir s3|filesystem + //s3 oss minio + --endpoint + --access_key_id + --secret_access_key + --bucket + --filepath + --region + --compression + --role_arn + --is_minio + --parallelism + //filesystem + --path + --parallelism + --meta_path + //incremental backup required + --backup_type + --base_id +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| +|user| user| +|password | User's password| +|backup_dir | Backup destination path type. s3 or filesystem| +|endpoint| The URL to connect to the service that backs up to s3| +|access_key_id| Access key ID backed up to s3| +|secret_access_key| Secret access key backed up to s3| +|bucket| The bucket that s3 needs to access when backed up| +|filepath| Relative file path for backup to s3| +|region| Object storage service region backed up to s3| +|compression| Compression format of files backed up to s3. | +|role_arn| Resource name of the role backed up to s3. | +|is_minio| specifies whether the s3 backed up to is minio| +|path| Local file system backup path| +|parallelism|Parallelism| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | +|backup_type|Specifies the backup type as incremental backup, incremental. | +|base_id|The ID of the last backup, mainly used to determine the timestamp of the last backup. | + +### Example + +- Full backup to local file system + +``` +./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "yourpath" +``` + +- Full backup to minio + +``` +./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "s3" --endpoint "http://127.0.0.1:9000" --access_key_id "S0kwLuB4JofVEIAxWTqf" --secret_access_key "X24O7t3hccmqUZqvqvmLN8464E2Nbr0DWOu9Qs5A" --bucket "bucket1" --filepath "/backup1" --is_minio +``` + +- Incremental backup to local file system + +``` +./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "yourpath" --backup_type "incremental" --base_id " "xxx" +``` + +## View backup + +### Grammar structure + +``` +mo_br list + --ID + //To query backup data. If the backup is on s3 (oss minio), you need to specify + --access_key_id + --secret_access_key + --not_check_data + --meta_path +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +| ID | Backup ID| +|access_key_id| Access key ID of s3 backed up| +|secret_access_key| Secret access key of s3 backed up| +|not_check_data | Only check the information in meta. Do not view backup data. By default, without this parameter, the backed up files will be checked. Currently only the backed up files are checked for existence. | +|meta_path | Specifies the meta file location. If not specified, the default is the mo_br.meta file in the same directory. | + +### Example + +- View a list of all backups + +```bash +./mo_br list ++----------------------------------------+--------+-----------------------------------+----------------------------+--------------+---------------------------+ +| ID | SIZE | PATH | AT TIME | DURATION | COMPLETE TIME | ++----------------------------------------+--------+-----------------------------------+----------------------------+--------------+---------------------------+ +| 4d21b228-10dd-11ef-9497-26dd28356ef2 | 586 kB | BackupDir: filesystem Path: | 2024-05-13 12:00:12 +0800 | 1.700945333s | 2024-05-13 12:00:13 +0800 | +| | | /Users/admin/soft/backup | | | | +| 01108122-10f9-11ef-9359-26dd28356ef2 | 8.3 MB | BackupDir: filesystem Path: | 2024-05-13 15:18:28 +0800 | 3.394437375s | 2024-05-13 15:18:32 +0800 | +| | | /Users/admin/soft/backup | | | | ++----------------------------------------+--------+-----------------------------------+----------------------------+--------------+---------------------------+ +``` + +- View the backup list of the specified ID. When the ID is determined by list, the backed up file will be detected. + +```bash +./mo_br list 4d21b228-10dd-11ef-9497-26dd28356ef2 ++----------------------------------------+--------+-----------------------------------+----------------------------+--------------+---------------------------+ +| ID | SIZE | PATH | AT TIME | DURATION | COMPLETE TIME | ++----------------------------------------+--------+-----------------------------------+----------------------------+--------------+---------------------------+ +| 4d21b228-10dd-11ef-9497-26dd28356ef2 | 586 kB | BackupDir: filesystem Path: | 2024-05-13 12:00:12 +0800 | 1.700945333s | 2024-05-13 12:00:13 +0800 | +| | | /Users/admin/soft/backup | | | | ++----------------------------------------+--------+-----------------------------------+----------------------------+--------------+---------------------------+ + +Checking the backup data(currently,no checksum)... + +check: /backup_meta +check: /mo_meta +check: hakeeper/hk_data +check: tae/tae_list +check: tae/tae_sum +check: config/log.toml_018f70d1-3100-7762-b28b-8f85ac4ed3cd +check: config/tn.toml_018f70d1-310e-78fc-ac96-aa5e06981bd7 +... +``` + +## Delete backup + +### Grammar structure + +``` +mo_br delete ID + //To delete the backup data. If the backup is on s3 (oss minio), you need to specify + --access_key_id + --secret_access_key + --not_delete_data + --meta_path +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +| ID | ID of the backup to be deleted | +|access_key_id| Access key ID of s3 backed up| +|secret_access_key| Secret access key of s3 backed up| +|not_delete_data|Only delete the information in meta. Backup data is not deleted. | +|meta_path|Specifies the meta file location. If not specified, the default is the mo_br.meta file in the same directory. | + +### Example + +- Delete local file system backup + +``` +./mo_br delete e4cade26-3139-11ee-8631-acde48001122 +``` + +- Delete a backup on the minio. + +``` +./mo_br delete e4cade26-3139-11ee-8631-acde48001122 --access_key_id "S0kwLuB4JofVEIAxWTqf" --secret_access_key "X24O7t3hccmqUZqvqvmLN8464E2Nbr0DWOu9Qs5A" +``` + +## Restore backup + +### Grammar structure + +- Restore backup of specified ID + +``` +mo_br restore ID + //Read the backup data of the specified ID. If the backup is on s3 (oss minio), you need to specify + --backup_access_key_id + --backup_secret_access_key +//Restore target path restore_directory + --restore_dir s3|filesystem + //s3 + --restore_endpoint + --restore_access_key_id + --restore_secret_access_key + --restore_bucket + --restore_filepath + --restore_region + --restore_compression + --restore_role_arn + --restore_is_minio + //filesystem + --restore_path + --dn_config_path + --meta_path + --checksum +--parallelism +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|ID | ID to restore| +|backup_access_key_id|Access key ID backed up in s3| +|backup_secret_access_key |Backup Secret access key in s3| +|restore_dir | Restore target path type. Used when specifying the destination path for recovery. s3|filesystem| +|restore_endpoint| URL to connect to restore to S3 service| +|restore_access_key_id| Restore Access key ID to s3| +|restore_secret_access_key| Restore Secret access key to s3| +|restore_bucket| The bucket that needs to be accessed to restore to s3| +|restore_filepath|Restore relative file path to s3| +|restore_region| Object storage service region restored to s3| +|restore_compression|Restore the compression format of S3 files to s3. | +|restore_role_arn| The resource name of the role restored to s3. | +|restore_is_minio|Specifies whether the restored s3 is minio| +|restore_path|Restore to local path| +|dn_config_path| dn configuration path| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | +|checksum |The parallelism of tae file copy during recovery, the default is 1| +|parallelism|Parallelism| + +- Do not specify restore backup ID + +``` +//recover. +mo_br restore + --backup_dir s3|filesystem Target path type for backup. Used when specifying the destination path for backup. + //s3 + --backup_endpoint + --backup_access_key_id + --backup_secret_access_key + --backup_bucket + --backup_filepath + --backup_region + --backup_compression + --backup_role_arn + --backup_is_minio + //filesystem + --backup_path + //Restore target path restore_directory + --restore_dir s3|filesystem The target path type for recovery. Used when specifying the destination path for recovery. + //s3 + --restore_endpoint + --restore_access_key_id + --restore_secret_access_key + --restore_bucket + --restore_filepath + --restore_region + --restore_compression + --restore_role_arn + --restore_is_minio + //filesystem + --restore_path + --dn_config_path + --meta_path + --checksum + --parallelism +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|backup_dir | The destination path type for recovery. Used when specifying the destination path for recovery. s3|filesystem| +|backup_endpoint| URL to connect to backup in s3| +|backup_access_key_id| Access key ID backed up in s3| +|backup_secret_access_key| Secret access key backed up in s3| +|backup_bucket| Bucket backed up in s3| +|backup_filepath| Relative file path of backup in s3| +|backup_region| Service region backed up in s3| +|backup_compression| Compression format for files backed up in s3. | +|backup_role_arn| Resource name of the role backed up in s3. | +|backup_is_minio| specifies whether the backed up s3 is minio| +|backup_path| The path of local backup| +|restore_dir | Restore target path type. Used when specifying the destination path for recovery. s3 or filesystem| +|restore_endpoint| URL to connect to restore to S3 service| +|restore_access_key_id| Restore Access key ID to s3| +|restore_secret_access_key| Restore Secret access key to s3| +|restore_bucket| The bucket that needs to be accessed to restore to s3| +|restore_filepath|Restore relative file path to s3| +|restore_region| Object storage service region restored to s3| +|restore_compression|Restore the compression format of S3 files to s3. | +|restore_role_arn| The resource name of the role restored to s3. | +|restore_is_minio|Specifies whether the restored s3 is minio| +|restore_path|Restore the path to local matrixone| +|dn_config_path| dn configuration path| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | +|checksum |The parallelism of tae file copy during recovery, the default is 1| +|parallelism|Parallelism| + +### Example + +Restore from file system to file system + +**Step 1:**Stop mo and delete mo-data + +**Step 2:**Execute the following recovery command + +``` +./mo_br restore fb26fd88-41bc-11ee-93f8-acde48001122 --restore_dir filesystem --restore_path "your_mopath" +``` + +After recovery, a new mo-data file will be generated in matrixone + +**Step 3:**Start mo + +## Verify the backup verification code + +Read each file in the backup folder and its sha256 file. Calculate the sha256 value of the file and compare it to the sha256 file value. sha256 files are created when files are created or updated. + +### Grammar structure + +- Verify backup of an ID + +``` +mo_br check ID + //Verify the backup data of the specified ID. If the backup is on s3 (oss minio), you need to specify + --backup_access_key_id string + --backup_secret_access_key string + --meta_path string specifies the meta file location. If not specified, the default is the mo_br.meta file in the same directory. +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|backup_access_key_id| Access key ID backed up in s3| +|backup_secret_access_key| Secret access key backed up in s3| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | + +- Verify backup, specify backup path + +``` +mo_br check + --backup_dir s3|filesystem + //s3 + --backup_endpoint + --backup_access_key_id + --backup_secret_access_key + --backup_bucket + --backup_filepath + --backup_region + --backup_compression + --backup_role_arn + --backup_is_minio + //filesystem + --backup_path + --meta_path +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|backup_dir | The path type where the backup is located. It must be specified when the ID is not specified. s3 or filesystem| +|backup_endpoint| URL to connect to backup in s3| +|backup_access_key_id| Access key ID backed up in s3| +|backup_secret_access_key| Secret access key backed up in s3| +|backup_bucket| Bucket backed up in s3| +|backup_filepath| Relative file path of backup in s3| +|backup_region| Service region backed up in s3| +|backup_compression| Compression format for files backed up in s3. | +|backup_role_arn| Resource name of the role backed up in s3. | +|backup_is_minio| specifies whether the backed up s3 is minio| +|backup_path| The path of local backup| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | + +### Example + +- Verify backup of an ID + +``` +./mo_br check 1614f462-126c-11ef-9af3-26dd28356ef3 ++----------------------------------------+--------+-----------------------------------+------------------------------+---------------+---------------------------+ +| ID | SIZE | PATH | AT TIME | DURATION | COMPLETE TIME | ++----------------------------------------+--------+-----------------------------------+------------------------------+---------------+---------------------------+ +| 1614f462-126c-11ef-9af3-26dd28356ef3 | 126 MB | BackupDir: filesystem Path: | 2024-05-15 11:34:28 +0800 | 22.455633916s | 2024-05-15 11:34:50 +0800 | +| | | /Users/admin/soft/incbackup/back2 | | | | ++----------------------------------------+--------+-----------------------------------+------------------------------+---------------+---------------------------+ +Checking the backup data... + +check: /backup_meta +check: /mo_meta +check: hakeeper/hk_data +check: tae/tae_list +check: tae/tae_sum +check: config/launch.toml_018f7a50-d300-7017-8580-150edf08733e +... +``` + +- Verify backups in a backup directory + +``` +(base) admin@admindeMacBook-Pro mo-backup % ./mo_br check --backup_dir filesystem --backup_path /Users/admin/soft/incbackup/back2 +2024/05/15 11:40:30.011160 +0800 INFO malloc/malloc.go:42 malloc {"max buffer size": 1073741824, "num shards": 16, "classes": 23, "min class size": 128 , "max class size": 1048576, "buffer objects per class": 23} +check: /backup_meta +check: /mo_meta +check: hakeeper/hk_data +check: tae/tae_list +check: tae/tae_sum +check: config/launch.toml_018f7a50-d300-7017-8580-150edf08733e +check: config/log.toml_018f7a50-d30c-7ed0-85bc-191e9f1eb753 +... +``` + +## Best Practices + +Below we will use a few simple examples to describe how to use mo_br to back up and restore data. + +## Example 1 Full backup and restore + +- Connect mo to create databases db1, db2. + +```sql +create database db1; +create database db2; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +|db1| +|db2| +| information_schema | +| mo_catalog | +| mo_debug | +| mo_task | +| mysql | +| system | +| system_metrics | ++--------------------+ +9 rows in set (0.00 sec) +``` + +- Create full backup + +``` +./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "/Users/admin/soft/backuppath/syncback1" + +Backup ID + 25536ff0-126f-11ef-9902-26dd28356ef3 + +./mo_br list ++-----------------------------------------+-----+------------------------------------------+--------------------------+-----------------+---------------------------+-----------------------+----------------+ +| ID | SIZE | PATH | AT TIME | DURATION | COMPLETE TIME | BACKUPTS | BACKUPTYPE | ++-----------------------------------------+-----+------------------------------------------+--------------------------+-----------------+---------------------------+-----------------------+----------------+ +| 25536ff0-126f-11ef-9902-26dd28356ef3 | 65 MB | BackupDir: filesystem Path: | 2024-05-15 11:56:44 +0800| 45404915410000 -1 | full | +| | | /Users/admin/soft/backuppath/syncback1 | | | | | | ++-----------------------------------------+-----+------------------------------------------+--------------------------+-----------------+---------------------------+-----------------------+----------------+ +``` + +- Connect mo to drop database db1 and create database db3. + +```sql +drop database db1; +create database db3; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +|db2| +|db3| +| information_schema | +| mo_catalog | +| mo_debug | +| mo_task | +| mysql | +| system | +| system_metrics | ++--------------------+ +9 rows in set (0.00 sec) +``` + +- Stop mo service, delete mo-data, restore backup + +``` +mo_ctl stop +rm -rf /Users/admin/soft/matrixone/mo-data + +./mo_br restore 25536ff0-126f-11ef-9902-26dd28356ef3 --restore_dir filesystem --restore_path "/Users/admin/soft/matrixone" +From: + BackupDir: filesystem + Path: /Users/admin/soft/backuppath/syncback1 + +To + BackupDir: filesystem +Path: /Users/admin/soft/matrixone + +TaePath + ./mo-data/shared +restore tae file path ./mo-data/shared, parallelism 1, parallel count num: 1 +restore file num: 1, total file num: 733, cost: 549µs +Copy tae file 1 + 018f7a41-1881-7999-bbd6-858c3d4acc18_00000 => mo-data/shared/018f7a41-1881-7999-bbd6-858c3d4acc18_00000 + ... +``` + +- Start mo and check the recovery status + +``` +mo_ctl start +``` + +```sql +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +|db1| +|db2| +| information_schema | +| mo_catalog | +| mo_debug | +| mo_task | +| mysql | +| system | +| system_metrics | ++--------------------+ +9 rows in set (0.00 sec) +``` + +As you can see, the recovery was successful. + +## Example 2 Incremental backup and recovery + +- Connect mo to create databases db1, db2 + +```sql +create database db1; +create database db2; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +|db1| +|db2| +| information_schema | +| mo_catalog | +| mo_debug | +| mo_task | +| mysql | +| system | +| system_metrics | ++--------------------+ +9 rows in set (0.00 sec) +``` + +- Create full backup + +``` +./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "/Users/admin/soft/backuppath/syncback2" + +Backup ID + 2289638c-1284-11ef-85e4-26dd28356ef3 +``` + +- Create incremental backup based on the above full backup + +``` +./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "/Users/admin/soft/backuppath/syncback2" --backup_type "incremental" --base_id "2289638c-1284-11ef-85e4-26dd28356ef3" + +Backup ID + 81531c5a-1284-11ef-9ba3-26dd28356ef3 + +./mo_br list ++-----------------------------------------+-----+------------------------------------------+--------------------------+-----------------+----------------------------+-----------------------+-------------+ +| ID | SIZE | PATH | AT TIME | DURATION | COMPLETE TIME | BACKUPTS | BACKUPTYPE | ++-----------------------------------------+-----+------------------------------------------+--------------------------+-----------------+----------------------------+-----------------------+-------------+ +| 2289638c-1284-11ef-85e4-26dd28356ef3 | 70 MB | BackupDir: filesystem Path: |2024-05-15 14:26:59 +0800 | 754419668571000 -1 | full | +| | | /Users/admin/soft/backuppath/syncback2 | | | | | | +81531c5a-1284-11ef-9ba3-26dd28356ef3 | BackupDir: filesystem Path: | 2024-05-15 14:29:38 +0800 54578690660000 -1 | incremental | +| | | /Users/admin/soft/backuppath/syncback2 | | | | | | ++-----------------------------------------+-----+------------------------------------------+--------------------------+-----------------+----------------------------+-----------------------+-------------+ +``` + +Comparing the duration of incremental backup and full backup, you can see that incremental backup takes less time. + +- Connect mo to drop database db1 and create database db3. + +```sql +drop database db1; +create database db3; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +|db2| +|db3| +| information_schema | +| mo_catalog | +| mo_debug | +| mo_task | +| mysql | +| system | +| system_metrics | ++--------------------+ +9 rows in set (0.00 sec) +``` + +- Stop mo service, delete mo-data, restore backup + +``` +mo_ctl stop +rm -rf /Users/admin/soft/matrixone/mo-data + +./mo_br restore 81531c5a-1284-11ef-9ba3-26dd28356ef3 --restore_dir filesystem --restore_path "/Users/admin/soft/matrixone" +2024/05/15 14:35:27.910925 +0800 INFO malloc/malloc.go:43 malloc {"max buffer size": 2147483648, "num shards": 8, "classes": 23, "min class size": 128 , "max class size": 1048576, "buffer objects per class": 22} +From: + BackupDir: filesystem + Path: /Users/admin/soft/backuppath/syncback2 + +To + BackupDir: filesystem + Path: /Users/admin/soft/matrixone + +TaePath + ./mo-data/shared +... +``` + +- Start mo and check the recovery status + +``` +mo_ctl start +``` + +```sql +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +|db1| +|db2| +| information_schema | +| mo_catalog | +| mo_debug | +| mo_task | +| mysql | +| system | +| system_metrics | ++--------------------+ +9 rows in set (0.00 sec) +``` + +As you can see, the recovery was successful. \ No newline at end of file diff --git a/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-physical-backup-restore.md b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-physical-backup-restore.md index 551a57e46..2bfd6f670 100644 --- a/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-physical-backup-restore.md +++ b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-physical-backup-restore.md @@ -1,19 +1,18 @@ -## Overview of principles +## Principle overview -A regular physical backup of a database is a direct copy of the database's physical storage files, including data files, log files, and control files, to create a separate copy of the database. This process is usually performed at the file system level and can be achieved by operating system commands. The resulting backup is a full backup of the database, containing all the data and objects. Backup files can be stored on multiple media and can be compressed and encrypted to save space and improve security. On recovery, these files can be copied directly to the desired location to quickly restore the entire database. In addition, physical backups support cross-platform migration for disaster recovery and database migration scenarios, but may require more storage space and time. +Regular physical backup of the database is to directly copy the physical storage files of the database, including data files, log files, control files, etc., to create an independent copy of the database. This process is usually performed at the file system level and can be implemented through operating system commands. The generated backup is a complete backup of the database, including all data and objects. Backup files can be stored on a variety of media and can be compressed and encrypted to save space and increase security. During recovery, these files can be copied directly to the required location to quickly recover the entire database. In addition, physical backup supports cross-platform migration and is suitable for disaster recovery and database migration scenarios, but may require more storage space and time. -A full backup is a backup process that backs up all data in a database. It creates a full copy of the database, which usually requires more storage space and longer to complete. Because it contains all the data, full backups are simpler to restore and can be restored directly to the state they were in. +Full backup refers to the backup process of backing up all data in the database. It creates a complete copy of the database, which usually requires more storage space and longer time to complete. Because it contains all data, full backup is relatively simple to restore and can be restored directly to the state at the time of backup. -Incremental backups are backups of data that has changed since the last backup. It replicates only blocks or data files that have been modified between backups, so backup sets are typically smaller and faster. Incremental backups can save storage space and backup time, but can be more complex when data is restored because a series of incremental backups need to be applied sequentially to restore to the latest state. - -MatrixOne supports incremental and full physical backup restores using `mo_br`: +Incremental backup means backing up data that has changed since the last backup. It only copies data blocks or data files that have been modified between backups, so the backup set is usually smaller and the backup speed is faster. Incremental backups can save storage space and backup time, but may be more complex during data recovery because a series of incremental backups need to be applied in sequence to restore to the latest state. +MatrixOne supports incremental and full physical backup recovery using `mo_br`: !!! note - mo_br Backup and recovery tool for enterprise services, you need to contact your MatrixOne account manager for the tool download path. + mo_br enterprise-level service backup and recovery tool, you need to contact your MatrixOne account manager to obtain the tool download path. -## Examples +## Example -### Example 1 Full Backup Recovery +### Example 1 Full backup and restore - Connect mo to create databases db1, db2. @@ -38,9 +37,9 @@ mysql> show databases; 9 rows in set (0.00 sec) ``` -- Create a full backup +- Create full backup -``` +```bash ./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "/Users/admin/soft/backuppath/syncback1" Backup ID @@ -55,7 +54,7 @@ Backup ID +--------------------------------------+-------+----------------------------------------+---------------------------+--------------+---------------------------+-----------------------+------------+ ``` -- Connect mo Delete database db1 and build database db3. +- Connect mo to drop database db1 and create database db3. ```sql drop database db1; @@ -102,7 +101,7 @@ Copy tae file 1 ... ``` -- Start mo, check recovery +- Start mo and check the recovery status ``` mo_ctl start @@ -128,9 +127,9 @@ mysql> show databases; As you can see, the recovery was successful. -### Example 2 Incremental Backup Recovery +### Example 2 Incremental backup and recovery -- Connect mo Create databases db1, db2 +- Connect mo to create databases db1, db2 ```sql create database db1; @@ -153,7 +152,7 @@ mysql> show databases; 9 rows in set (0.00 sec) ``` -- Creating Full Backups +- Create full backup ``` ./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "/Users/admin/soft/backuppath/syncback2" @@ -162,7 +161,7 @@ Backup ID 2289638c-1284-11ef-85e4-26dd28356ef3 ``` -- Create incremental backups based on the above full backup +- Create incremental backup based on the above full backup ``` ./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "/Users/admin/soft/backuppath/syncback2" --backup_type "incremental" --base_id "2289638c-1284-11ef-85e4-26dd28356ef3" @@ -181,9 +180,9 @@ Backup ID +--------------------------------------+-------+----------------------------------------+---------------------------+--------------+---------------------------+-----------------------+-------------+ ``` -Comparing the time consumption of incremental and full backups, you can see that incremental backups take less time. +Comparing the duration of incremental backup and full backup, you can see that incremental backup takes less time. -- Connect mo Delete database db1 and build database db3. +- Connect mo to drop database db1 and create database db3. ```sql drop database db1; @@ -227,7 +226,7 @@ TaePath ... ``` -- Start mo, check recovery +- Start mo and check the recovery status ``` mo_ctl start @@ -251,4 +250,4 @@ mysql> show databases; 9 rows in set (0.00 sec) ``` -As you can see, the recovery was successful. +As you can see, the recovery was successful. \ No newline at end of file diff --git a/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-pitr-backup-restore.md b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-pitr-backup-restore.md new file mode 100644 index 000000000..f9f6faef7 --- /dev/null +++ b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-pitr-backup-restore.md @@ -0,0 +1,229 @@ +# mo_br tool for PITR backup and recovery + +## PITR backup and recovery implementation principle + +The implementation of PITR (point-in-time recovery) backup and recovery is based on full backup and incremental logs. By recording the transaction operations of the database and applying these incremental logs, the system can restore the database to a specified point in time. The recovery process starts with a full backup, restoring the database to its base state, and then sequentially applies transactions from the incremental log until the target point in time is reached, allowing for precise data recovery. This mechanism effectively copes with data loss caused by misoperation or failure and ensures the integrity and consistency of the database. + +## Application scenarios + +PITR is a common function in database management. It is mainly used to restore the database to a specific point in time. It is usually used in the following scenarios: + +- **Misoperation recovery:**When some data in the database is accidentally deleted, accidentally modified, or accidentally operated, you can use PITR to restore the database to the state before the accidental operation to avoid data loss. + +- **Disaster recovery:**After encountering a catastrophic failure of the database (such as hardware failure, software error, natural disaster, etc.), PITR can help restore the database to a point in time before the failure, ensuring business continuity and data Integrity. +- **Data Audit and Compliance:**For businesses and organizations that need to conduct data audits or need to comply with data retention policies, PITR can provide historical data recovery and viewing capabilities to meet compliance requirements. + +- **Testing and development environment management:**During the development and testing process, PITR can help restore the database to the previous test data set to support repeated running of test cases and management of the development environment. + +- **Database Backup Management:**Used in conjunction with regular backups, PITR can provide more fine-grained recovery options, making the recovery process more flexible and precise. + +## MatrixOne support for PITR + +MatrixOne supports the following two methods for PITR: + +- sql statement +- mo_br tool + +This document mainly introduces how to use `mo_br` to perform PITR. + +!!! note + mo_br enterprise-level service backup and recovery tool, you need to contact your MatrixOne account manager to obtain the tool download path. + +## Preparation before starting + +- Completed [Standalone Deployment MatrixOne](../../../Get-Started/install-standalone-matrixone.md) + +- Completed mo_br tool deployment + +## Example + +A large SaaS provider manages a multi-tenant environment that provides various services to different business customers. These tenants host their databases through clusters, storing business-critical data. One day, when the system administrator was performing an upgrade operation, he accidentally triggered some erroneous script operations, which affected multiple databases and tables in different tenants. In order to ensure that the system can be restored as quickly as possible and reduce data loss, the team decided to PITR for recovery. + +### Create business data + +There are two tenants acc1 and acc2 under this system, which represent different customers. Each tenant has product information and order information. + +```sql +create account acc1 admin_name 'root' identified by '111'; +create account acc2 admin_name 'root' identified by '111'; + +#Connect acc1 +create database db1; +use db1; +--Create a product table to store product information +CREATE TABLE products ( + id INT PRIMARY KEY AUTO_INCREMENT, +product_name VARCHAR(100) NOT NULL, + category VARCHAR(50), + price DECIMAL(10, 2) NOT NULL, + stock_quantity INT NOT NULL +); + +INSERT INTO products (product_name, category, price, stock_quantity) VALUES +('Gaming Laptop', 'Electronics', 1200.99, 30), +('Wireless Mouse', 'Accessories', 25.50, 100), +('Mechanical Keyboard', 'Accessories', 75.00, 50), +('Smartphone', 'Electronics', 699.99, 50), +('Headphones', 'Accessories', 199.99, 80), +('Tablet', 'Electronics', 299.99, 40), +('Smartwatch', 'Accessories', 149.99, 60), +('Charger', 'Accessories', 19.99, 200), +('Webcam', 'Electronics', 59.99, 75); + +create database db2; +use db2; +--Create a table to store customer orders +CREATE TABLE orders ( + id int PRIMARY KEY auto_increment, + product_name VARCHAR(100) NOT NULL, + quantity INT NOT NULL, + order_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +INSERT INTO orders (product_name, quantity) VALUES +('Laptop', 2), +('Mouse', 5), +('Keyboard', 3); + +#Connect acc2 +create database db1; +use db1; +--Create a product table to store product information +CREATE TABLE products ( + id INT PRIMARY KEY AUTO_INCREMENT, + product_name VARCHAR(100) NOT NULL, + category VARCHAR(50), + price DECIMAL(10, 2) NOT NULL, + stock_quantity INT NOT NULL +); + +INSERT INTO products (product_name, category, price, stock_quantity) VALUES +('Gaming Laptop', 'Electronics', 1200.99, 30), +('Wireless Mouse', 'Accessories', 25.50, 100), +('Mechanical Keyboard', 'Accessories', 75.00, 50), +('Smartphone', 'Electronics', 699.99, 50), +('Headphones', 'Accessories', 199.99, 80), +('Tablet', 'Electronics', 299.99, 40), +('Smartwatch', 'Accessories', 149.99, 60), +('Charger', 'Accessories', 19.99, 200), +('Webcam', 'Electronics', 59.99, 75); + +create database db2; +use db2; +--Create a table to store customer orders +CREATE TABLE orders ( + id int PRIMARY KEY auto_increment, + product_name VARCHAR(100) NOT NULL, + quantity INT NOT NULL, + order_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); + +INSERT INTO orders (product_name, quantity) VALUES +('Smartphone', 10), +('Headphones', 15), +('Tablet', 7); +``` + +### Create pitr + +```bash +#Create a cluster-level pitr +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "root" --password "111" --pname "pitr1" --level "cluster" --rangevalue 1 --rangeunit "d " + +>./mo_br pitr show --host "127.0.0.1" --port 6001 --user "dump" --password "111" +PITR NAME CREATED TIME MODIFIED TIME PITR LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME PITR LENGTH PITR UNIT +pitr1 2024-10-28 17:22:26 2024-10-28 17:22:26 cluster ***1 d + +#Create acc1 tenant level pitr +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "acc1#root" --password "111" --pname "pitr2" --level "account" --account "acc1" --rangevalue 10 --rangeunit "h" + +>./mo_br pitr show --host "127.0.0.1" --port 6001 --user "acc1#root" --password "111" --account "acc1" +PITR NAME CREATED TIME MODIFIED TIME PITR LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME PITR LENGTH PITR UNIT +pitr2 2024-10-28 17:24:18 2024-10-28 17:24:18 account acc1 **10 h + +#Create acc2 tenant level pitr +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "acc2#root" --password "111" --pname "pitr3" --level "account" --account "acc2" --rangevalue 10 --rangeunit "h" + + +>./mo_br pitr show --host "127.0.0.1" --port 6001 --user "acc2#root" --password "111" --account "acc2" +PITR NAME CREATED TIME MODIFIED TIME PITR LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME PITR LENGTH PITR UNIT +pitr3 2024-10-28 17:25:32 2024-10-28 17:25:32 account acc2 **10 h +``` + +### Simulate upgrade error + +Due to an upgrade script error, both db1 and db2 under acc1 were deleted, db1 under acc2 was deleted, and the orders table data under db2 was cleared. + +```sql +--Execute under acc1 +drop database db1; +drop database db2; + +--Execute under acc2 +drop database db1; +use db2; +truncate table orders; +``` + +### Restore pitr + +- Full tenant recovery for acc1 + + ```bash + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc1#root" --password "111" --pname "pitr2" --timestamp "2024-10-28 17:30: 00" --account "acc1" + ``` + + Connect acc1 query, you can see that the data is successfully restored + + ```sql + mysql> show databases; + +--------------------+ + | Database | + +--------------------+ + | db1 | + | db2 | + | information_schema | + | mo_catalog | + | mysql | + | system | + | system_metrics | + +--------------------+ + 7 rows in set (0.01 sec) + ``` + +- Database table level recovery for acc2 + + ```bash + #restoredb1 + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc2#root" --password "111" --pname "pitr3" --timestamp "2024-10-28 17:30: 00" --account "acc2" --database "db1" + + #Restore orders table + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc2#root" --password "111" --pname "pitr3" --timestamp "2024-10-28 17:30: 00" --account "acc2" --database "db2" --table "orders" + ``` + + Connect acc2 query and you can see that the data is successfully restored + + ```sql + mysql> show databases; + +--------------------+ + | Database | + +--------------------+ + | db1 | + | db2 | + | information_schema | + | mo_catalog | + | mysql | + | system | + | system_metrics | + +--------------------+ + 7 rows in set (0.01 sec) + + mysql> select * from db2.orders; + +------+--------------+----------+---------------------+ + | id | product_name | quantity | order_date | + +------+--------------+----------+---------------------+ + | 1 | Smartphone | 10 | 2024-10-28 17:11:27 | + | 2 | Headphones | 15 | 2024-10-28 17:11:27 | + | 3 | Tablet | 7 | 2024-10-28 17:11:27 | + +------+--------------+----------+---------------------+ + 3 rows in set (0.00 sec) + ``` \ No newline at end of file diff --git a/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-snapshot-backup-restore.md b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-snapshot-backup-restore.md index 26098c654..77fe3cc65 100644 --- a/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-snapshot-backup-restore.md +++ b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-snapshot-backup-restore.md @@ -1,72 +1,70 @@ -# mo_br tool for snapshot backup recovery +# mo_br tool for snapshot backup and recovery -## Snapshot Backup Recovery Implementation Principles +## Snapshot backup and recovery implementation principle -Database snapshot backup restores by creating a read-only static view of the database at a specific point in time, known as a snapshot. Snapshots utilize the storage system's copy-on-write (COW) technology to copy and store the original data page only before it is modified, creating a stateful copy of the database at the time the snapshot was created. When you need to recover data, you can pick the data from the snapshot and copy or restore it to a new or existing database. Snapshot files are initially small and grow as the source database changes, so their size needs to be monitored and managed as necessary. Snapshots must be on the same server instance as the source database, and since they are read-only, writes cannot be made directly on them. Note that snapshot recovery operations overwrite the current data, so caution is required. +Database snapshot backup and recovery create a read-only static view of the database at a specific point in time. This view is called a snapshot. Snapshots use the storage system's copy-on-write (COW) technology to copy and store the original data page only before it is modified, thereby generating a copy of the state of the database at the time the snapshot was created. When data needs to be restored, the data in the snapshot can be selected and copied or restored to a new or existing database. Snapshot files start small and gradually grow as changes are made to the source database, so their size needs to be monitored and managed if necessary. Snapshots must be on the same server instance as the source database, and since they are read-only, write operations cannot be performed directly on them. It should be noted that the snapshot recovery operation will overwrite the current data, so you need to operate with caution. ## Application scenarios -Database snapshots are a powerful tool to improve database availability and performance in multiple scenarios. Here are some application scenarios for snapshots: +Database snapshots are a powerful tool that can improve database availability and performance in a variety of scenarios. The following are some application scenarios of snapshots: -- **Data Backup and Recovery**: Snapshots can be used as a way to backup a database, allowing a read-only copy of the database to be created for data backup and recovery without stopping the database service. +- **Data Backup and Recovery**: Snapshot can be used as a method of database backup, which allows the creation of a read-only copy of the database without stopping the database service for data backup and recovery. -- **Reports and Data Analysis**: Snapshots can be used to avoid impacting online transactions when databases are required to remain static for report generation or data analysis. +- **Reporting and Data Analysis**: When the database needs to remain static for report generation or data analysis, snapshots can be used to avoid affecting online transaction processing. +- **Development and Testing**: Before developing new features or testing the system, a copy of the database can be created through a snapshot so that testing can be performed without affecting the production environment. -- **Development and testing**: Before developing a new feature or testing system, a copy of the database can be created by snapshot so that testing can be done without affecting the production environment. +- **Data Migration**: During the data migration process, snapshots can be used to ensure data consistency and avoid data changes during the migration process. -- **Data migration**: During data migration, snapshots can be used to ensure data consistency and avoid data changes during migration. - -- **High-risk operational protection**: Before performing operations that may have an impact on database stability, such as database upgrades, structural changes, etc., snapshots can be created so that they can be quickly restored if the operation fails. +- **High-risk operation protection**: Before performing operations that may affect database stability (such as database upgrades, structural changes, etc.), you can create a snapshot so that you can quickly recover if the operation fails. ## MatrixOne support for snapshots -MatrixOne supports two ways to perform tenant-level snapshot backup restores: +MatrixOne supports the following two methods for snapshot backup and recovery: - sql statement - mo_br tool -This document focuses on using `mo_br` for tenant-level snapshot backup restores. +This document mainly introduces how to use `mo_br` to perform cluster/tenant level snapshot backup and recovery. !!! note - mo_br Backup and recovery tool for enterprise services, you need to contact your MatrixOne account manager for the tool download path. + mo_br enterprise-level service backup and recovery tool, you need to contact your MatrixOne account manager to obtain the tool download path. -## Prepare before you start +## Preparation before starting -- Completed [standalone deployment of](../../../Get-Started/install-standalone-matrixone.md) MatrixOne +- Completed [Standalone Deployment MatrixOne](../../../Get-Started/install-standalone-matrixone.md) - Completed mo_br tool deployment -## Examples +## Example -## Example 1 Table Level Recovery +## Example 1 Table level recovery -- Connecting Matrixone System Tenants to Execute Table-Building Statements +- Connect to the Matrixone system tenant to execute table creation statements ```sql create db if not exists snapshot_read; use snapshot_read; create table test_snapshot_read (a int); -INSERT INTO test_snapshot_read (a) VALUES(1), (2), (3), (4), (5),(6), (7), (8), (9), (10), (11), (12),(13), (14), (15), (16), (17), (18), (19), (20),(21), (22), (23), (24), (25), (26), (27), (28), (29), (30),(31), (32), (33), (34), (35), (36), (37), (38), (39), (40),(41), (42), (43), (44), (45), (46), (47), (48), (49), (50),(51), (52), (53), (54), (55), (56), (57), (58), (59), (60),(61), (62), (63), (64), (65), (66), (67), (68), (69), (70),(71), (72), (73), (74), (75), (76), (77), (78), (79), (80), (81), (82), (83), (84), (85), (86), (87), (88), (89), (90),(91), (92), (93), (94), (95), (96), (97), (98), (99), (100); - +INSERT INTO test_snapshot_read (a) VALUES(1), (2), (3), (4), (5),(6), (7), (8), (9), (10), (11) , (12),(13), (14), (15), (16), (17), (18), (19), (20),(21), (22), (23), ( 24), (25), (26), (27), (28), (29), (30), (31), (32), (33), (34), (35), (36) , (37), (38), (39), (40), (41), (42), (43), (44), (45), (46), (47), (48), ( 49), (50),(51), (52), (53), (54), (55), (56), (57), (58), (59), (60),(61) , (62), (63), (64), (65), (66), (67), (68), (69), (70), (71), (72), (73), ( 74), (75), (76), (77), (78), (79), (80), (81), (82), (83), (84), (85), (86) , (87), (88), (89), (90), (91), (92), (93), (94), (95), (96), (97), (98), ( 99), (100); mysql> select count(*) from snapshot_read.test_snapshot_read; +----------+ | count(*) | +----------+ -| 100 | +| 100 | +----------+ ``` -- Create a snapshot +- Create snapshot -``` -./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --level "account" --sname "sp_01" --account "sys" +```bash +./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --level "account" --sname "sp_01" --account "sys" ./mo_br snapshot show --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" -SNAPSHOT NAME TIMESTAMP SNAPSHOT LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME -sp_01 2024-05-10 02:06:08.01635 account sys +SNAPSHOT NAME TIMESTAMP SNAPSHOT LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME +sp_01 2024-05-10 02:06:08.01635 account sys ``` -- Connect the Matrixone system tenant by deleting some of the data in the table. +- Connect to the Matrixone system tenant and delete some data in the table. ```sql delete from snapshot_read.test_snapshot_read where a <= 50; @@ -75,44 +73,43 @@ mysql> select count(*) from snapshot_read.test_snapshot_read; +----------+ | count(*) | +----------+ -| 50 | +| 50 | +----------+ ``` - Table level restored to this tenant -``` +```bash ./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --db "snapshot_read" --table "test_snapshot_read" --sname "sp_01" ``` -- Connect Matrixone System Tenant Query Recovery +- Connect to the Matrixone system tenant to query the recovery status ```sql mysql> select count(*) from snapshot_read.test_snapshot_read; +----------+ | count(*) | +----------+ -| 100 | +| 100 | +----------+ ``` -## Example 2 Database Level Recovery +## Example 2 Database level recovery -- Connect Matrixone system tenant to execute sql statement +- Connect to the Matrixone system tenant to execute sql statements ```sql create db if not exists snapshot_read; use snapshot_read; create table test_snapshot_read (a int); -INSERT INTO test_snapshot_read (a) VALUES(1), (2), (3), (4), (5),(6), (7), (8), (9), (10), (11), (12),(13), (14), (15), (16), (17), (18), (19), (20),(21), (22), (23), (24), (25), (26), (27), (28), (29), (30),(31), (32), (33), (34), (35), (36), (37), (38), (39), (40),(41), (42), (43), (44), (45), (46), (47), (48), (49), (50),(51), (52), (53), (54), (55), (56), (57), (58), (59), (60),(61), (62), (63), (64), (65), (66), (67), (68), (69), (70),(71), (72), (73), (74), (75), (76), (77), (78), (79), (80), (81), (82), (83), (84), (85), (86), (87), (88), (89), (90),(91), (92), (93), (94), (95), (96), (97), (98), (99), (100); -create table test_snapshot_read_1(a int); -INSERT INTO test_snapshot_read_1 (a) VALUES(1), (2), (3), (4), (5),(6), (7), (8), (9), (10), (11), (12),(13), (14), (15), (16), (17), (18), (19), (20),(21), (22), (23), (24), (25), (26), (27), (28), (29), (30),(31), (32), (33), (34), (35), (36), (37), (38), (39), (40),(41), (42), (43), (44), (45), (46), (47), (48), (49), (50),(51), (52), (53), (54), (55), (56), (57), (58), (59), (60),(61), (62), (63), (64), (65), (66), (67), (68), (69), (70),(71), (72), (73), (74), (75), (76), (77), (78), (79), (80), (81), (82), (83), (84), (85), (86), (87), (88), (89), (90),(91), (92), (93), (94), (95), (96), (97), (98), (99), (100); - +INSERT INTO test_snapshot_read (a) VALUES(1), (2), (3), (4), (5),(6), (7), (8), (9), (10), (11) , (12),(13), (14), (15), (16), (17), (18), (19), (20),(21), (22), (23), ( 24), (25), (26), (27), (28), (29), (30), (31), (32), (33), (34), (35), (36) , (37), (38), (39), (40), (41), (42), (43), (44), (45), (46), (47), (48), ( 49), (50),(51), (52), (53), (54), (55), (56), (57), (58), (59), (60),(61) , (62), (63), (64), (65), (66), (67), (68), (69), (70), (71), (72), (73), ( 74), (75), (76), (77), (78), (79), (80), (81), (82), (83), (84), (85), (86) , (87), (88), (89), (90), (91), (92), (93), (94), (95), (96), (97), (98), ( 99), (100); +create table test_snapshot_read_1(A int); +INSERT INTO test_snapshot_read_1 (a) VALUES(1), (2), (3), (4), (5),(6), (7), (8), (9), (10), (11) , (12),(13), (14), (15), (16), (17), (18), (19), (20),(21), (22), (23), ( 24), (25), (26), (27), (28), (29), (30), (31), (32), (33), (34), (35), (36) , (37), (38), (39), (40), (41), (42), (43), (44), (45), (46), (47), (48), ( 49), (50),(51), (52), (53), (54), (55), (56), (57), (58), (59), (60),(61) , (62), (63), (64), (65), (66), (67), (68), (69), (70), (71), (72), (73), ( 74), (75), (76), (77), (78), (79), (80), (81), (82), (83), (84), (85), (86) , (87), (88), (89), (90), (91), (92), (93), (94), (95), (96), (97), (98), ( 99), (100); mysql> select count(*) from snapshot_read.test_snapshot_read; +----------+ | count(*) | +----------+ -| 200 | +| 200 | +----------+ 1 row in set (0.00 sec) @@ -120,32 +117,31 @@ mysql> select count(*) from snapshot_read.test_snapshot_read_1; +----------+ | count(*) | +----------+ -| 100 | +| 100 | +----------+ 1 row in set (0.01 sec) ``` -- Create a snapshot +- Create snapshot -``` +```bash ./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --level "account" --sname "sp_02" --account "sys" ./mo_br snapshot show --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" -SNAPSHOT NAME TIMESTAMP SNAPSHOT LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME -sp_02 2024-05-10 02:47:15.638519 account sys +SNAPSHOT NAME TIMESTAMP SNAPSHOT LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME +sp_02 2024-05-10 02:47:15.638519 account sys ``` -- Connection Matrixone system tenant deletes some data +- Connect to Matrixone system tenant to delete some data ```sql delete from snapshot_read.test_snapshot_read where a <= 50; delete from snapshot_read.test_snapshot_read_1 where a >= 50; - mysql> select count(*) from snapshot_read.test_snapshot_read; +----------+ | count(*) | +----------+ -| 100 | +| 100 | +----------+ 1 row in set (0.00 sec) @@ -153,25 +149,25 @@ mysql> select count(*) from snapshot_read.test_snapshot_read_1; +----------+ | count(*) | +----------+ -| 49 | +| 49 | +----------+ 1 row in set (0.01 sec) ``` -- Database level restore to this tenant - -``` +- Database level recovery to this tenant + +```bash ./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --db "snapshot_read" --sname "sp_02" ``` -- Connect Matrixone System Tenant Query Recovery +- Connect to the Matrixone system tenant to query the recovery status ```sql mysql> select count(*) from snapshot_read.test_snapshot_read; +----------+ | count(*) | +----------+ -| 200 | +| 200 | +----------+ 1 row in set (0.00 sec) @@ -179,16 +175,16 @@ mysql> select count(*) from snapshot_read.test_snapshot_read_1; +----------+ | count(*) | +----------+ -| 100 | +| 100 | +----------+ 1 row in set (0.00 sec) ``` -## Example 3 Tenant Level Recovery +## Example 3 Tenant level recovery -Tenant Level Recovery +Tenant level recovery -- Connect Matrixone system tenant to execute sql statement +- Connect to the Matrixone system tenant to execute sql statements ```sql create database if not exists snapshot_read; @@ -209,17 +205,7 @@ mysql> show databases; 8 rows in set (0.00 sec) ``` -- Create a snapshot - -``` -./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --level "account" --sname "sp_03" --account "sys" - -./mo_br snapshot show --host "127.0.0.1" --port 6001 --user "dump" --password "111" -SNAPSHOT NAME TIMESTAMP SNAPSHOT LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME -sp_03 2024-05-11 03:20:16.065685 account sys -``` - -- Connecting Matrixone System Tenant Delete Database +- Connect the Matrixone system tenant to delete the database ```sql drop database snapshot_read; @@ -241,17 +227,17 @@ mysql> show databases; - Tenant level restored to this tenant -``` +```bash ./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --sname "sp_03" ``` -- Tenant level restored to new tenant +- Tenant level reversion to new tenant -``` +```bash ./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --sname "sp_03" --new_account "acc2" --new_admin_name "admin" --new_admin_password "111"; ``` -- Connect Matrixone System Tenant Query Recovery +- Connect to the Matrixone system tenant to query the recovery status ```sql mysql> show databases; @@ -270,7 +256,7 @@ mysql> show databases; 8 rows in set (0.00 sec) ``` -- Connect New Tenant acc2 Query Recovery +- Connect to the new tenant acc2 to query the recovery status ```sql mysql> show databases; @@ -285,4 +271,76 @@ mysql> show databases; | system_metrics | +--------------------+ 6 rows in set (0.00 sec) +``` + +## Example 4 Cluster level recovery + +```sql +--Executed under tenant acc1, acc2 +create database db1; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| db1 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +6 rows in set (0.01 sec) +``` + +- Create snapshot + +```bash +./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "root" --password "111" --level "cluster" --sname "cluster_sp1" + +>./mo_br snapshot show --host "127.0.0.1" --port 6001 --user "dump" --password "111" --cluster "sys" +SNAPSHOT NAME TIMESTAMP SNAPSHOT LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME +cluster_sp1 2024-10-14 03:52:55.657359 cluster +``` + +- Connect the Matrixone system tenant to delete the database + +```sql +--Executed under tenant acc1, acc2 +drop database db1; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +5 rows in set (0.01 sec) +``` + +- recover + +```bash +./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "root" --password "111" --sname "cluster_sp1" +``` + +- Connect to tenants acc1 and acc2 to query the recovery status + +```sql +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| db1 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +6 rows in set (0.01 sec) ``` \ No newline at end of file diff --git a/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr.md b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr.md index 8fc14bfc5..8c7bc7ba5 100644 --- a/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr.md +++ b/docs/MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr.md @@ -1,15 +1,15 @@ -# mo_br Backup and Recovery +# mo_br backup and recovery -Database physical backup and snapshot backup are two important data protection strategies that play an important role in many scenarios. Physical backups enable fast and complete database recovery by replicating the physical files of the database, such as data files and log files, and are especially suitable for overall database migration or disaster recovery situations. Snapshot backup, on the other hand, provides a fast and storage-efficient way of backing up data by recording its status at a specific point in time, for scenarios that require point-in-time recovery or read-only query operations, such as report generation or data analysis. The combination of physical backup recovery, which can take longer, and snapshot backup, which provides fast data access, provides comprehensive protection of the database, ensuring data security and business continuity. +Database physical backup and snapshot backup are two important data protection strategies, and they play an important role in many scenarios. Physical backup can achieve fast and complete database recovery by copying the physical files of the database, such as data files and log files, and is especially suitable for overall database migration or disaster recovery. On the other hand, snapshot backup provides a fast and storage-efficient backup method by recording the status of data at a specific point in time. It is suitable for scenarios that require point-in-time recovery or read-only query operations, such as generating reports or performing data processing. analyze. The recovery of physical backups may take a long time, while snapshot backups can provide fast data access. The combination of the two can provide comprehensive protection for the database and ensure data security and business continuity. -MatrixOne supports regular physical and snapshot backups via the `mo_br` utility. This section describes how `mo_br` is used. +MatrixOne supports regular physical backups and snapshot backups via the `mo_br` utility. This chapter will introduce how to use `mo_br`. !!! note - mo_br Physical backup and recovery tool for enterprise level services, you need to contact your MatrixOne account manager for the tool download path. + mo_br is a physical backup and recovery tool for enterprise-level services. You need to contact your MatrixOne account manager to obtain the tool download path. -## Reference Command Guide +## Reference command guide -help - Print Reference Guide +help -print reference guide ``` ./mo_br help @@ -42,7 +42,7 @@ Use "mo_br [command] --help" for more information about a command. ### Create a backup -#### Syntax structure +#### Grammar structure ``` mo_br backup @@ -66,84 +66,85 @@ mo_br backup --path --parallelism --meta_path - //incremental backup required + //incremental backup and restore needed --backup_type --base_id + ``` **Parameter description** -| parameters | clarification | -| ---- | ---- | -|host | Target MatrixOne's IP| -|port|port number| +| Parameters | Description | +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| |user| user| |password | User's password| -|backup_dir | Destination path type for backups. s3 or filesystem| -|endpoint| URL to connect to the service that backs up to s3| -|access_key_id| Access key ID for backup to s3| -|secret_access_key| Secret access key for backup to s3| -|bucket| Backup to the bucket s3 needs access to| -|filepath| Relative file paths for backups to s3 -|region| Backup to s3's object storage service area| -|compression| The compressed format of the files backed up to s3.| -|role_arn| The resource name of the role backed up to s3.| -|is_minio| Specify whether the backup to s3 is a minio| +|backup_dir | Backup destination path type. s3 or filesystem| +|endpoint| The URL to connect to the service that backs up to s3| +|access_key_id| Access key ID backed up to s3| +|secret_access_key| Secret access key backed up to s3| +|bucket| The bucket that s3 needs to access when backed up| +|filepath| Relative file path for backup to s3| +|region| Object storage service region backed up to s3| +|compression| Compression format of files backed up to s3. | +|role_arn| Resource name of the role backed up to s3. | +|is_minio| specifies whether the s3 backed up to is minio| |path| Local file system backup path| -|parallelism|parallelism| -|meta_path|Specifies the location of the meta file. It can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory.| -|backup_type|Specifies that the backup type is incremental, incremental.| -|base_id|The ID of the last backup, mainly used to determine the timestamp of the last backup.| +|parallelism|Parallelism| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | +|backup_type|Specifies the backup type as incremental backup, incremental. | +|base_id|The ID of the last backup, mainly used to determine the timestamp of the last backup. | -#### Examples +#### Example - Full backup to local file system -``` -./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "yourpath" +```bash +./mo_br backup --host "127.0.0.1" --port 6001 --user "root" --password "111" --backup_dir "filesystem" --path "yourpath" ``` - Full backup to minio -``` -./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "s3" --endpoint "http://127.0.0.1:9000" --access_key_id "S0kwLuB4JofVEIAxxxx" --secret_access_key "X24O7t3hccmqUZqvqvmLN8464E2Nbr0DWOu9xxxx" --bucket "bucket1" --filepath "/backup1" --is_minio +```bash +./mo_br backup --host "127.0.0.1" --port 6001 --user "root" --password "111" --backup_dir "s3" --endpoint "http://127.0.0.1:9000" --access_key_id "S0kwLuB4JofVEIAxWTqf" --secret_access_key "X24O7t3hccmqUZqvqvmLN8464E2Nbr0DWOu9Qs5A" --bucket "bucket1" --filepath "/backup1" --is_minio ``` - Incremental backup to local file system -``` -./mo_br backup --host "127.0.0.1" --port 6001 --user "dump" --password "111" --backup_dir "filesystem" --path "yourpath" --backup_type "incremental" --base_id "xxx" +```bash +./mo_br backup --host "127.0.0.1" --port 6001 --user "root" --password "111" --backup_dir "filesystem" --path "yourpath" --backup_type "incremental" --base_id "xxx" ``` -### View backups +### View backup -#### Syntax structure +#### Grammar structure ``` mo_br list - -- ID - // To query the backup data. If the backup is on s3(oss minio), you need to specify the - --access_key_id - --secret_access_key - --not_check_data - --meta_path + --ID + //To query backup data. If the backup is on s3 (oss minio), you need to specify + --access_key_id + --secret_access_key + --not_check_data + --meta_path ``` **Parameter description** -| parameters | clarification | -| ---- | ---- | -| ID | ID of the backup| -|access_key_id| Access key ID of the backup to s3| -|secret_access_key| Secret access key for the backup to s3| -|not_check_data | View only the information in the meta. Does not check the backup data. The default without this parameter is that it will check the backed up files. Currently, it will only check if the backup file exists.| -|meta_path | Specifies the location of the meta file. If not specified, the default is the mo_br.meta file in the same directory.| +| Parameters | Description | +| ----| ----| +| ID | Backup ID| +|access_key_id| Access key ID of s3 backed up| +|secret_access_key| Secret access key of s3 backed up| +|not_check_data | Only check the information in meta. Do not view backup data. By default, without this parameter, the backed up files will be checked. Currently only the backed up files are checked for existence. | +|meta_path | Specifies the meta file location. If not specified, the default is the mo_br.meta file in the same directory. | -#### Examples +#### Example - View a list of all backups -``` +```bash ./mo_br list +--------------------------------------+--------+--------------------------------+---------------------------+--------------+---------------------------+ | ID | SIZE | PATH | AT TIME | DURATION | COMPLETE TIME | @@ -155,9 +156,9 @@ mo_br list +--------------------------------------+--------+--------------------------------+---------------------------+--------------+---------------------------+ ``` -- View the list of backups with the specified ID, the ID determined by list detects the backed up files. +- View the backup list of the specified ID. When the ID is determined by list, the backed up file will be detected. -``` +```bash ./mo_br list 4d21b228-10dd-11ef-9497-26dd28356ef2 +--------------------------------------+--------+--------------------------------+---------------------------+--------------+---------------------------+ | ID | SIZE | PATH | AT TIME | DURATION | COMPLETE TIME | @@ -178,14 +179,14 @@ check: config/tn.toml_018f70d1-310e-78fc-ac96-aa5e06981bd7 ... ``` -### Delete Backup +### Delete backup -#### Syntax structure +#### Grammar structure ``` -mo_br delete ID - //To delete backup data. If the backup is on s3 (oss minio), you need to specify the - --access_key_id +mo_br delete ID + //To delete the backup data. If the backup is on s3 (oss minio), you need to specify + --access_key_id --secret_access_key --not_delete_data --meta_path @@ -193,89 +194,89 @@ mo_br delete ID **Parameter description** -| parameters | clarification | -| ---- | ---- | -| ID | ID of the backup to be deleted| -|access_key_id| Access key ID of the backup to s3| -|secret_access_key| Secret access key for the backup to s3| -|not_delete_data|Only the information in the meta is deleted. Backup data is not deleted.| -|meta_path|Specifies the location of the meta file. If not specified, the default is the mo_br.meta file in the same directory.| +| Parameters | Description | +| ----| ----| +| ID | ID of the backup to be deleted | +|access_key_id| Access key ID of s3 backed up| +|secret_access_key| Secret access key of s3 backed up| +|not_delete_data|Only delete the information in meta. Backup data is not deleted. | +|meta_path|Specifies the meta file location. If not specified, the default is the mo_br.meta file in the same directory. | -#### Examples +#### Example -- Delete Local File System Backup - -``` +- Delete local file system backup + +```bash ./mo_br delete e4cade26-3139-11ee-8631-acde48001122 ``` -- Delete a backup on minio. +- Delete a backup on the minio. -``` +```bash ./mo_br delete e4cade26-3139-11ee-8631-acde48001122 --access_key_id "S0kwLuB4JofVEIAxWTqf" --secret_access_key "X24O7t3hccmqUZqvqvmLN8464E2Nbr0DWOu9Qs5A" ``` -### Restore Backup +### Restore backup -#### Syntax structure +#### Grammar structure -- Restore a backup with the specified ID +- Restore backup of specified ID ``` mo_br restore ID - //Reads the backup data with the specified ID. If the backup is on s3(oss minio), you need to specify the + //Read the backup data of the specified ID. If the backup is on s3 (oss minio), you need to specify --backup_access_key_id - --backup_secret_access_key - - //Destination path to restore restore_directory - --restore_dir s3|filesystem - //s3 - --restore_endpoint - --restore_access_key_id - --restore_secret_access_key - --restore_bucket - --restore_filepath - --restore_region - --restore_compression - --restore_role_arn + --backup_secret_access_key + + //Restore target path restore_directory + --restore_dir s3|filesystem + //s3 + --restore_endpoint + --restore_access_key_id + --restore_secret_access_key + --restore_bucket + --restore_filepath + --restore_region + --restore_compression + --restore_role_arn --restore_is_minio //filesystem - --restore_path - --dn_config_path - --meta_path + --restore_path + --dn_config_path + --meta_path --checksum - --parallelism + --parallelism ``` **Parameter description** -| parameters | clarification | -| ---- | ---- | -|ID | ID to be recovered| -|backup_access_key_id|Access key ID of the backup in s3| -|backup_secret_access_key |Secret access key backed up in s3| -|restore_dir | Destination path type for recovery. Use when specifying the destination path for recovery. s3|filesystem| -|restore_endpoint| Connect to the URL to restore to the S3 service| -|restore_access_key_id| Access key ID restored to s3| -|restore_secret_access_key| Recover the secret access key to s3| -|restore_bucket| Recover to the bucket that s3 needs to access| -|restore_filepath|Relative file path to restore to s3| -|restore_region| Restore the object storage service area to s3| -|restore_compression|The compressed format of the S3 file restored to s3.| -|restore_role_arn| Resource name of the role restored to s3.| -|restore_is_minio|Specifies whether the recovered s3 is a minio| +| Parameters | Description | +| ------------ | ----| +|ID | ID to restore| +|backup_access_key_id|Access key ID backed up in s3| +|backup_secret_access_key |Backup Secret access key in s3| +|restore_dir | Restore target path type. Used when specifying the destination path for recovery. s3|filesystem| +|restore_endpoint| URL to connect to restore to S3 service| +|restore_access_key_id| Restore Access key ID to s3| +|restore_secret_access_key| Restore Secret access key to s3| +|restore_bucket| The bucket that needs to be accessed to restore to s3| +|restore_filepath|Restore relative file path to s3| +|restore_region| Object storage service region restored to s3| +|restore_compression|Restore the compression format of S3 files to s3. | +|restore_role_arn| The resource name of the role restored to s3. | +|restore_is_minio|Specifies whether the restored s3 is minio| |restore_path|Restore to local path| -|dn_config_path| dn Configuration path| -|meta_path|Specifies the location of the meta file. It can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory.| -|checksum |Parallelism of tae file replication during recovery, default is 1| -|parallelism|parallelism| +|dn_config_path| dn configuration path| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | +|checksum |The parallelism of tae file copy during recovery, the default is 1| +|parallelism|Parallelism| - Do not specify a restore backup ID ``` -//Recovery +//recover. mo_br restore - --backup_dir s3|filesystem Destination path type for backups. Used when specifying the destination path for backups. + --backup_dir s3|filesystem The destination path type for the backup. Used when specifying the destination path for backup. //s3 --backup_endpoint --backup_access_key_id @@ -288,8 +289,8 @@ mo_br restore --backup_is_minio //filesystem --backup_path - //Destination path to restore restore_directory - --restore_dir s3|filesystem //Destination path type for recovery. Used when specifying the destination path for recovery. + //Recovery target path, restore_directory + --restore_dir s3|filesystem,The target path type for recovery. Used when specifying the destination path for recovery. //s3 --restore_endpoint --restore_access_key_id @@ -310,76 +311,76 @@ mo_br restore **Parameter description** -| parameters | clarification | -| ---- | ---- | -|backup_dir | Destination path type for recovery. Use when specifying the destination path for recovery. s3|filesystem| -|backup_endpoint| Connect to the URL of the backup on s3 | -|backup_access_key_id| Access key ID of the backup in s3| -|backup_secret_access_key| 备份在 s3 的 Secret access key| -|backup_bucket| Backup bucket in s3| -|backup_filepath| Relative file paths for backups in s3| -|backup_region| Backup service area in s3| -|backup_compression| The compressed format of the files backed up in s3.| -|backup_role_arn| The resource name of the role backed up in s3.| -|backup_is_minio| Specifies whether the backup s3 is minio| -|backup_path| Path to local backup| -|restore_dir | The type of destination path to recover. To specify the destination path for recovery, use. s3 or filesystem| -|restore_endpoint| Connect to the URL to restore to the S3 service| -|restore_access_key_id| Access key ID restored to s3| -|restore_secret_access_key| Recover the secret access key to s3| -|restore_bucket| Recover to the bucket that s3 needs to access| -|restore_filepath|Relative file path to restore to s3| -|restore_region| Restore the object storage service area to s3| -|restore_compression|The compressed format of the S3 file restored to s3.| -|restore_role_arn| Resource name of the role restored to s3.| -|restore_is_minio|Specifies whether the recovered s3 is a minio| -|restore_path|Restore the path to the local matrixone| -|dn_config_path| dn Configuration path| -|meta_path|Specifies the location of the meta file. It can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory.| -|checksum |Parallelism of tae file replication during recovery, default is 1| -|parallelism|parallelism| - -#### Examples - -Restore from File System to File System - -**Step one:** Stop mo, delete mo-data - -**Step Two:** Execute the following recovery command +| Parameters | Description | +| ----| ----| +|backup_dir | The destination path type for recovery. Used when specifying the destination path for recovery. s3|filesystem| +|backup_endpoint| URL to connect to backup in s3| +|backup_access_key_id| Access key ID backed up in s3| +|backup_secret_access_key| Secret access key backed up in s3| +|backup_bucket| Bucket backed up in s3| +|backup_filepath| Relative file path of backup in s3| +|backup_region| Service region backed up in s3| +|backup_compression| Compression format for files backed up in s3. | +|backup_role_arn| Resource name of the role backed up in s3. | +|backup_is_minio| specifies whether the backed up s3 is minio| +|backup_path| The path of local backup| +|restore_dir | Restore target path type. Used when specifying the destination path for recovery. s3 or filesystem| +|restore_endpoint| URL to connect to restore to S3 service| +|restore_access_key_id| Restore Access key ID to s3| +|restore_secret_access_key| Restore Secret access key to s3| +|restore_bucket| The bucket that needs to be accessed to restore to s3| +|restore_filepath|Restore relative file path to s3| +|restore_region| Object storage service region restored to s3| +|restore_compression|Restore the compression format of S3 files to s3. | +|restore_role_arn| The resource name of the role restored to s3. | +|restore_is_minio|Specifies whether the restored s3 is minio| +|restore_path|Restore the path to local matrixone| +|dn_config_path| dn configuration path| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | +|checksum |The parallelism of tae file copy during recovery, the default is 1| +|parallelism|Parallelism| + +#### Example + +Restore from file system to file system + +**Step 1:**Stop mo and delete mo-data + +**Step 2:**Execute the following recovery command ``` ./mo_br restore fb26fd88-41bc-11ee-93f8-acde48001122 --restore_dir filesystem --restore_path "your_mopath" ``` -After recovery a new mo-data file is generated at matrixone +After recovery, a new mo-data file will be generated in matrixone -**Step three:** Start mo +**Step 3:**Start mo -### Verify the check code for the backup +### Verify backup verification code -Read each file in the backup folder and its sha256 file. Calculates the sha256 value of the file and compares it to the sha256 file value. The sha256 file is created when the file is created or updated. +Read each file in the backup folder and its sha256 file. Calculate the sha256 value of the file and compare it to the sha256 file value. sha256 files are created when files are created or updated. -#### Syntax structure +#### Grammar structure -- Verify a backup of an ID +- Verify backup of an ID -``` +```bash mo_br check ID - //Checks the backup data for the specified ID. If the backup is on s3 (oss minio), you need to specify + //Verify the backup data of the specified ID. If the backup is on s3 (oss minio), you need to specify --backup_access_key_id string --backup_secret_access_key string - --meta_path string //Specifies the meta file location. If not specified, the default is the mo_br.meta file in the same directory. + --meta_path string 指定meta文件位置.如果不指定,默认是同一目录下的mo_br.meta文件. ``` **Parameter description** -| parameters | clarification | -| ---- | ---- | -|backup_access_key_id| Access key ID of the backup in s3| +| Parameters | Description | +| ----| ----| +|backup_access_key_id| Access key ID backed up in s3| |backup_secret_access_key| Secret access key backed up in s3| -|meta_path|Specifies the location of the meta file. It can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory.| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | -- Verify the backup, specifying the path to the backup +- Verify backup, specify backup path ``` mo_br check @@ -401,26 +402,26 @@ mo_br check **Parameter description** -| parameters | clarification | -| ---- | ---- | -|backup_dir | The type of path where the backup is located, which must be specified if no ID is specified. s3 or filesystem| -|backup_endpoint| Connect to the URL of the backup on s3| -|backup_access_key_id| Access key ID of the backup in s3| +| Parameters | Description | +| ----| ----| +|backup_dir | The path type where the backup is located. It must be specified when the ID is not specified. s3 or filesystem| +|backup_endpoint| URL to connect to backup in s3| +|backup_access_key_id| Access key ID backed up in s3| |backup_secret_access_key| Secret access key backed up in s3| -|backup_bucket| Backup bucket in s3| -|backup_filepath| Relative file paths for backups in s3| -|backup_region| Backup service area in s3| -|backup_compression| The compressed format of the files backed up in s3.| -|backup_role_arn| The resource name of the role backed up in s3.| -|backup_is_minio| Specifies whether the backup s3 is minio| -|backup_path| Path to local backup| -|meta_path|Specifies the location of the meta file. It can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory.| +|backup_bucket| Bucket backed up in s3| +|backup_filepath| Relative file path of backup in s3| +|backup_region| Service region backed up in s3| +|backup_compression| Compression format for files backed up in s3. | +|backup_role_arn| Resource name of the role backed up in s3. | +|backup_is_minio| specifies whether the backed up s3 is minio| +|backup_path| The path of local backup| +|meta_path|Specifies the meta file location. Can only be a path in the file system. If not specified, the default is the mo_br.meta file in the same directory. | -#### Examples +#### Example -- Verify a backup of an ID +- Verify backup of an ID -``` +```bash ./mo_br check 1614f462-126c-11ef-9af3-26dd28356ef3 +--------------------------------------+--------+-----------------------------------+---------------------------+---------------+---------------------------+ | ID | SIZE | PATH | AT TIME | DURATION | COMPLETE TIME | @@ -442,7 +443,7 @@ check: config/launch.toml_018f7a50-d300-7017-8580-150edf08733e - Verify backups in a backup directory -``` +```bash (base) admin@admindeMacBook-Pro mo-backup % ./mo_br check --backup_dir filesystem --backup_path /Users/admin/soft/incbackup/back2 2024/05/15 11:40:30.011160 +0800 INFO malloc/malloc.go:42 malloc {"max buffer size": 1073741824, "num shards": 16, "classes": 23, "min class size": 128, "max class size": 1048576, "buffer objects per class": 23} check: /backup_meta @@ -455,11 +456,11 @@ check: config/log.toml_018f7a50-d30c-7ed0-85bc-191e9f1eb753 ... ``` -## snapshot backup +## Snapshot backup -### Create a snapshot +### Create snapshot -#### Syntax structure +#### Grammar structure ``` mo_br snapshot create @@ -475,46 +476,52 @@ mo_br snapshot create **Parameter description** | Parameters | Description | -| ---- | ---- | -|host | Target MatrixOne's IP| -|port|port number| +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| |user | user| |password | User's password| -|level | Scope of snapshot backup, only account is supported for the time being| -|account| Tenant object name for snapshot backups| -|sname | Snapshot name| +|level | snapshot backup scope, account|cluster| +|account| The tenant object name of the snapshot backup. This parameter does not need to be filled in at the cluster level| +|sname | snapshot name| -#### Examples +#### Example -- To create a snapshot for system tenant sys: +- The cluster administrator creates a cluster-level snapshot: -``` -./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --level "account" --sname "snapshot_01" --account "sys" +```bash +./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "root" --password "111" --level "cluster" --sname "cluster_sp1" ``` -- System tenant creates snapshot for normal tenant acc1: +- The system tenant administrator creates a tenant-level snapshot for the system tenant sys: +```bash +./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "root" --password "111" --level "account" --sname "snapshot_01" --account "sys" ``` - ./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --level "account" --sname "snapshot_02" --account "acc1" + +- The system tenant administrator creates a tenant-level snapshot for the common tenant acc1: + +```bash + ./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "root" --password "111" --level "account" --sname "snapshot_02" --account "acc1" ``` -- Normal tenant creation snapshot: +- A normal tenant administrator creates a tenant-level snapshot: - - Create Normal Tenant acc1 + - Create a common tenant acc1 ```sql create account acc1 admin_name admin IDENTIFIED BY '111'; ``` - - acc1 Create snapshot + - acc1 creates snapshot ``` ./mo_br snapshot create --host "127.0.0.1" --port 6001 --user "acc1#admin" --password "111" --level "account" --account "acc1" --sname "snapshot_03" ``` -### View snapshots +### View snapshot -#### Syntax structure +#### Grammar structure ``` mo_br snapshot show @@ -522,6 +529,7 @@ mo_br snapshot show --port --user --password + --cluster --account --db --table @@ -533,48 +541,49 @@ mo_br snapshot show **Parameter description** | Parameters | Description | -| ---- | ---- | -|host | Target MatrixOne's IP| -|port|port number| -|user | subscribers| +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| +|user | user| |password | User's password| +|cluster|Fixed to fill in sys, other values ​​​​do not take effect, only used by sys administrators| |account| Tenant name to filter, for sys administrators only| -|db | The name of the database to be filtered| -|table | Table name to filter| -|sname | Name of the snapshot to filter| -|beginTs |Start time of the snapshot timestamp to be filtered| -|endTs | The end time of the snapshot timestamp to be filtered| +|db | Database name to filter | +|table | table name to filter| +|sname | Snapshot name to filter | +|beginTs |Start time of snapshot timestamp to filter| +|endTs | The end time of the snapshot timestamp to be filtered | -#### Examples +#### Example -- To view snapshots created under System Tenants: +- View snapshots created under the system tenant administrator: -``` -./mo_br snapshot show --host "127.0.0.1" --port 6001 --user "dump" --password "111" +```bash +./mo_br snapshot show --host "127.0.0.1" --port 6001 --user "root" --password "111" SNAPSHOT NAME TIMESTAMP SNAPSHOT LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME snapshot_02 2024-05-11 02:29:23.07401 account acc1 snapshot_01 2024-05-11 02:26:03.462462 account sys ``` -- View the snapshot created under acc1: +- View snapshots created under acc1 by normal tenant administrators: -``` +```bash ./mo_br snapshot show --host "127.0.0.1" --port 6001 --user "acc1#admin" --password "111" SNAPSHOT NAME TIMESTAMP SNAPSHOT LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME snapshot_03 2024-05-11 02:29:31.572512 account acc1 ``` -- View the snapshot created for tenant acc1 under System Tenant and filter the start time: +- View the snapshot created for tenant acc1 under the system tenant administrator and filter the start time: -``` -./mo_br snapshot show --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "acc1" --beginTs "2024-05-11 00:00:00" +```bash +./mo_br snapshot show --host "127.0.0.1" --port 6001 --user "root" --password "111" --account "acc1" --beginTs "2024-05-11 00:00:00" SNAPSHOT NAME TIMESTAMP SNAPSHOT LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME snapshot_02 2024-05-11 02:29:23.07401 account acc1 ``` -### Delete Snapshot +### Delete snapshot -#### Syntax structure +#### Grammar structure ``` mo_br snapshot drop @@ -588,30 +597,30 @@ mo_br snapshot drop **Parameter description** | Parameters | Description | -| ---- | ---- | -|host | Target MatrixOne's IP| -|port|port number| -|user | subscribers| +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| +|user | user| |password | User's password| -|sname | Name of the snapshot to filter| +|sname | Snapshot name to filter | -#### Examples +#### Example -- To delete a snapshot created by a system tenant: +- Delete snapshots created by the system administrator: -``` -./mo_br snapshot drop --host "127.0.0.1" --port 6001 --user "dump" --password "111" --sname "snapshot_01" +```bash +./mo_br snapshot drop --host "127.0.0.1" --port 6001 --user "root" --password "111" --sname "snapshot_01" ``` -- To delete a snapshot created by a normal tenant: +- Delete snapshots created by normal tenant administrators: -``` +```bash ./mo_br snapshot drop --host "127.0.0.1" --port 6001 --user "acc1#admin" --password "111" --sname "snapshot_03" ``` -### Restoring a snapshot +### Restore snapshot -#### Syntax structure +#### Grammar structure ``` mo_br snapshot restore @@ -631,43 +640,387 @@ mo_br snapshot restore **Parameter description** | Parameters | Description | -| ---- | ---- | -|host | Target MatrixOne's IP| -|port|port number| -|user | subscribers| +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| +|user | user| +|password | User's password| +|sname | The name of the snapshot to be restored | +|account| Tenant name to restore, for sys administrators only | +|db | The name of the database to be restored | +|table |The name of the table to be restored| +|newaccount | Newly created tenant name | +|newaccountadmin | Tenant Administrator| +|newaccountpwd | Tenant administrator password | + +__NOTE__: Only the system tenant can perform restore data to a new tenant, and only tenant-level restores are allowed. + +#### Example + +- Table level recovery + +```bash +./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "root" --password "111" --account "sys" --db "snapshot_read" --table "test_snapshot_read" --sname "sp_01" +``` + +- Database level recovery + +```bash +./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "root" --password "111" --account "sys" --db "snapshot_read" --sname "sp_02" +``` + +- Tenant level restore to this tenant + +```bash +./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "root" --password "111" --account "sys" --sname "sp_03" +``` + +- Tenant level revert to new tenant + +```bash +./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "root" --password "111" --account "sys" --sname "sp_03" --new_account "acc2" --new_admin_name "admin" --new_admin_password "111"; +``` + +- Cluster level recovery + +```bash +./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "root" --password "111" --sname "cluster_sp1" +``` + +## PITR backup + +### Create PITR + +#### Grammar structure + +``` +mo_br pitr create + --host + --port + --user + --password + --pname + --level + --account + --database + --table + --rangevalue + --rangeunit +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| +|user | user| +|password | User's password| +|pname | pitr name| +|level | Backup scope, cluster | account | database | table, when level is equal to the high level, the low-level object name does not need to be filled in and cannot be filled in. For example, level = account, then database and table cannot be filled in | +|account| Backup tenant name| +|database| Backup database name| +|table| Backup table name| +|rangevalue|Time range value, 1-100| +|rangeunit| Time range unit, optional range h (hour), d (day, default), mo (month), y (year) | + +#### Example + +- Cluster level + +Only system tenants can create cluster-level pitrs. + +``` +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "root" --password "111" --pname "pitr01" --level "cluster" --rangevalue 10 --rangeunit "h" +``` + +- Tenant level + +System tenants can create tenant-level pitrs for themselves and other tenants. + +```bash +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr02" --level "account" --account "sys" --rangevalue 1 --rangeunit "d" + +mo create account acc01 admin_name = 'test_account' identified by '111'; +mo create account acc02 admin_name = 'test_account' identified by '111'; + +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr015" --level "account" --account "acc01" --rangevalue 1 --rangeunit "y" + +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr016" --level "account" --account "acc02" --rangevalue 1 --rangeunit "y" +``` + +Ordinary tenants can only create tenant-level pitrs for themselves. + +```bash +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr07" --level "account" --account "acc01" --rangevalue 1 --rangeunit "h" +``` + +- database level + +```bash +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr011" --level "database" --account "sys" --database "abc" --rangevalue 1 --rangeunit "y" +``` + +- table level + +```bash +./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr011" --level "database" --account "sys" --database "abc" --rangevalue 1 --rangeunit "y" +``` + +### View PITR + +#### Grammar structure + +``` +mo_br pitr show + --hostname + --port + --user + --password + --cluster + --account + --database + --table +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| +|user | user| |password | User's password| -|sname | Name of the snapshot to be restored| -|account| The name of the tenant to restore, for sys administrators only| -|db | Name of the database to be recovered| -|table | The name of the table to be recovered| -|newaccount | Newly created tenant name| -|newaccountadmin | tenant manager| -|newaccountpwd | Tenant Administrator Password| +|pname | pitr name| +|cluster | Fixed filling in sys, other values ​​​​do not take effect, only used by sys administrators | +|account| Backup tenant name| +|database| Backup database name| +|table| Backup table name| + +#### Example + +- View all pitr + +```bash +./mo_br pitr show --host "127.0.0.1" --port 6001 --user "dump" --password "111" +PITR NAME CREATED TIME MODIFIED TIME PITR LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME PITR LENGTH PITR UNIT +pitr016 2024-08-08 06:55:40 2024-08-08 06:55:40 account acc02 * * 1 y +pitr015 2024-08-08 06:55:04 2024-08-08 06:55:04 account acc01 * * 1 y +pitr012 2024-08-08 06:52:30 2024-08-08 06:52:30 table sys abc test 1 y +pitr011 2024-08-08 06:50:43 2024-08-08 06:50:43 database sys abc * 1 y +pitr05 2024-08-08 06:45:56 2024-08-08 06:45:56 account sys * * 1 y +pitr04 2024-08-08 06:45:52 2024-08-08 06:45:52 account sys * * 1 mo +pitr03 2024-08-08 06:45:42 2024-08-08 06:45:42 account sys * * 1 d +pitr02 2024-08-08 06:45:25 2024-08-08 06:45:25 account sys * * 1 h +pitr01 2024-08-08 06:32:31 2024-08-08 06:32:31 cluster * * * 10 h +``` -__NOTE__: Only system tenants can perform restore data to a new tenant, and only tenant-level restores are allowed. +- View cluster level pitr -#### Examples +```bash +./mo_br pitr show --host "127.0.0.1" --port 6001 --user "dump" --password "111" --cluster "sys" +PITR NAME CREATED TIME MODIFIED TIME PITR LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME PITR LENGTH PITR UNIT +pitr01 2024-08-08 06:32:31 2024-08-08 06:32:31 cluster * * * 10 h +``` -- Table level restored to this tenant +- View tenant level pitr + +```bash +./mo_br pitr show --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" +PITR NAME CREATED TIME MODIFIED TIME PITR LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME PITR LENGTH PITR UNIT +pitr012 2024-08-08 06:52:30 2024-08-08 06:52:30 table sys abc test 1 y +pitr011 2024-08-08 06:50:43 2024-08-08 06:50:43 database sys abc * 1 y +pitr05 2024-08-08 06:45:56 2024-08-08 06:45:56 account sys * * 1 y +pitr04 2024-08-08 06:45:52 2024-08-08 06:45:52 account sys * * 1 mo +pitr03 2024-08-08 06:45:42 2024-08-08 06:45:42 account sys * * 1 d +pitr02 2024-08-08 06:45:25 2024-08-08 06:45:25 account sys * * 1 h +``` +- View database level pitr + +```bash +./mo_br pitr show --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --database "abc" +PITR NAME CREATED TIME MODIFIED TIME PITR LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME PITR LENGTH PITR UNIT +pitr012 2024-08-08 06:52:30 2024-08-08 06:52:30 table sys abc test 1 y +pitr011 2024-08-08 06:50:43 2024-08-08 06:50:43 database sys abc * 1 y ``` -./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --db "snapshot_read" --table "test_snapshot_read" --sname "sp_01" + +- View table level pitr + +```bash + ./mo_br pitr show --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --database "abc" --table "test" +PITR NAME CREATED TIME MODIFIED TIME PITR LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME PITR LENGTH PITR UNIT +pitr012 2024-08-08 06:52:30 2024-08-08 06:52:30 table sys abc test 1 y ``` -- Database level restore to this tenant +### Change PITR + +#### Grammar structure ``` -./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --db "snapshot_read" --sname "sp_02" +mo_br pitr alter + --host + --port + --user + --password + --pname + --rangevalue + --rangeunit +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| +|user | user| +|password | User's password| +|pname | pitr name| +|rangevalue|Time range value, 1-100| +|rangeunit| Time range unit, optional range h (hour), d (day, default), mo (month), y (year) | + +#### Example + +```bash +./mo_br pitr alter --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr01" --rangevalue 10 --rangeunit "d" + ./mo_br pitr show --host "127.0.0.1" --port 6001 --user "dump" --password "111" --cluster "sys" +PITR NAME CREATED TIME MODIFIED TIME PITR LEVEL ACCOUNT NAME DATABASE NAME TABLE NAME PITR LENGTH PITR UNIT +pitr01 2024-08-08 06:32:31 2024-08-08 07:31:06 cluster * * * 10 d ``` -- Tenant level restored to this tenant +### Restore PITR + +#### Grammar structure ``` -./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --sname "sp_03" +mo_br pitr restore + --host string cluster IP + --port int cluster port number + --user string username + --password user password + //Object name to be restored + --cluster restores the entire cluster, only for cluster administrators. Fixed filling in sys. After filling in this field, only need to fill in --timestamp + --account Tenant name to filter, only for cluster administrators + --database The name of the database to filter + --table The name of the table to filter + //Time to restore + --timestamp + //Create and restore to new tenant + --new_account string Newly created tenant name + --new_admin_name string tenant administrator + --new_admin_password string Tenant administrator password ``` -- Tenant level restored to new tenant +**Parameter description** + +| Parameters | Description | +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| +|user | user| +|password | User's password| +|cluster | Restore the entire cluster, only for cluster administrators. Fixed filling in sys. After filling in this field, just fill in --timestamp| +|account | Tenant name to filter, only for cluster administrators | +|database | Database name to filter | +|table | table name to filter| +|timestamp | time to restore | +|new_account |Time to restore| +|new_admin_name | Create and restore to new tenant, newly created tenant name| +|new_admin_password | Tenant Administrator | +|timestamp | Tenant administrator password | + +#### example + +- system tenant + + ```bash + #cluster + ./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr01" --level "cluster" --rangevalue 10 --rangeunit "h" + + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr01" --cluster "sys" --timestamp "2024-08-08 15:42:20.249966" + + #tenent + ./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr100" --level "account" --account "sys" --rangevalue 10 --rangeunit "h" + ##Use account pitr to restore account + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr100" --timestamp "2024-08-08 15:47:15.216472" --account "sys" + ##Use account pitr to restore db + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr100" --timestamp "2024-08-08 15:47:15.216472" --account "sys" --database "abc" + ##Use account pitr to restore table + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr100" --timestamp "2024-08-08 15:47:15.216472" --account "sys" --database "abc" --table "test" + + #Database level + ./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr101" --level "database" --account "sys" --database "abc" --rangevalue 10 --rangeunit "h" + ##Use db pitr to restore db + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr101" --timestamp "2024-08-08 15:56:18.610295" --account "sys" --database "abc" + ##Use db pitr to restore table + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr101" --timestamp "2024-08-08 15:56:18.610295" --account "sys" --database "abc" --table "test" + + #Table level + ./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr102" --level "table" --account "sys" --database "abc" --table "test" --rangevalue 10 --rangeunit "h" + ##用 table pitr 恢复 table + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr102" --timestamp "2024-08-08 16:00:41.433477" --account "sys" --database "abc" --table "test" + ``` + +- Ordinary tenant + + ```bash + #Tenant level + ./mo_br pitr create --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr200" --level "account" --account "acc01" --rangevalue 10 --rangeunit "h" + ##Use account pitr to restore account + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr200" --timestamp "2024-08-08 16:04:17.276521" --account "acc01" + ##Restore db using account pitr + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr200" --timestamp "2024-08-08 16:04:17.276521" --account "acc01" --database "abc" + ##Use account pitr to restore the table + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr200" --timestamp "2024-08-08 16:04:17.276521" --account "acc01" --database "abc" --table "test" + + #database level + ./mo_br pitr create --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr201" --level "database" --account "acc01" --database "abc" --rangevalue 10 --rangeunit "h" + ##Restore db using db pitr + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr201" --timestamp "2024-08-08 16:06:50.374948" --account "acc01" --database "abc" + ##Use db pitr to restore table + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr201" --timestamp "2024-08-08 16:06:50.374948" --account "acc01" --database "abc" --table "test" + + #Table level + ./mo_br pitr create --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr202" --level "table" --account "acc01" --database "abc" --table "test" --rangevalue 10 --rangeunit "h" + ##Restoring table using table pitr + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "acc01#test_account" --password "111" --pname "pitr202" --timestamp "2024-08-08 16:06:50.374948" --account "acc01" --database "abc" --table "test" + + #The system tenant creates a pitr for the ordinary tenant and restores the ordinary tenant + + ./mo_br pitr create --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr300" --level "account" --account "acc01" --rangevalue 1 --rangeunit "y" + + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr300" --timestamp "2024-08-08 16:09:17.035136" --account "acc01" + + #System tenant restores normal tenant to new tenant + ./mo_br pitr restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr300" --timestamp "2024-08-08 16:09:17.035136" --account "acc03" --new_account "acc03" --new_admin_name "test_account" --new_admin_password "111" + ``` + +### Delete PITR + +#### Grammar structure ``` -./mo_br snapshot restore --host "127.0.0.1" --port 6001 --user "dump" --password "111" --account "sys" --sname "sp_03" --new_account "acc2" --new_admin_name "admin" --new_admin_password "111"; +mo_br pitr drop + --host + --port + --user + --password + --pname +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|host | IP of target MatrixOne| +|port|Port number| +|user | user| +|password | User's password| +|pname | pitr name| + +#### 示例 + +```bash +./mo_br pitr drop --host "127.0.0.1" --port 6001 --user "dump" --password "111" --pname "pitr01" ``` \ No newline at end of file diff --git a/docs/MatrixOne/Maintain/backup-restore/modump-backup-restore.md b/docs/MatrixOne/Maintain/backup-restore/modump-backup-restore.md index 24278e742..0bbca3cb3 100644 --- a/docs/MatrixOne/Maintain/backup-restore/modump-backup-restore.md +++ b/docs/MatrixOne/Maintain/backup-restore/modump-backup-restore.md @@ -1,38 +1,38 @@ -# mo-dump Backup and Recovery +# mo-dump backup and recovery -For businesses that produce a lot of data every day, it's important to back up the database. In case of system crash or hardware failure, or user misoperation, you can recover data and restart the system without data loss. +For enterprises, a large amount of data is generated every day, so database backup is very important. In the event of a system crash, hardware failure, or user misoperation, you can restore data and restart the system without causing data loss. -In addition, data backups serve as safeguards before upgrading a MatrixOne installation, while data backups can also be used to transfer a MatrixOne installation to another system. +In addition, data backup also serves as a safeguard before upgrading the MatrixOne installation, and data backup can also be used to transfer the MatrixOne installation to another system. -MatrixOne supports logical backups via the `mo-dump` utility. `modump` is a command-line utility that generates logical backups of MatrixOne databases. It generates SQL statements that can be used to recreate database objects and data. You can find its syntax description and usage guide in the [mo-dump](../../Develop/export-data/modump.md) chapter. +MatrixOne supports logical backup via the `mo-dump` utility. `modump` is a command line utility for generating logical backups of MatrixOne databases. It generates SQL statements that can be used to recreate database objects and data. You can find its syntax description and usage guidelines in the [mo-dump](../../Develop/export-data/modump.md) chapter. -We'll walk through a simple example of how to use the `mo-dump` utility to complete the data backup and restore process. +We will use a simple example to describe how to use the `mo-dump` utility to complete the data backup and restore process. ## Steps -### 1. Deployment of mo-dump +### 1. Deploy mo-dump -See the [mo-dump tool writing](../../Develop/export-data/modump.md) chapter to complete the deployment of `the mo-dump` tool. +See the [mo-dump tool writing](../../Develop/export-data/modump.md) chapter to complete the deployment of the `mo-dump` tool. ### 2. Generate a backup of a single database -An example is the database *t* and its table *t1* created using the following SQL: +The example is as follows, database *t*and its table *t1*created using the following SQL: -``` +```sql DROP DATABASE IF EXISTS `t`; CREATE DATABASE `t`; USE `t`; create table t1 ( - c1 int primary key auto_increment, - c2 tinyint not null default 4, - c3 smallint, - c4 bigint, - c5 tinyint unsigned, - c6 smallint unsigned, - c7 int unsigned, - c8 bigint unsigned, - c9 float, + c1 int primary key auto_increment, + c2 tinyint not null default 4, + c3 smallint, + c4 bigint, + c5 tinyint unsigned, + c6 smallint unsigned, + c7 int unsigned, + c8 bigint unsigned, + c9 float, c10 double, c11 date, c12 datetime, @@ -45,38 +45,38 @@ create table t1 c19 blob, c20 uuid ); -insert into t1 values (1, 1, 1, 1, 1, 1, 1, 1, 1, 1, '2019-01-01', '2019-01-01 00:00:00', '2019-01-01 00:00:00', 'a', 'a', '{"a":1}','1212.1212', 'a', 'aza', '00000000-0000-0000-0000-000000000000'); +insert into t1 values ​​(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, '2019-01-01', '2019-01-01 00:00:00', '2019-01 -01 00:00:00', 'a', 'a', '{"a":1}','1212.1212', 'a', 'aza', '00000000-0000-0000-0000-000000000000' ); ``` -If you want to generate a backup of a single database, you can run the following command. This command will generate a backup of the database named *t* with the structure and data in the *t.sql* file. +If you want to generate a backup of a single database, you can run the following command. This command will generate a backup of the database named *t*that contains the structure and data in the *t.sql*file. -``` +```bash ./mo-dump -u root -p 111 -h 127.0.0.1 -P 6001 -db t > t.sql ``` -If you want to generate a backup of a single table in the database, you can run the following command. This command generates a backup of the *t1* table of the database named *t*, which contains the structure and data in the *t.sql* file. +If you want to generate a backup of a single table in your database, you can run the following command. This command will generate a backup of the *t1*table for the database named *t*, containing the structure and data in the *t.sql*file. -``` +```bash ./mo-dump -u root -p 111 -db t -tbl t1 > t1.sql ``` !!! note - If you want to generate a backup of multiple databases/tables, you need to separate the database names/table names with `,` . + If you want to generate backups of multiple databases/tables, you need to separate the database names/table names with `,`. -### 3. Restore Backup to MatrixOne Server +### 3. Restore backup to MatrixOne server -Restoring an exported *sql* file to a MatrixOne database is relatively simple. To recover your database, you must first create an empty database and use the *MySQL client* to recover. +Restoring exported *sql*files to a MatrixOne database is relatively simple. To restore your database, you must first create an empty database and restore it using the *MySQL client*. -Connect MatrixOne to the same server as the MySQL client and make sure the exported *sql* file is also on the same server. +Connect MatrixOne to the same server as the MySQL client, and make sure the exported *sql*file is also on the same server. -``` +```sql mysql> create database t if not exists; mysql> source /YOUR_SQL_FILE_PATH/t.sql ``` -After successfully executing the above command, execute the following command to check if all objects were created on the named *t* database. +After successfully executing the above command, execute the following command to check whether all objects are created on the database named *t*. -``` +```sql mysql> use t; mysql> show tables; mysql> select count(*) from t1; diff --git a/docs/MatrixOne/Maintain/cdc/mo-mysql.md b/docs/MatrixOne/Maintain/cdc/mo-mysql.md new file mode 100644 index 000000000..2fca0d5e8 --- /dev/null +++ b/docs/MatrixOne/Maintain/cdc/mo-mysql.md @@ -0,0 +1,201 @@ +# MatrixOne to MySQL CDC function + +## Scene description + +An online retail company uses MatrixOne as the production database for its order management system to store order data. In order to support the real-time analysis needs of the business (such as order quantity, sales trends, customer behavior, etc.), order data needs to be synchronized from MatrixOne to the MySQL analysis database in real time for use by the data analysis team and business systems. Through the `mo_cdc` tool, real-time synchronization of order data can be efficiently achieved, allowing the analysis system to obtain the latest order information at any time. + +-Source database (production database): The `orders` table in MatrixOne contains order data and records the details of each order, including order ID, customer ID, order time, order amount and status. +-Target database (analysis database): `orders_backup` table in MySQL, used for real-time statistics and analysis of order information. Ensure that the sales team can grasp sales dynamics in real time. +-Synchronization requirements: Use `mo_cdc` to synchronize data in MatrixOne's `orders` table to MySQL's `orders_backup` table in real time to ensure that the analysis system data is consistent with the production system. + +## Operation process + +### Create table structure + +Ensure that the table structures in the source database MatrixOne and the target database MySQL are identical to facilitate seamless data synchronization. + +- `orders` table in MatrixOne: + + ```sql + CREATE TABLE source_db.orders ( + order_id INT PRIMARY KEY, + customer_id INT, + order_date DATETIME, + amount DECIMAL(10, 2), + status VARCHAR(20) + ); + INSERT INTO source_db.orders (order_id, customer_id, order_date, amount, status) VALUES + (1, 101, '2024-01-15 14:30:00', 99.99, 'Shipped'), + (2, 102, '2024-02-10 10:00:00', 149.50, 'Delivered'), + (3, 103, '2024-03-05 16:45:00', 75.00, 'Processing'), + (4, 104, '2024-04-20 09:15:00', 200.00, 'Shipped'), + (5, 105, '2024-05-12 14:00:00', 49.99, 'Delivered'); + ``` + +- `orders_backup` table in MySQL: + + ```sql + CREATE TABLE analytics_db.orders_backup ( + order_id INT PRIMARY KEY, + customer_id INT, + order_date DATETIME, + amount DECIMAL(10, 2), + status VARCHAR(20) + ); + ``` + +### Create `mo_cdc` synchronization task + +Create a synchronization task through the `mo_cdc` tool to push MatrixOne's order data to MySQL in real time. + +```bash +>./mo_cdc task create \ + --task-name "task1" \ + --source-uri "mysql://root:111@127.0.0.1:6001" \ + --sink-type "mysql" \ + --sink-uri "mysql://root:111@127.0.0.1:3306" \ + --tables "source_db.orders:analytics_db.orders_backup" \ + --level "account" \ + --account "sys" +``` + +View task status + +```bash +> ./mo_cdc task show \ + --task-name "task1" \ + --source-uri "mysql://root:111@127.0.0.1:6001" +[ + { + "task-id": "0192d76f-d89a-70b3-a60d-615c5f2fd33d", + "task-name": "task1", + "source-uri": "mysql://root:******@127.0.0.1:6001", + "sink-uri": "mysql://root:******@127.0.0.1:3306", + "state": "running", + "checkpoint": "{\n \"source_db.orders\": 2024-10-29 16:43:00.318404 +0800 CST,\n}", + "timestamp": "2024-10-29 16:43:01.299298 +0800 CST" + } +] +``` + +Connect to downstream mysql to view full data synchronization status + +```sql +mysql> select * from analytics_db.orders_backup; ++----------+-------------+---------------------+--------+------------+ +| order_id | customer_id | order_date | amount | status | ++----------+-------------+---------------------+--------+------------+ +| 1 | 101 | 2024-01-15 14:30:00 | 99.99 | Shipped | +| 2 | 102 | 2024-02-10 10:00:00 | 149.50 | Delivered | +| 3 | 103 | 2024-03-05 16:45:00 | 75.00 | Processing | +| 4 | 104 | 2024-04-20 09:15:00 | 200.00 | Shipped | +| 5 | 105 | 2024-05-12 14:00:00 | 49.99 | Delivered | ++----------+-------------+---------------------+--------+------------+ +5 rows in set (0.01 sec) +``` + +### Incremental synchronization task + +After the task is established, perform data change operations on the upstream MatrixOne + +```sql +INSERT INTO source_db.orders (order_id, customer_id, order_date, amount, status) VALUES +(6, 106, '2024-10-29 12:00:00', 150.00, 'New'); +DELETE FROM source_db.orders WHERE order_id = 6; +UPDATE source_db.orders SET status = 'Delivered' WHERE order_id = 4; + +mysql> select * from source_db.orders; ++----------+-------------+---------------------+--------+------------+ +| order_id | customer_id | order_date | amount | status | ++----------+-------------+---------------------+--------+------------+ +| 4 | 104 | 2024-04-20 09:15:00 | 200.00 | Delivered | +| 1 | 101 | 2024-01-15 14:30:00 | 99.99 | Shipped | +| 2 | 102 | 2024-02-10 10:00:00 | 149.50 | Delivered | +| 3 | 103 | 2024-03-05 16:45:00 | 75.00 | Processing | +| 5 | 105 | 2024-05-12 14:00:00 | 49.99 | Delivered | ++----------+-------------+---------------------+--------+------------+ +5 rows in set (0.00 sec) +``` + +Connect to downstream mysql to view incremental data synchronization status + +```sql +mysql> select * from analytics_db.orders_backup; ++----------+-------------+---------------------+--------+------------+ +| order_id | customer_id | order_date | amount | status | ++----------+-------------+---------------------+--------+------------+ +| 1 | 101 | 2024-01-15 14:30:00 | 99.99 | Shipped | +| 2 | 102 | 2024-02-10 10:00:00 | 149.50 | Delivered | +| 3 | 103 | 2024-03-05 16:45:00 | 75.00 | Processing | +| 4 | 104 | 2024-04-20 09:15:00 | 200.00 | Delivered | +| 5 | 105 | 2024-05-12 14:00:00 | 49.99 | Delivered | ++----------+-------------+---------------------+--------+------------+ +5 rows in set (0.00 sec) +``` + +### Resume upload from breakpoint + +Now the mission is interrupted due to an accident. + +```bash +> ./mo_cdc task pause \ + --task-name "task1" \ + --source-uri "mysql://root:111@127.0.0.1:6001" +``` + +During the task interruption, data continues to be inserted into the upstream MatrixOne. + +```sql +INSERT INTO source_db.orders (order_id, customer_id, order_date, amount, status) VALUES +(11, 111, '2024-06-15 08:30:00', 250.75, 'Processing'); +INSERT INTO source_db.orders (order_id, customer_id, order_date, amount, status) VALUES +(12, 112, '2024-07-22 15:45:00', 399.99, 'Shipped'); +INSERT INTO source_db.orders (order_id, customer_id, order_date, amount, status) VALUES +(13, 113, '2024-08-30 10:20:00', 599.99, 'Delivered'); + +mysql> select * from source_db.orders; ++----------+-------------+---------------------+--------+------------+ +| order_id | customer_id | order_date | amount | status | ++----------+-------------+---------------------+--------+------------+ +| 1 | 101 | 2024-01-15 14:30:00 | 99.99 | Shipped | +| 2 | 102 | 2024-02-10 10:00:00 | 149.50 | Delivered | +| 3 | 103 | 2024-03-05 16:45:00 | 75.00 | Processing | +| 4 | 104 | 2024-04-20 09:15:00 | 200.00 | Delivered | +| 5 | 105 | 2024-05-12 14:00:00 | 49.99 | Delivered | +| 11 | 111 | 2024-06-15 08:30:00 | 250.75 | Processing | +| 12 | 112 | 2024-07-22 15:45:00 | 399.99 | Shipped | +| 13 | 113 | 2024-08-30 10:20:00 | 599.99 | Delivered | ++----------+-------------+---------------------+--------+------------+ +8 rows in set (0.01 sec) +``` + +Manual recovery tasks. + +```bash +> ./mo_cdc task resume \ + --task-name "task1" \ + --source-uri "mysql://root:111@127.0.0.1:6001" +``` + +Connect to the downstream mysql to check the resumed transmission status. + +```sql +mysql> select * from analytics_db.orders_backup; ++----------+-------------+---------------------+--------+------------+ +| order_id | customer_id | order_date | amount | status | ++----------+-------------+---------------------+--------+------------+ +| 1 | 101 | 2024-01-15 14:30:00 | 99.99 | Shipped | +| 2 | 102 | 2024-02-10 10:00:00 | 149.50 | Delivered | +| 3 | 103 | 2024-03-05 16:45:00 | 75.00 | Processing | +| 4 | 104 | 2024-04-20 09:15:00 | 200.00 | Delivered | +| 5 | 105 | 2024-05-12 14:00:00 | 49.99 | Delivered | +| 11 | 111 | 2024-06-15 08:30:00 | 250.75 | Processing | +| 12 | 112 | 2024-07-22 15:45:00 | 399.99 | Shipped | +| 13 | 113 | 2024-08-30 10:20:00 | 599.99 | Delivered | ++----------+-------------+---------------------+--------+------------+ +8 rows in set (0.00 sec) +``` + +## Apply effects + +Through this solution, retail companies can synchronize order data to the analysis database in real time to implement application scenarios such as order statistics, sales trend analysis, and customer behavior insights to support business decisions. At the same time, breakpoint resumption ensures data consistency when network delays or task interruptions occur, so that the data analysis system always maintains an accurate and reliable data source. \ No newline at end of file diff --git a/docs/MatrixOne/Maintain/cdc/mocdc.md b/docs/MatrixOne/Maintain/cdc/mocdc.md new file mode 100644 index 000000000..3d4a93c81 --- /dev/null +++ b/docs/MatrixOne/Maintain/cdc/mocdc.md @@ -0,0 +1,240 @@ +# mo_cdc data synchronization + +**CDC (Change Data Capture)**is a technology that captures data changes in the database in real time and can record insert, update and delete operations. It monitors database changes to achieve real-time synchronization and incremental processing of data to ensure data consistency between different systems. CDC is suitable for scenarios such as real-time data synchronization, data migration, disaster recovery and audit tracking. By reading transaction logs, it reduces the pressure of full data replication and improves system performance and efficiency. Its advantages include low latency, high real-time performance, flexible support for multiple databases and systems, and adaptability to changing large-scale data environments. + +MatrixOne supports table-level data synchronization through the `mo_cdc` utility. This chapter will introduce how to use `mo_cdc`. + +!!! note + mo_cdc is a data synchronization tool for enterprise-level services. You need to contact your MatrixOne account manager to obtain the tool download path. + +## Reference command guide + +help -print reference guide + +```bash +>./mo_cdc help +This command allows you to manage CDC Task, including task create, task show, task pause, task resume, task restart, and task drop. + +Usage: + mo_cdc [flags] + mo_cdc [command] + +Available Commands: + completion Generate the autocompletion script for the specified shell + help Help about any command + task Manage Task + +Flags: + -h, --help help for mo_cdc + +Use "mo_cdc [command] --help" for more information about a command. +``` + +## Create task + +### Grammar structure + +``` +mo_cdc task create + --task-name + --source-uri + --sink-type + --sink-uri + --tables + --level + --account + --no-full +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|task-name | Synchronization task name| +|source-uri|Source (mo) connection string| +|sink-type| Downstream type, currently supports mysql| +|tables | The name of the table that needs to be synchronized, separate multiple table names with commas | +|level | The range of tables selected for synchronization, currently only supports tenants| +|account| Synchronized tenant, when level is account, you need to specify | +|no-full| Optional, full volume is enabled by default, adding this parameter means full volume is turned off| + +#### example + +```bash +>./mo_cdc task create --task-name "task1" --source-uri "mysql://root:111@127.0.0.1:6001" --sink-uri "mysql://root:111@127.0.0.1:3306" --sink-type "mysql" --tables "db1.t1:db1.t1,db1.t2:db1.t2" --level "account" --account "sys" + +>./mo_cdc task create --task-name "task2" --source-uri "mysql://acc1:admin:111@127.0.0.1:6001" --sink-uri "mysql://root:111@127.0.0.1:3306" --sink-type "mysql" --tables "db1.table1:db2.tab1" --level "account" --account "acc1" +``` + +## View tasks + +Viewing cdc tasks can only view tasks created by the currently connected user. + +### Grammar structure + +``` +mo_cdc task show + --source-uri + --all + --task-name +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|source-uri|Source server address| +|all| View all synchronization tasks| +|task-name | Synchronization task name| + +**return information** + +| Field | Description | +| ----| ----| +|task-id|task id| +|task-name| Task name| +|source-uri | Source server address | +|sink-uri | Downstream identification resources| +|state | task status, running or stopped| +|checkpoint | synchronization progress| +|timestamp | current timestamp| + +#### Example + +```bash +#View all sync tasks +>./mo_cdc task show "task1" --source-uri "mysql://root:111@127.0.0.1:6001" --all +[ + { + "task-id": "0192bd8a-781b-776e-812a-3f4440fceff9", + "task-name": "task1", + "source-uri": "mysql://root:******@127.0.0.1:6001", + "sink-uri": "mysql://root:******@127.0.0.1:3306", + "state": "running", + "checkpoint": "{\n \"db1.t1\": 2024-10-24 16:12:56.254918 +0800 CST,\n \"db1.t2\": 2024-10-24 16:12:56.376204 +0800 CST,\n}", + "timestamp": "2024-10-24 16:12:56.897015 +0800 CST" + } +] + +#View specific sync tasks +>./mo_cdc task show "task1" --source-uri "mysql://acc1:admin:111@127.0.0.1:6001" --task-name "task2" +[ + { + "task-id": "0192bd94-e716-73c4-860e-a392a0d68d6f", + "task-name": "task2", + "source-uri": "mysql://acc1:admin:******@127.0.0.1:6001", + "sink-uri": "mysql://root:******@127.0.0.1:3306", + "state": "running", + "checkpoint": "{\n \"db1.table1\": 2024-10-24 16:14:43.552274 +0800 CST,\n}", + "timestamp": "2024-10-24 16:14:43.664386 +0800 CST" + } +] +``` + +## Pause task + +### Grammar structure + +``` +mo_cdc task pause + --source-uri + --all + --task-name +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|source-uri|Source server address| +|all| Pause all synchronization tasks| +|task-name | Synchronization task name| + +#### Example + +```bash +#Pause specific tasks +./mo_cdc task pause --source-uri "mysql://acc1:admin:111@127.0.0.1:6001" --task-name "task2" + +#Pause all tasks +./mo_cdc task pause --source-uri "mysql://acc1:admin:111@127.0.0.1:6001" --all +``` + +## Recovery task + +The task can only be resumed when its status is stopped, and the resume process will resume the upload from the breakpoint. If the task is suspended for longer than the GC retention period, operations during this period will not be synchronized, and the system will only synchronize the final data status of the task. + +### Grammar structure + +``` +mo_cdc task resume + --source-uri + --task-name +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|source-uri|Source server address| +|task-name | Synchronization task name| + +#### Example + +```bash +./mo_cdc task resume --source-uri "mysql://acc1:admin:111@127.0.0.1:6001" --task-name "task2" +``` + +## Restart task + +Restarting the cdc task will ignore the synchronization progress record before the task and restart the synchronization from the beginning. + +### Grammar structure + +``` +mo_cdc task restart + --source-uri + --task-name +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|source-uri|Source server address| +|task-name | Synchronization task name| + +#### Example + +```bash +./mo_cdc task restart --source-uri "mysql://acc1:admin:111@127.0.0.1:6001" --task-name "task2" +``` + +## Delete task + +### Grammar structure + +``` +mo_cdc task drop + --source-uri + --all + --task-name +``` + +**Parameter description** + +| Parameters | Description | +| ----| ----| +|source-uri|Source server address| +|all|Delete all synchronization tasks| +|task-name | Delete a synchronization task with a specific name | + +#### Example + +```bash +#Delete specific tasks +./mo_cdc task drop --source-uri "mysql://acc1:admin:111@127.0.0.1:6001" --task-name "task2" + +#Delete all tasks +./mo_cdc task drop --source-uri "mysql://acc1:admin:111@127.0.0.1:6001" --all +``` diff --git a/docs/MatrixOne/Maintain/mo_ctl.md b/docs/MatrixOne/Maintain/mo_ctl.md deleted file mode 100644 index d9c6bfdc3..000000000 --- a/docs/MatrixOne/Maintain/mo_ctl.md +++ /dev/null @@ -1,358 +0,0 @@ -# mo_ctl Tool - -`mo_ctl` is a command-line tool that helps you deploy and install MatrixOne, start and stop control, and connect to the database. - -## Feature Overview - -The operating systems that `mo_ctl` has adapted so far are shown in the table below: - -| OS | Version | -| ------ | ----------------------- | -| Debian | 11 and above | -| Ubuntu | 20.04 and above | -| macOS | Monterey 12.3 and above | - -The current function list of `mo_ctl` is shown in the table below. - -| Command | Function | -| ------- | -------- | -| `mo_ctl help` | See a list of statements and functions for the `mo_ctl` tool itself | -| `mo_ctl precheck` | Check dependencies required for MatrixOne source code installation, namely golang, gcc, git, MySQL Client | -| `mo_ctl deploy` | Download and install and compile the corresponding version of MatrixOne; the default is to install the latest stable version | -| `mo_ctl start` | Start MatrixOne service | -| `mo_ctl status` | Check if the MatrixOne service is running | -| `mo_ctl stop` | Stop all MatrixOne service processes | -| `mo_ctl restart` | Restart MatrixOne service | -| `mo_ctl connect` | Call MySQL Client to connect to MatrixOne service | -| `mo_ctl upgrade` | Upgrade/downgrade MatrixOne from the current version to a release version or commit id version | -| `mo_ctl set_conf` | Set various usage parameters | -| `mo_ctl get_conf` | View current parameters | -| `mo_ctl uninstall` | Uninstall MatrixOne from MO_PATH path | -| `mo_ctl watchdog` | Set a scheduled task to ensure the availability of MatrixOne service, check the status of MatrixOne every minute, and automatically pull up the service if the service is found to be suspended | -| `mo_ctl sql` | Execute SQL directly through commands or a text file composed of SQL | -| `mo_ctl ddl_convert` | A tool to convert MySQL DDL statements into MatrixOne statements | -| `mo_ctl get_cid` | View the source version of the current MatrixOne download repository | -| `mo_ctl get_branch` | View the branch version of the current MatrixOne download repository | -| `mo_ctl pprof` | Used to collect MatrixOne profiling data | - -## Install mo_ctl - -Depending on whether you have internet access, you can choose to install the `mo_ctl` tool online or offline; you need to be careful to consistently execute commands as root or with sudo privileges (and add sudo before each command). Meanwhile, `install.sh` will use the `unzip` command to decompress the `mo_ctl` package; please ensure the `unzip` command is installed. - -### Install Online - -``` -wget https://raw.githubusercontent.com/matrixorigin/mo_ctl_standalone/main/install.sh && sudo bash +x ./install.sh - -# alternate address -wget https://ghproxy.com/https://github.com/matrixorigin/mo_ctl_standalone/blob/main/install.sh && sudo bash +x install.sh -``` - -For users running this command in a macOS environment, if you are a non-root user, run `install.sh` with the following statement: - -``` -sudo -u $(whoami) bash +x ./install.sh -``` - -### Install Offline - -``` -# 1. First, download the installation script to the local computer, and then upload it to the installation machine -wget https://raw.githubusercontent.com/matrixorigin/mo_ctl_standalone/main/install.sh -wget https://github.com/matrixorigin/mo_ctl_standalone/archive/refs/heads/main.zip -O mo_ctl.zip - -# alternate address -wget https://ghproxy.com/https://github.com/matrixorigin/mo_ctl_standalone/blob/main/install.sh -wget https://ghproxy.com/https://github.com/matrixorigin/mo_ctl_standalone/archive/refs/heads/main.zip -O mo_ctl.zip - -# 2. Install from offline package -bash +x ./install.sh mo_ctl.zip -``` - -## Quick Start - -You can quickly install and deploy the stand-alone version of MatrixOne through the following steps. For more information, see [Deploy standalone MatrixOne](../Get-Started/install-standalone-matrixone.md). - -1. Use the `mo_ctl help` command to view the tool guide. - -2. Use the `mo_ctl precheck` command to check whether the pre-dependency conditions are met. - -3. Use the `mo_ctl get_conf` command to set relevant parameters, and the possible parameter configurations are as follows: - - ``` - # check default parameter values - mo_ctl set_conf MO_PATH="/data/mo/matrixone" # set your own mo path - mo_ctl set_conf MO_GIT_URL="https://ghproxy.com/https://github.com/matrixorigin/matrixone.git" # in case have network issues, you can set this conf by overwriting default value MO_GIT_URL="https:// github.com/matrixorigin/matrixone.git" - ``` - -4. Use the `mo_ctl deploy`command to install and deploy the latest stable version of MatrixOne. - -5. Use the `mo_ctl start` command to launch the MatrixOne service with the command . - -6. Use the `mo_ctl connect` command to connect to the MatrixOne service. - -## Reference command guide - -### help - print reference guide - -``` -mo_ctl help -Usage : mo_ctl [option_1] [option_2] - -[option_1] : available: connect | ddl_connect | deploy | get_branch | get_cid | get_conf | help | pprof | precheck | query | restart | set_conf | sql | start | status | stop | uninstall | upgrade | watchdog - 1) connect : connect to mo via mysql client using connection info configured - 2) ddl_convert : convert ddl file to mo format from other types of database - 3) deploy : deploy mo onto the path configured - 4) get_branch : upgrade or downgrade mo from current version to a target commit id or stable version - 5) get_cid : print mo git commit id from the path configured - 6) get_conf : get configurations - 7) help : print help information - 8) pprof : collect pprof information - 9) precheck : check pre-requisites for mo_ctl - 10) restart : a combination operation of stop and start - 11) set_conf : set configurations - 12) sql : execute sql from string, or a file or a path containg multiple files - 13) start : start mo-service from the path configured - 14) status : check if there's any mo process running on this machine - 15) stop : stop all mo-service processes found on this machine - 16) uninstall : uninstall mo from path MO_PATH=/data/mo/20230712_1228//matrixone - 17) upgrade : upgrade or downgrade mo from current version to a target commit id or stable version - 18) watchdog : setup a watchdog crontab task for mo-service to keep it alive - e.g. : mo_ctl status - - [option_2] : Use " mo_ctl [option_1] help " to get more info - e.g. : mo_ctl deploy help -``` - -Use `mo_ctl [option_1] help` for instructions on using the next-level `mo_ctl [option_1]` functionality. - -### precheck - check pre-dependency - -Before installing MatrixOne from the source code, use `mo_ctl precheck` to check the pre-dependency conditions. The pre-dependence depends on `go`/`gcc`/`git`/`mysql(client)`. - -``` -mo_ctl precheck help -Usage : mo_ctl precheck # check pre-requisites for mo_ctl - Check list : go gcc git mysql -``` - -### deploy - install MatrixOne - -Use `mo_ctl deploy [mo_version] [force]` to install and deploy a stable version of MatrixOne, or a specified version. The `force` option can delete the existing version of MatrixOne in the same directory and force a new version to be reinstalled. - -``` -mo_ctl deploy help -Usage : mo_ctl deploy [mo_version] [force] # deploy mo onto the path configured - [mo_version]: optional, specify an mo version to deploy - [force] : optional, if specified will delete all content under MO_PATH and deploy from beginning - e.g. : mo_ctl deploy # default, same as mo_ctl deploy v1.2.4 - : mo_ctl deploy main # deploy development latest version - : mo_ctl deploy d29764a # deploy development version d29764a - : mo_ctl deploy v1.2.4 # deploy stable verson v1.2.4 - : mo_ctl deploy force # delete all under MO_PATH and deploy verson v1.2.4 - : mo_ctl deploy v1.2.4 force # delete all under MO_PATH and deploy stable verson v1.2.4 from beginning -``` - -### start - launch MatrixOne - -Use `mo_ctl start` to start MatrixOne service; the startup file path is located under `MO_PATH`. - -``` -mo_ctl start help -Usage : mo_ctl start # start mo-service from the path configured -``` - -### stop - stop MatrixOne - -Use `mo_ctl stop [force]` to stop all MatrixOne services on local machine, and if multiple MatrixOne services are running, they will also be stopped. - -``` - mo_ctl stop help -Usage : mo_ctl stop [force] # stop all mo-service processes found on this machine - [force] : optional, if specified, will try to kill mo-services with -9 option, so be very carefully - e.g.: mo_ctl stop # default, stop all mo-service processes found on this machine - : mo_ctl stop force # stop all mo-services with kill -9 command -``` - -### restart - restart MatrixOne - -Use `mo_ctl restart [force]` to stop all MatrixOne services on this machine and restart the MatrixOne services located in the path of `MO_PATH`. - -``` -mo_ctl restart help -Usage : mo_ctl restart [force] # a combination operation of stop and start - [force] : optional, if specified, will try to kill mo-services with -9 option, so be very carefully - e.g. : mo_ctl restart # default, stop all mo-service processes found on this machine and start mo-serivce under path of conf MO_PATH - : mo_ctl restart force # stop all mo-services with kill -9 command and start mo-serivce under path of conf MO_PATH -``` - -### connect - using mysql-client to connect MatrixOne - -Use `mo_ctl connect` to connect to MatrixOne service, the connection parameters are set by `mo_ctl` tool. - -``` -mo_ctl connect help -Usage : mo_ctl connect # connect to mo via mysql client using connection info configured -``` - -### status - Check the status of MatrixOne - -Use `mo_ctl status` to check whether MatrixOne is running or not. - -``` -mo_ctl status help -Usage : mo_ctl status # check if there's any mo process running on this machine -``` - -### get_cid - print MatrixOne code commit id - -Use `mo_ctl get_cid` to print the MatrixOne repository commit id under the current `MO_PATH` path. - -``` -mo_ctl get_cid help -Usage : mo_ctl get_cid # print mo commit id from the path configured -``` - -### get_branch - print MatrixOne code commit id - -Use `mo_ctl get_branch` to print the MatrixOne codebase branch under the current `MO_PATH` path. - -``` -mo_ctl get_branch help -Usage : mo_ctl get_branch # print which git branch mo is currently on -``` - -### pprof - Collect performance information - -Use `mo_ctl pprof [item] [duration]` to collect related performance information of MatrixOne, mainly for developers to debug. - -``` -mo_ctl pprof help -Usage : mo_ctl pprof [item] [duration] # collect pprof information - [item] : optional, specify what pprof to collect, available: profile | heap | allocs - 1) profile : default, collect profile pprof for 30 seconds - 2) heap : collect heap pprof at current moment - 3) allocs : collect allocs pprof at current moment - [duration] : optional, only valid when [item]=profile, specifiy duration to collect profile - e.g. : mo_ctl pprof - : mo_ctl pprof profile # collect duration will use conf value PPROF_PROFILE_DURATION from conf file or 30 if it's not set - : mo_ctl pprof profile 30 - : mo_ctl pprof heap -``` - -### set_conf - configuration parameters - -Use `mo_ctl set_conf [conf_list]` to configure 1 or more usage parameters. - -``` -mo_ctl set_conf help -Usage : mo_ctl setconf [conf_list] # set configurations - [conf_list] : configuration list in key=value format, seperated by comma - e.g. : mo_ctl setconf MO_PATH=/data/mo/matrixone,MO_PW=M@trix0riginR0cks,MO_PORT=6101 # set multiple configurations - : mo_ctl setconf MO_PATH=/data/mo/matrixone # set single configuration -``` - -### get_conf - get parameter list - -Use `mo_ctl get_conf [conf_list]` to get one or more current configuration items. - -``` -mo_ctl get_conf help -Usage : mo_ctl getconf [conf_list] # get configurations - [conf_list] : optional, configuration list in key, seperated by comma. - : use 'all' or leave it as blank to print all configurations - e.g. : mo_ctl getconf MO_PATH,MO_PW,MO_PORT # get multiple configurations - : mo_ctl getconf MO_PATH # get single configuration - : mo_ctl getconf all # get all configurations - : mo_ctl getconf # get all configurations -``` - -#### mo_ctl get_conf - Detailed Parameter List - -Using `mo_ctl get_conf` will print a list of all the parameters used by the current tool. Their meanings and value ranges are shown in the table below. - -| Parameter Name | Function | Value Specification | -| ---------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------- | -| MO_PATH | Location of MatrixOne's code repository and executables | Folder path | -| MO_LOG_PATH | Location of MatrixOne's logs | Folder path, default: ${MO_PATH}/matrixone/logs | -| MO_HOST | IP address for connecting to MatrixOne service | IP address, default: 127.0.0.1 | -| MO_PORT | Port number for connecting to MatrixOne service | Port number, default: 6001 | -| MO_USER | Username for connecting to MatrixOne service | Username, default: root | -| MO_PW | Password for connecting to MatrixOne service | Password, default: 111 | -| CHECK_LIST | Dependencies required for precheck | Default: ("go" "gcc" "git" "mysql") | -| GCC_VERSION | gcc version to be checked in precheck | Default: 8.5.0 | -| GO_VERSION | go version to be checked in precheck | Default: 1.22 | -| MO_GIT_URL | Repository URL for fetching MatrixOne source code | Default: | -| MO_DEFAULT_VERSION | Default version of MatrixOne to be fetched | Default: v1.2.4 | -| GOPROXY | Address of GOPROXY used for faster dependency retrieval in China | Default: , direct | -| STOP_INTERVAL | Interval to wait for service status check after stopping the service | Default: 5 seconds | -| START_INTERVAL | Interval to wait for service status check after starting the service | Default: 2 seconds | -| MO_DEBUG_PORT | Debug port for MatrixOne, usually used by developers | Default: 9876 | -| MO_CONF_FILE | Launch configuration file for MatrixOne | Default: ${MO_PATH}/matrixone/etc/launch/launch.toml | -| RESTART_INTERVAL | Interval to wait for service status check after restarting the service | Default: 2 seconds | -| PPROF_OUT_PATH | Output path for collecting golang performance data | Default: /tmp/pprof-test/ | -| PPROF_PROFILE_DURATION | Duration for collecting golang performance data | Default: 30 seconds | - -### ddl_convert - DDL format conversion - -Use `mo_ctl ddl_convert [options] [src_file] [tgt_file]` to convert a DDL file from other database syntax formats to MatrixOne's DDL format, currently only supported by `mysql_to_mo` mode. - -``` -mo_ctl ddl_convert help -Usage : mo_ctl ddl_convert [options] [src_file] [tgt_file] # convert a ddl file to mo format from other types of database - [options] : available: mysql_to_mo - [src_file] : source file to be converted, will use env DDL_SRC_FILE from conf file by default - [tgt_file] : target file of converted output, will use env DDL_TGT_FILE from conf file by default - e.g. : mo_ctl ddl_convert mysql_to_mo /tmp/mysql.sql /tmp/mo.sql -``` - -### sql - Execute SQL - -Use `mo_ctl sql [sql]` to execute SQL text or SQL files. - -``` -mo_ctl sql help -Usage : mo_ctl sql [sql] # execute sql from string, or a file or a path containg multiple files - [sql] : a string quote by "", or a file, or a path - e.g. : mo_ctl sql "use test;select 1;" # execute sql "use test;select 1" - : mo_ctl sql /data/q1.sql # execute sql in file /data/q1.sql - : mo_ctl sql /data/ # execute all sql files with .sql postfix in /data/ -``` - -### uninstall - Uninstall MatrixOne - -Use `mo_ctl uninstall` to uninstall MatrixOne from MO_PATH. - -``` -mo_ctl uninstall help -Usage : mo_ctl uninstall # uninstall mo from path MO_PATH=/data/mo//matrixone - # note: you will need to input 'Yes/No' to confirm before uninstalling -``` - -### upgrade - upgrade/downgrade MatrixOne version - -Use `mo_ctl upgrade version` or `mo_ctl upgrade commitid` to upgrade or downgrade MatrixOne from the current version to a stable version or a commit id version. - -``` -mo_ctl upgrade help -Usage : mo_ctl upgrade [version_commitid] # upgrade or downgrade mo from current version to a target commit id or stable version - [commitid] : a commit id such as '38888f7', or a stable version such as 'v1.2.4' - : use 'latest' to upgrade to latest commit on main branch if you don't know the id - e.g. : mo_ctl upgrade 38888f7 # upgrade/downgrade to commit id 38888f7 on main branch - : mo_ctl upgrade latest # upgrade/downgrade to latest commit on main branch - : mo_ctl upgrade v1.2.4 # upgrade/downgrade to stable version v1.2.4 -``` - -### watchdog - Keep Alive MatrixOne - -Use `mo_ctl watchdog [options]` to set a scheduled task to ensure the availability of MatrixOne service, check the status of MatrixOne every minute, and automatically pull up the service if it finds that the service is suspended. - -``` -mo_ctl watchdog help -Usage : mo_ctl watchdog [options] # setup a watchdog crontab task for mo-service to keep it alive - [options] : available: enable | disable | status - e.g. : mo_ctl watchdog enable # enable watchdog service for mo, by default it will check if mo-servie is alive and pull it up if it's dead every one minute - : mo_ctl watchdog disable # disable watchdog - : mo_ctl watchdog status # check if watchdog is enabled or disabled - : mo_ctl watchdog # same as mo_ctl watchdog status -``` diff --git a/docs/MatrixOne/Maintain/mount-data-by-docker.md b/docs/MatrixOne/Maintain/mount-data-by-docker.md index 13518e796..92e4ebebe 100644 --- a/docs/MatrixOne/Maintain/mount-data-by-docker.md +++ b/docs/MatrixOne/Maintain/mount-data-by-docker.md @@ -24,12 +24,14 @@ To ensure the safety of the data directory, mount the local data directory to th 3. Mount the local **empty directory** to the Docker container directory */mo-data*, execute the following command: ```shell - sudo docker run --name --privileged -d -p 6001:6001 -v ${local_data_path}/mo-data:/mo-data:rw matrixorigin/matrixone:1.2.4 + sudo docker run --name --privileged -d -p 6001:6001 -v ${local_data_path}/mo-data:/mo-data:rw matrixorigin/matrixone:2.0.0 ``` | Parameters | Description | | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | ${local_data_path}/mo-data:/mo-data | mount the local disk directory *${local_data_path}/mo-data* to the container directory */mo-data*
__Tips__: The local data directory to be mounted must be an empty directory. | + | ${local_data_path}/mo-data:/mo-data | mount the local disk directory *${local_data_path}/mo-data* to the container directory */mo-data*
+ +__Tips__: The local data directory to be mounted must be an empty directory. | ## Mount the *customized configuration file* @@ -44,7 +46,7 @@ If you need to modify the configuration file. In that case, it would be best to 2. To launch MatrixOne MatrixOne has not been running in Docker, execute the following command: ``` - docker run -d -p 6001:6001 --name matrixone --privileged=true matrixorigin/matrixone:1.2.4 + docker run -d -p 6001:6001 --name matrixone --privileged=true matrixorigin/matrixone:2.0.0 ``` 3. Check the containerID that MatrixOne has been running in Docker, and copy the configuration file directory to the local directory: @@ -66,7 +68,7 @@ If you need to modify the configuration file. In that case, it would be best to 6. Mount the configuration file to the Docker container directory and launch MatrixOne. Execute the following command: ```shell - sudo docker run --name --privileged -d -p 6001:6001 -v ${local_config_path}/etc:/etc:rw --entrypoint "/mo-service" matrixorigin/matrixone:1.2.4 -launch /etc/launch/launch.toml + sudo docker run --name --privileged -d -p 6001:6001 -v ${local_config_path}/etc:/etc:rw --entrypoint "/mo-service" matrixorigin/matrixone:2.0.0 -launch /etc/launch/launch.toml ``` | Parameters | Description | diff --git a/docs/MatrixOne/Overview/feature/mysql-compatibility.md b/docs/MatrixOne/Overview/feature/mysql-compatibility.md index d022381c3..f8ff706a5 100644 --- a/docs/MatrixOne/Overview/feature/mysql-compatibility.md +++ b/docs/MatrixOne/Overview/feature/mysql-compatibility.md @@ -131,6 +131,7 @@ MatrixOne is highly compatible with the MySQL 8.0 protocol and commonly used fea * TIMESTAMP: The maximum value range of MySQL is `'1970-01-01 00:00:01.000000'` UTC to `'2038-01-19 03:14:07.999999'` UTC, the maximum range of MatrixOne is `'0001- 01-01 00:00:00'` UTC to `'9999-12-31 23:59:59'` UTC. * MatrixOne supports `UUID` type. * MatrixOne supports vector types. +* MatrixOne supports datalink type. * Spatial types are not supported. * `SET` types are not supported. * `MEDIUMINT` type is not supported. diff --git a/docs/MatrixOne/Overview/matrixone-feature-list.md b/docs/MatrixOne/Overview/matrixone-feature-list.md index 5213aacc8..71109008a 100644 --- a/docs/MatrixOne/Overview/matrixone-feature-list.md +++ b/docs/MatrixOne/Overview/matrixone-feature-list.md @@ -11,7 +11,7 @@ This document lists the features supported by the latest version of MatrixOne an | ALTER DATABASE | N | | CREATE TABLE | Y | | ALTER TABLE | E, The clauses: `CHANGE [COLUMN]`, `MODIFY [COLUMN]`, `RENAME COLUMN`, `ADD [CONSTRAINT [symbol]] PRIMARY KEY`, `DROP PRIMARY KEY`, and `ALTER COLUMN ORDER BY` can be used in ALTER It can be freely combined in the TABLE statement. Still, it is not supported to be used with other clauses for the time being. | -| RENAME TABLE | N, Can be replaced by `ALTER TABLE tbl RENAME TO new_tbl` | +| RENAME TABLE | Y | | DROP TABLE | Y | | CREATE INDEX | Y, Secondary indexes have no speedup | | DROP INDEX | Y | @@ -182,12 +182,16 @@ This document lists the features supported by the latest version of MatrixOne an | Role-Based Access Control (RBAC) | Y | | Multi-Account | Y | -## Backup and Restore +## Backup and restore -| Backup and Restore | Supported(Y)/Not supported (N) /Experimental (E) | -| ------------ | ---------------------------------- | -| Logical Backup and Restore | Y, Only the modump tool is supported | -| Physical Backup and Restore | Y | +| Backup and Recovery | Supported (Y)/Not Supported (N)/Experimental (E) | +| --------------------------| ----------------------------------| +| Logical backup recovery | Y, only supports mo-dump tool | +| Physical backup recovery | Y, only supports mobackup tool | +| Snapshot backup and recovery | Y, supports mobackup tool and SQL | +| PITR | Y, supports mobackup tool and SQL | +| CDC synchronization | Y, only supports matrixone to mysql, supports mo_cdc tool | +| Primary and backup disaster recovery | Y, only supports cold backup | ## Management Tool diff --git a/docs/MatrixOne/Overview/whats-new.md b/docs/MatrixOne/Overview/whats-new.md index a9ae08084..2798fdc81 100644 --- a/docs/MatrixOne/Overview/whats-new.md +++ b/docs/MatrixOne/Overview/whats-new.md @@ -1,5 +1,5 @@ # **What's New** -The lastest version of MatrixOne is v24.1.2.4, releases on 11th Sep, 2024. See the following: +The lastest version of MatrixOne is v24.2.0.0, releases on 31th Oct, 2024. See the following: -* [v24.1.2.4 Release Notes](../Release-Notes/v24.1.2.4.md) +* [v24.2.0.0 Release Notes](../Release-Notes/v24.2.0.0.md) diff --git a/docs/MatrixOne/Reference/Data-Types/data-types.md b/docs/MatrixOne/Reference/Data-Types/data-types.md index ec0cfd88e..13b399b7e 100644 --- a/docs/MatrixOne/Reference/Data-Types/data-types.md +++ b/docs/MatrixOne/Reference/Data-Types/data-types.md @@ -573,4 +573,41 @@ mysql> select * from t1; | [1, 2, 3] | [4, 5] | +-----------+--------+ 1 row in set (0.00 sec) +``` + +## Datalink data type + +|Type | Explanation | +|----------------|---------------------| +|datalink | Special data class used to store links to documents (such as satge) or files | + +### Example + +```sql +drop table test01; +create table test01 (col1 int, col2 datalink); +create stage stage01 url='file:///Users/admin/case/'; +insert into test01 values (1, 'file:///Users/admin/case/t1.csv'); +insert into test01 values (2, 'file:///Users/admin/case/t1.csv?size=2'); +insert into test01 values (3, 'file:///Users/admin/case/t1.csv?offset=4'); +insert into test01 values (4, 'file:///Users/admin/case/t1.csv?offset=4&size=2'); +insert into test01 values (5, 'stage://stage01/t1.csv'); +insert into test01 values (6, 'stage://stage01/t1.csv?size=2'); +insert into test01 values (7, 'stage://stage01/t1.csv?offset=4'); +insert into test01 values (8, 'stage://stage01/t1.csv?offset=4&size=2'); + +mysql> select * from test01; ++------+-------------------------------------------------+ +| col1 | col2 | ++------+-------------------------------------------------+ +| 1 | file:///Users/admin/case/t1.csv | +| 2 | file:///Users/admin/case/t1.csv?size=2 | +| 3 | file:///Users/admin/case/t1.csv?offset=4 | +| 4 | file:///Users/admin/case/t1.csv?offset=4&size=2 | +| 5 | stage://stage01/t1.csv | +| 6 | stage://stage01/t1.csv?size=2 | +| 7 | stage://stage01/t1.csv?offset=4 | +| 8 | stage://stage01/t1.csv?offset=4&size=2 | ++------+-------------------------------------------------+ +8 rows in set (0.01 sec) ``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Data-Types/datalink-type.md b/docs/MatrixOne/Reference/Data-Types/datalink-type.md new file mode 100644 index 000000000..379859b19 --- /dev/null +++ b/docs/MatrixOne/Reference/Data-Types/datalink-type.md @@ -0,0 +1,87 @@ +# DATALINK type + +The `DATALINK` type is a special data type used to store links to documents (such as satges) or files. Its main purpose is to store the link address of the document in the database, rather than storing the document itself. This type can be applied in various scenarios, especially when dealing with large-scale document management, providing quick access to documents without actually storing the documents in a database. + +Use the `DATALINK` data type to: + +- Saves storage space: the document is actually stored in external storage (such as an object storage system), while the database only saves the link. +- Convenient document access: By storing links, the system can quickly access documents without additional storage and processing. +- Improve data operation efficiency: Avoid processing large files directly in the database, improving the speed and efficiency of data operations. + +## Insert DATALINK type data + +**GRAMMAR STRUCTURE** + +``` +INSERT INTO TABLE_NAME VALUES ('/>|//>?&' ) +``` + +**Parameter explanation** + +| Parameters | Description | +| ----| ----| +| file | Points to the local file system file location. | +| stage | points to stage pointing to file location. | +| offset | Optional. Offset, indicating the starting point of the read content. | +| size | Optional. Specifies the size of the read content, in subsections. | + +## Read DATALINK type data + +If you want to read the data pointed by `DATALINK` to a file link, you can use the [load_file](../../Reference/Functions-and-Operators/Other/load_file.md) function. + +## Example + +There is a file `t1.csv` under `/Users/admin/case` + +```bash +(base) admin@192 case % cat t1.csv +this is a test message +``` + +```sql +drop table test01; +create table test01 (col1 int, col2 datalink); +create stage stage01 url='file:///Users/admin/case/'; +insert into test01 values (1, 'file:///Users/admin/case/t1.csv'); +insert into test01 values (2, 'file:///Users/admin/case/t1.csv?size=2'); +insert into test01 values (3, 'file:///Users/admin/case/t1.csv?offset=4'); +insert into test01 values (4, 'file:///Users/admin/case/t1.csv?offset=4&size=2'); +insert into test01 values (5, 'stage://stage01/t1.csv'); +insert into test01 values (6, 'stage://stage01/t1.csv?size=2'); +insert into test01 values (7, 'stage://stage01/t1.csv?offset=4'); +insert into test01 values (8, 'stage://stage01/t1.csv?offset=4&size=2'); + +mysql> select * from test01; ++------+-------------------------------------------------+ +| col1 | col2 | ++------+-------------------------------------------------+ +| 1 | file:///Users/admin/case/t1.csv | +| 2 | file:///Users/admin/case/t1.csv?size=2 | +| 3 | file:///Users/admin/case/t1.csv?offset=4 | +| 4 | file:///Users/admin/case/t1.csv?offset=4&size=2 | +| 5 | stage://stage01/t1.csv | +| 6 | stage://stage01/t1.csv?size=2 | +| 7 | stage://stage01/t1.csv?offset=4 | +| 8 | stage://stage01/t1.csv?offset=4&size=2 | ++------+-------------------------------------------------+ +8 rows in set (0.01 sec) + +mysql> select col1, load_file(col2) from test01; ++------+-------------------------+ +| col1 | load_file(col2) | ++------+-------------------------+ +| 1 | this is a test message + | +| 2 | th | +| 3 | is a test message + | +| 4 | i | +| 5 | this is a test message + | +| 6 | th | +| 7 | is a test message + | +| 8 | i | ++------+-------------------------+ +8 rows in set (0.01 sec) +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Json/jq.md b/docs/MatrixOne/Reference/Functions-and-Operators/Json/jq.md new file mode 100644 index 000000000..e46fd2250 --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Json/jq.md @@ -0,0 +1,49 @@ +# **JQ()** + +## **Function description** + +The `JQ()` function is used to parse and transform JSON data based on jq expressions. Different from [`TRY_JQ()`](./try_jq.md), `TRY_JQ()` supports returning null value when an error occurs, while `JQ()` directly throws an exception when encountering an error. + +## **Grammar structure** + +```sql +select jq(jsonDoc, pathExpression); +``` + +## **Parameter explanation** + +| Parameters | Description | +| ----| ----| +| jsonDoc | This is a column or expression containing JSON data. | +| pathExpression | Used to specify how to extract fields from JSON data | + +## **Example** + +```sql +mysql> select jq('{"foo": 128}', '.foo'); ++------------------------+ +| jq({"foo": 128}, .foo) | ++------------------------+ +| 128 | ++------------------------+ +1 row in set (0.01 sec) + +mysql> select jq(null, '.foo'); ++----------------+ +| jq(null, .foo) | ++----------------+ +| NULL | ++----------------+ +1 row in set (0.00 sec) + +mysql> select jq('{"id": "sample", "10": {"b": 42}}', '{(.id): .["10"].b}'); ++-----------------------------------------------------------+ +| jq({"id": "sample", "10": {"b": 42}}, {(.id): .["10"].b}) | ++-----------------------------------------------------------+ +| {"sample":42} | ++-----------------------------------------------------------+ +1 row in set (0.00 sec) + +mysql> select jq('[1, 2, 3]', '.foo & .bar'); +ERROR 1105 (HY000): unexpected token "&" +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Json/json-functions.md b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json-functions.md deleted file mode 100644 index 591321b47..000000000 --- a/docs/MatrixOne/Reference/Functions-and-Operators/Json/json-functions.md +++ /dev/null @@ -1,253 +0,0 @@ -# JSON Functions - -MatrixOne supports the following JSON functions: - -|Name|Description| -|---|---| -|JSON_EXTRACT()|Return data from JSON document| -|JSON_QUOTE() | Quote JSON document| -|JSON_UNQUOTE() |Unquote JSON value| - -## `JSON_EXTRACT()` - -A JSON path expression selects a value within a JSON document. - -Path expressions are useful with functions that extract parts of or modify a JSON document, to specify where within that document to operate. For example, the following query extracts from a JSON document the value of the member with the name key: - -```sql -mysql> SELECT JSON_EXTRACT('{"id": 14, "name": "Aztalan"}', '$.name'); -+---------------------------------------------------------+ -| JSON_EXTRACT('{"id": 14, "name": "Aztalan"}', '$.name') | -+---------------------------------------------------------+ -| "Aztalan" | -+---------------------------------------------------------+ -``` - -Path syntax uses a leading $ character to represent the JSON document under consideration, optionally followed by selectors that indicate successively more specific parts of the document: - -- A period followed by a key name names the member in an object with the given key. The key name must be specified within double quotation marks if the name without quotes is not legal within path expressions (for example, if it contains a space). - -- [N] appended to a *path* that selects an array names the value at position `N` within the array. Array positions are integers beginning with zero. If the array is negative, an error is generated. - -- Paths can contain * or ** wildcards: - - + .[*] evaluates to the values of all members in a JSON object. - - + [*] evaluates to the values of all elements in a JSON array. - - + prefix**suffix evaluates to all paths that begin with the named prefix and end with the named suffix. - -- A path that does not exist in the document (evaluates to nonexistent data) evaluates to NULL. - -**Example**: - -``` -[3, {"a": [5, 6], "b": 10}, [99, 100]] -``` - -- $[0] evaluates to 3. - -- $[1] evaluates to {"a": [5, 6], "b": 10}. - -- $[2] evaluates to [99, 100]. - -- $[3] evaluates to NULL (it refers to the fourth array element, which does not exist). - -Because $[1] and $[2] evaluate to nonscalar values, they can be used as the basis for more-specific path expressions that select nested values. Examples: - -- $[1].a evaluates to [5, 6]. - -- $[1].a[1] evaluates to 6. - -- $[1].b evaluates to 10. - -- $[2][0] evaluates to 99. - -As mentioned previously, path components that name keys must be quoted if the unquoted key name is not legal in path expressions. Let $ refer to this value: - -``` -{"a fish": "shark", "a bird": "sparrow"} -``` - -The keys both contain a space and must be quoted: - -- $."a fish" evaluates to shark. - -- $."a bird" evaluates to sparrow. - -Paths that use wildcards evaluate to an array that can contain multiple values: - -```sql -mysql> SELECT JSON_EXTRACT('{"a": 1, "b": 2, "c": [3, 4, 5]}', '$.*'); -+---------------------------------------------------------+ -| JSON_EXTRACT('{"a": 1, "b": 2, "c": [3, 4, 5]}', '$.*') | -+---------------------------------------------------------+ -| [1, 2, [3, 4, 5]] | -+---------------------------------------------------------+ - -mysql> SELECT JSON_EXTRACT('{"a": 1, "b": 2, "c": [3, 4, 5]}', '$.c[*]'); -+------------------------------------------------------------+ -| JSON_EXTRACT('{"a": 1, "b": 2, "c": [3, 4, 5]}', '$.c[*]') | -+------------------------------------------------------------+ -| [3, 4, 5] | -+------------------------------------------------------------+ -``` - -In the following example, the path $**.b evaluates to multiple paths ($.a.b and $.c.b) and produces an array of the matching path values: - -```sql -mysql> SELECT JSON_EXTRACT('{"a": {"b": 1}, "c": {"b": 2}}', '$**.b'); -+---------------------------------------------------------+ -| JSON_EXTRACT('{"a": {"b": 1}, "c": {"b": 2}}', '$**.b') | -+---------------------------------------------------------+ -| [null, 1, 2] | -+---------------------------------------------------------+ -``` - -In the following example, showes the querying JSON values from columns: - -```sql -CREATE table t1 (a json,b int); -INSERT into t1(a,b) values ('{"a":1,"b":2,"c":3}',1); - -mysql> SELECT json_extract(t1.a,'$.a') from t1 where t1.b=1; -+-------------------------+ -| json_extract(t1.a, $.a) | -+-------------------------+ -| 1 | -+-------------------------+ -1 row in set (0.00 sec) - -INSERT into t1(a,b) values ('{"a":4,"b":5,"c":6}',2); - -mysql> SELECT json_extract(t1.a,'$.b') from t1 where t1.b=2; -+-------------------------+ -| json_extract(t1.a, $.b) | -+-------------------------+ -| 5 | -+-------------------------+ -1 row in set (0.00 sec) - -mysql> SELECT json_extract(t1.a,'$.a') from t1; -+-------------------------+ -| json_extract(t1.a, $.a) | -+-------------------------+ -| 1 | -| 4 | -+-------------------------+ -2 rows in set (0.00 sec) - -INSERT into t1(a,b) values ('{"a":{"q":[1,2,3]}}',3); - -mysql> SELECT json_extract(t1.a,'$.a.q[1]') from t1 where t1.b=3; -+------------------------------+ -| json_extract(t1.a, $.a.q[1]) | -+------------------------------+ -| 2 | -+------------------------------+ -1 row in set (0.01 sec) - -INSERT into t1(a,b) values ('[{"a":1,"b":2,"c":3},{"a":4,"b":5,"c":6}]',4); - -mysql> SELECT json_extract(t1.a,'$[1].a') from t1 where t1.b=4; -+----------------------------+ -| json_extract(t1.a, $[1].a) | -+----------------------------+ -| 4 | -+----------------------------+ -1 row in set (0.00 sec) -``` - -## `JSON_QUOTE()` - -The `JSON_QUOTE` function converts a string value into a string in `JSON` format. It wraps the string with double quotes, escapes inner quotes and other characters, quotes the string as a `JSON` value, and then returns the result as a utf8mb4 string. If the parameter is `NULL`, it returns `NULL`. - -The `JSON_QUOTE` function is commonly used to generate valid `JSON` strings to include in JSON documents. - -The syntax structure is: `select JSON_QUOTE(string_value);` - -string_value is the string to be converted to a `JSON` string. The function returns a string in `JSON` format, where the original string is surrounded by quotes and properly escaped. - -**Example**: - -```sql -mysql> SELECT JSON_QUOTE('null'), JSON_QUOTE('"null"'); -+------------------+--------------------+ -| json_quote(null) | json_quote("null") | -+------------------+--------------------+ -| "null" | "\"null\"" | -+------------------+--------------------+ -1 row in set (0.00 sec) -mysql> SELECT JSON_QUOTE('[1, 2, 3]'); -+-----------------------+ -| json_quote([1, 2, 3]) | -+-----------------------+ -| "[1, 2, 3]" | -+-----------------------+ -1 row in set (0.00 sec) - -mysql> SELECT JSON_QUOTE('hello world'); -+-------------------------+ -| json_quote(hello world) | -+-------------------------+ -| "hello world" | -+-------------------------+ -1 row in set (0.00 sec) -``` - -As we can see, the original string is surrounded by quotes, and the double quotes within the string are escaped. This allows it to be used in JSON format, such as as a property value in a JSON object. - -## `JSON_UNQUOTE()` - -The `JSON_UNQUOTE()` function extracts an unquoted `JSON` value from a `JSON` string and returns it as a string. - -Specifically, the `JSON_UNQUOTE()` function takes a `JSON` string as input, extracts a `JSON` value from it, and returns it as a string. If the input `JSON` string does not contain a valid `JSON` value or the parameter is `NULL`, the function returns `NULL`. An error occurs if the value begins and ends with double quotes but is not a valid `JSON` string literal. - -The syntax structure: `select JSON_UNQUOTE(string_value);` - -In the string, specific sequences have special meanings and start with a backslash (), known as escape characters, with the following rules. For example, \b is interpreted as backspace, while \B is interpreted as B. For all other escape sequences, the backslash is ignored. That is, the escape character is interpreted as unescaped. For example, \x is just x. These sequences are case-sensitive. - -|Escape Sequence|Character Represented by Sequence| -|---|---| -|\"|A double quote (") character| -|\b|A backspace character| -|\f|A formfeed character| -|\n|A newline (linefeed) character| -|\r|A carriage return character| -|\t|A tab character| -|\\|A backslash (\) character| -|\uXXXX|UTF-8 bytes for Unicode value XXXX| - -**Example**: - -```sql -mysql> SET @j = '"abc"'; -Query OK, 0 rows affected (0.00 sec) - -mysql> SELECT @j, JSON_UNQUOTE(@j); -+-------+------------------+ -| @j | json_unquote(@j) | -+-------+------------------+ -| "abc" | abc | -+-------+------------------+ -1 row in set (0.00 sec) - -mysql> SET @j = '[1, 2, 3]'; -Query OK, 0 rows affected (0.00 sec) - -mysql> SELECT @j, JSON_UNQUOTE(@j); -+-----------+------------------+ -| @j | json_unquote(@j) | -+-----------+------------------+ -| [1, 2, 3] | [1, 2, 3] | -+-----------+------------------+ -1 row in set (0.00 sec) - -mysql> SELECT JSON_UNQUOTE('"\\t\\u0032"'); -+----------------------------+ -| json_unquote("\\t\\u0032") | -+----------------------------+ -| 2 | -+----------------------------+ -1 row in set (0.00 sec) -``` diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_extract.md b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_extract.md new file mode 100644 index 000000000..6629583fd --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_extract.md @@ -0,0 +1,161 @@ +# **JSON_EXTRACT()** + +## **Function description** + +`JSON EXTRACT()` is a JSON query function that can be used to query JSON documents. + +-If used as a column after select, it is recommended to use the functions [JSON_EXTRACT_STRING()](./json_extract_string.md) and [JSON_EXTRACT_FLOAT64()](./json_extract_float64.md); + +-If comparing in a where condition: + -If the type is json object comparison, use `JSON EXTRACT()` + -If the type is a string type, use [JSON_EXTRACT_STRING()](./json_extract_string.md); + -If the type is float or int, use [JSON_EXTRACT_FLOAT64()](./json_extract_float64.md). + +## **Grammar structure** + +```sql +select json_extract(jsonDoc, pathExpression); +``` + +## **Parameter explanation** + +| Parameters | Description | +| ----| ----| +| jsonDoc | This is an expression containing JSON data. | +| pathExpression | Represents the path to access a value in the JSON document. The path starts with `$`, which represents the root of the JSON document, and can be followed by a period `.` and a key name or square brackets [ ] to access the elements of the array. | + +Path expressions must start with the `$` character: + +- `.` followed by a key name names the member in the object using the given key. Key names need to be enclosed in quotes. + +- `[N]`: After selecting the *path*of the array, name the value at position `N` in the array. Array positions are zero-based integers. If the array is negative, an error is reported. + +- The path can contain the `*` or `**` wildcard characters: + + + `.[*]` Computes the values ​​of all members in a JSON object. + + + `[*]` Computes the values ​​of all elements in a JSON array. + + + `prefix**suffix`: Count all paths starting with a named prefix and ending with a named suffix. + +- Paths that do not exist in the document (or data that does not exist) evaluate to `NULL`. + +The following set of JSON arrays: + +``` +[3, {"a": [5, 6], "b": 10}, [99, 100]] +``` + +- `$[0]` means 3. + +- `$[1]` means {"a": [5, 6], "b": 10}. + +- `$[2]` means [99, 100]. + +- `$[3]` is NULL (the array path starts from `$[0]`, and `$[3]` represents the fourth group of data, which does not exist). + +Since `$[1]` and `$[2]` evaluate to non-scalar values, expressions can be nested. For example: + +- `$[1].a` means [5, 6]. + +- `$[1].a[1]` means 6. + +- `$[1].b` means 10. + +- `$[2][0]` means 99. + +Key names require double quotes in path expressions. `$` quotes this key value and also requires double quotes: + +``` +{"a fish": "shark", "a bird": "sparrow"} +``` + +Both keys contain a space and must be enclosed in quotes: + +- `$."a fish"` means `shark`. + +- `$."a bird"` means `sparrow`. + +## **Example** + +```sql +mysql> select JSON_EXTRACT('{"a": 1, "b": 2, "c": [3, 4, 5]}', '$.*'); ++---------------------------------------------------------+ +| JSON_EXTRACT('{"a": 1, "b": 2, "c": [3, 4, 5]}', '$.*') | ++---------------------------------------------------------+ +| [1, 2, [3, 4, 5]] | ++---------------------------------------------------------+ + +mysql> SELECT JSON_EXTRACT('{"a": 1, "b": 2, "c": [3, 4, 5]}', '$.c[*]'); ++------------------------------------------------------------+ +| JSON_EXTRACT('{"a": 1, "b": 2, "c": [3, 4, 5]}', '$.c[*]') | ++------------------------------------------------------------+ +| [3, 4, 5] | ++------------------------------------------------------------+ + +mysql> select json_extract('{"a":{"q":[1,2,3]}}','$.a.q[1]'); ++---------------------------------------------+ +| json_extract({"a":{"q":[1,2,3]}}, $.a.q[1]) | ++---------------------------------------------+ +| 2 | ++---------------------------------------------+ +1 row in set (0.00 sec) + +mysql> select JSON_EXTRACT('{"a": {"b": 1}, "c": {"b": 2}}', '$**.b'); ++---------------------------------------------------------+ +| JSON_EXTRACT('{"a": {"b": 1}, "c": {"b": 2}}', '$**.b') | ++---------------------------------------------------------+ +| [null, 1, 2] | ++---------------------------------------------------------+ + +#In the following example, querying JSON values ​​from a column will be shown: +create table t1 (a json,b int); +insert into t1(a,b) values ('{"a":1,"b":2,"c":3}',1); + +mysql> select json_extract(t1.a,'$.a') from t1 where t1.b=1; ++-------------------------+ +| json_extract(t1.a, $.a) | ++-------------------------+ +| 1 | ++-------------------------+ +1 row in set (0.00 sec) + +insert into t1(a,b) values ('{"a":4,"b":5,"c":6}',2); + +mysql> select json_extract(t1.a,'$.b') from t1 where t1.b=2; ++-------------------------+ +| json_extract(t1.a, $.b) | ++-------------------------+ +| 5 | ++-------------------------+ +1 row in set (0.00 sec) + +mysql> select json_extract(t1.a,'$.a') from t1; ++-------------------------+ +| json_extract(t1.a, $.a) | ++-------------------------+ +| 1 | +| 4 | ++-------------------------+ +2 rows in set (0.00 sec) + +insert into t1(a,b) values ('{"a":{"q":[1,2,3]}}',3); + +mysql> select json_extract(t1.a,'$.a.q[1]') from t1 where t1.b=3; ++------------------------------+ +| json_extract(t1.a, $.a.q[1]) | ++------------------------------+ +| 2 | ++------------------------------+ +1 row in set (0.01 sec) + +insert into t1(a,b) values ('[{"a":1,"b":2,"c":3},{"a":4,"b":5,"c":6}]',4); + +mysql> select json_extract(t1.a,'$[1].a') from t1 where t1.b=4; ++----------------------------+ +| json_extract(t1.a, $[1].a) | ++----------------------------+ +| 4 | ++----------------------------+ +1 row in set (0.00 sec) +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_extract_float64.md b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_extract_float64.md new file mode 100644 index 000000000..32b0f6e06 --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_extract_float64.md @@ -0,0 +1,62 @@ +# **JSON_EXTRACT_FLOAT64()** + +## **Function description** + +`JSON_EXTRACT_FLOAT64()` is used to extract the numeric value of the specified path from JSON data. + +If comparing in where condition: + +- If the type is json object comparison, use [JSON EXTRACT()](./json_extract.md); +- If the type is a string type, use [JSON_EXTRACT_STRING()](./json_extract_string.md); +- If type is float or int, use `JSON_EXTRACT_FLOAT64()`. + +## **Grammar structure** + +```sql +select col_name from tab_name where json_extract_float64(jsonDoc, pathExpression)= number; +``` + +## **Parameter explanation** + +| Parameters | Description | +| ----| ----| +| jsonDoc | This is a column or expression containing JSON data. | +| pathExpression | Represents the path to access a value in the JSON document. Only one path can be received at a time. The path starts with `$`, indicating the root of the JSON document, and can be followed by a period `.` and a key name or square brackets [ ] to access the elements of the array. | +| number | Specifies the value to be extracted from the JSON data, which is a numeric type. | + +Path expressions must start with the `$` character: + +- `,` followed by a key name names the member in the object using the given key. Key names need to be enclosed in double quotes. + +- `[N]`: After selecting the *path*of the array, name the value at position `N` in the array. Array positions are zero-based integers. If the array is negative, an error is reported. + +- The path can contain the `*` or `**` wildcard characters: + + + `.[*]` Computes the values ​​of all members in a JSON object. + + + `[*]` Computes the values ​​of all elements in a JSON array. + + + `prefix**suffix`: Count all paths starting with a named prefix and ending with a named suffix. + +- Paths that do not exist in the document (or data that does not exist) evaluate to `NULL`. + +## **Example** + +```sql +create table student(n1 int,n2 json); +insert into student values + (1,'{"name": "tom", "age": 18, "score": 90,"gender": "male"}'), + (2,'{"name": "bob", "age": 20, "score": 80,"gender": "male"}'), + (3,'{"name": "jane", "age": 17, "score": 95,"gender": "female"}'), + (4,'{"name": "lily", "age": 19, "score": 79,"gender": "female"}'); + +mysql> select n1 from student where json_extract_float64(n2,'$.age')=19; ++------+ +| n1 | ++------+ +| 4 | ++------+ +1 row in set (0.00 sec) + +select json_extract_float64(n2,'$.age')=19; +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_extract_string.md b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_extract_string.md new file mode 100644 index 000000000..5830214d4 --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_extract_string.md @@ -0,0 +1,71 @@ +# **JSON_EXTRACT_FLOAT64()** + +## **Function description** + +`JSON_EXTRACT_FLOAT64()` is used to extract the numeric value of the specified path from JSON data. + +If comparing in where condition: + +- If the type is json object comparison, use [JSON EXTRACT()](./json_extract.md); +- If the type is a string type, use [JSON_EXTRACT_STRING()](./json_extract_string.md); +- If type is float or int, use `JSON_EXTRACT_FLOAT64()`. + +## **Grammar structure** + +```sql +select col_name from tab_name where json_extract_float64(jsonDoc, pathExpression)= number; +``` + +## **Parameter explanation** + +| Parameters | Description | +| ----| ----| +| jsonDoc | This is a column or expression containing JSON data. | +| pathExpression | Represents the path to access a value in the JSON document. Only one path can be received at a time. The path starts with `$`, indicating the root of the JSON document, and can be followed by a period `.` and a key name or square brackets [ ] to access the elements of the array. | +| number | Specifies the value to be extracted from the JSON data, which is a numeric type. | + +Path expressions must start with the `$` character: + +- `,` followed by a key name names the member in the object using the given key. Key names need to be enclosed in double quotes. + +- `[N]`: After selecting the *path*of the array, name the value at position `N` in the array. Array positions are zero-based integers. If the array is negative, an error is reported. + +- The path can contain the `*` or `**` wildcard characters: + + + `.[*]` Computes the values ​​of all members in a JSON object. + + + `[*]` Computes the values ​​of all elements in a JSON array. + + + `prefix**suffix`: Count all paths starting with a named prefix and ending with a named suffix. + +- Paths that do not exist in the document (or data that does not exist) evaluate to `NULL`. + +## **Example** + +```sql +create table student(n1 int,n2 json); +insert into student values + (1,'{"name": "tom", "age": 18, "score": 90,"gender": "male"}'), + (2,'{"name": "bob", "age": 20, "score": 80,"gender": "male"}'), + (3,'{"name": "jane", "age": 17, "score": 95,"gender": "female"}'), + (4,'{"name": "lily", "age": 19, "score": 79,"gender": "female"}'); + +mysql> select n1 from student where json_extract_float64(n2,'$.age')=19; ++------+ +| n1 | ++------+ +| 4 | ++------+ +1 row in set (0.00 sec) + +mysql> select json_extract_float64(n2,'$.age')=19 from student; ++--------------------------------------+ +| json_extract_float64(n2, $.age) = 19 | ++--------------------------------------+ +| false | +| false | +| false | +| true | ++--------------------------------------+ +4 rows in set (0.00 sec) +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_quote.md b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_quote.md new file mode 100644 index 000000000..ba500c50b --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_quote.md @@ -0,0 +1,46 @@ +# **JSON_QUOTE()** + +## **Function description** + +The `JSON_QUOTE` function is used to convert a string value to a string in JSON format. Quote the string as a JSON value by wrapping the string in double quotes and escaping inner quotes and other characters, then returning the result as a `utf8mb4` string. If the argument is NULL, NULL is returned. + +The `JSON_QUOTE` function is typically used to generate a valid JSON string for inclusion in a JSON document. + +## **Grammar structure** + +``` +select JSON_QUOTE(string_value); +``` + +## **Parameter description** + +`string_value` is the string to be converted to a JSON string. This function returns a JSON-formatted string, where the original string has been surrounded by quotes and escaped appropriately. + +## **Example** + +```sql +mysql> SELECT JSON_QUOTE('null'), JSON_QUOTE('"null"'); ++------------------+--------------------+ +| json_quote(null) | json_quote("null") | ++------------------+--------------------+ +| "null" | "\"null\"" | ++------------------+--------------------+ +1 row in set (0.00 sec) +mysql> SELECT JSON_QUOTE('[1, 2, 3]'); ++-----------------------+ +| json_quote([1, 2, 3]) | ++-----------------------+ +| "[1, 2, 3]" | ++-----------------------+ +1 row in set (0.00 sec) + +mysql> SELECT JSON_QUOTE('hello world'); ++-------------------------+ +| json_quote(hello world) | ++-------------------------+ +| "hello world" | ++-------------------------+ +1 row in set (0.00 sec) +``` + +As you can see, the original string is surrounded by quotes and the double quotes in the string are escaped. This way, you can use it as a JSON-formatted value, for example, as a property value of a JSON object. \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_row.md b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_row.md new file mode 100644 index 000000000..db7b46824 --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_row.md @@ -0,0 +1,33 @@ +# **JSON_ROW()** + +## **Function description** + +`JSON_ROW()` is used to convert each row into a json array. + +## **Grammar structure** + +```sql +select json_row(col) from tabl_name; +``` + +## **Example** + +```sql +create table student(n1 int,n2 json); +insert into student values + (1,'{"name": "tom", "age": 18, "score": 90,"gender": "male"}'), + (2,'{"name": "bob", "age": 20, "score": 80,"gender": "male"}'), + (3,'{"name": "jane", "age": 17, "score": 95,"gender": "female"}'), + (4,'{"name": "lily", "age": 19, "score": 79,"gender": "female"}'); + +mysql> select json_row(null,n2) from student; ++---------------------------------------------------------------------+ +| json_row(null, n2) | ++---------------------------------------------------------------------+ +| [null,{"age": 18, "gender": "male", "name": "tom", "score": 90}] | +| [null,{"age": 20, "gender": "male", "name": "bob", "score": 80}] | +| [null,{"age": 17, "gender": "female", "name": "jane", "score": 95}] | +| [null,{"age": 19, "gender": "female", "name": "lily", "score": 79}] | ++---------------------------------------------------------------------+ +4 rows in set (0.00 sec) +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_unquote.md b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_unquote.md new file mode 100644 index 000000000..8b170a42c --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Json/json_unquote.md @@ -0,0 +1,60 @@ +# **JSON_UNQUOTE()** + +## **Function description** + +The `JSON_UNQUOTE()` function is used to extract an unquoted JSON value from a JSON string and return it as a string. + +Specifically, the `JSON_UNQUOTE()` function takes a `JSON` string as input, extracts a `JSON` value from it, and returns it as a string. If the input `JSON` string does not contain a valid `JSON` value or the argument is `NULL`, the function returns `NULL`. If the argument is NULL, NULL is returned. If the value starts and ends with double quotes but is not a valid `JSON` string literal, an error occurs. + +## **Grammar structure** + +```sql +select JSON_UNQUOTE(string_value); +``` + +In strings, certain sequences have special meanings. These sequences start with a backslash (\) and are called escape characters. The rules are as follows. For all other escape sequences, backslashes are ignored. That is, escaped characters are interpreted as if they were not escaped. For example, \x is x. These sequences are case-sensitive. For example, \b is interpreted as backspace, and \B is interpreted as B. + +|The character represented by the escape sequence| +|---|---| +|\"|Double quotes (") | +|\b|Backspace| +|\f|form feed character| +|\n|Line break| +|\r|Carriage return character| +|\t|tab| +|\\|Backslash (\) | +|\uXXXX|Unicode value XXXX UTF-8 byte| + +## **Example** + +```sql +mysql> SET @j = '"abc"'; +Query OK, 0 rows affected (0.00 sec) + +mysql> SELECT @j, JSON_UNQUOTE(@j); ++-------+------------------+ +| @j | json_unquote(@j) | ++-------+------------------+ +| "abc" | abc | ++-------+------------------+ +1 row in set (0.00 sec) + +mysql> SET @j = '[1, 2, 3]'; +Query OK, 0 rows affected (0.00 sec) + +mysql> SELECT @j, JSON_UNQUOTE(@j); ++-----------+------------------+ +| @j | json_unquote(@j) | ++-----------+------------------+ +| [1, 2, 3] | [1, 2, 3] | ++-----------+------------------+ +1 row in set (0.00 sec) + +mysql> SELECT JSON_UNQUOTE('"\\t\\u0032"'); ++----------------------------+ +| json_unquote("\\t\\u0032") | ++----------------------------+ +| 2 | ++----------------------------+ +1 row in set (0.00 sec) +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Json/try_jq.md b/docs/MatrixOne/Reference/Functions-and-Operators/Json/try_jq.md new file mode 100644 index 000000000..fee75f7b4 --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Json/try_jq.md @@ -0,0 +1,54 @@ +# **TRY_JQ()** + +## **Function description** + +The `TRY_JQ()` function is used to parse and transform JSON data based on jq expressions. Different from [`JQ()`](./jq.md), `TRY_JQ()` supports returning null value when an error occurs, while `JQ()` directly throws an exception when encountering an error. + +## **Grammar structure** + +```sql +select try_jq(jsonDoc, pathExpression); +``` + +## **Parameter explanation** + +| Parameters | Description | +| ----| ----| +| jsonDoc | This is a column or expression containing JSON data. | +| pathExpression | Used to specify how to extract fields from JSON data | + +## **Example** + +```sql +mysql> select try_jq('{"foo": 128}', '.foo'); ++----------------------------+ +| try_jq({"foo": 128}, .foo) | ++----------------------------+ +| 128 | ++----------------------------+ +1 row in set (0.00 sec) + +mysql> select try_jq(null, '.foo'); ++--------------------+ +| try_jq(null, .foo) | ++--------------------+ +| NULL | ++--------------------+ +1 row in set (0.00 sec) + +mysql> select try_jq('{"id": "sample", "10": {"b": 42}}', '{(.id): .["10"].b}'); ++---------------------------------------------------------------+ +| try_jq({"id": "sample", "10": {"b": 42}}, {(.id): .["10"].b}) | ++---------------------------------------------------------------+ +| {"sample":42} | ++---------------------------------------------------------------+ +1 row in set (0.00 sec) + +mysql> select try_jq('[1, 2, 3]', '.foo & .bar'); ++--------------------------------+ +| try_jq([1, 2, 3], .foo & .bar) | ++--------------------------------+ +| NULL | ++--------------------------------+ +1 row in set (0.00 sec) +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Other/load_file.md b/docs/MatrixOne/Reference/Functions-and-Operators/Other/load_file.md new file mode 100644 index 000000000..6fc4d4e05 --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Other/load_file.md @@ -0,0 +1,72 @@ +# **LOAD_FILE()** + +## **Function description** + +The `LOAD_FILE()` function is used to read the contents of the file pointed to by the datalink type. + +## **Function syntax** + +``` +>LOAD_FILE(datalink_type_data); +``` + +## **Parameter explanation** + +| Parameters | Description | +| ----| ----| +| datalink_type_data | datalink type data can be converted using the [cast()](../../../Reference/Operators/operators/cast-functions-and-operators/cast/) function | + +## Example + +There is a file `t1.csv` under `/Users/admin/case` + +```bash +(base) admin@192 case % cat t1.csv +this is a test message +``` + +```sql +create table t1 (col1 int, col2 datalink); +create stage stage1 url='file:///Users/admin/case/'; +insert into t1 values (1, 'file:///Users/admin/case/t1.csv'); +insert into t1 values (2, 'stage://stage1//t1.csv'); + +mysql> select * from t1; ++------+---------------------------------+ +| col1 | col2 | ++------+---------------------------------+ +| 1 | file:///Users/admin/case/t1.csv | +| 2 | stage://stage1//t1.csv | ++------+---------------------------------+ +2 rows in set (0.00 sec) + +mysql> select col1, load_file(col2) from t1; ++------+-------------------------+ +| col1 | load_file(col2) | ++------+-------------------------+ +| 1 | this is a test message + | +| 2 | this is a test message + | ++------+-------------------------+ +2 rows in set (0.01 sec) + + +mysql> select load_file(cast('file:///Users/admin/case/t1.csv' as datalink)); ++--------------------------------------------------------------+ +| load_file(cast(file:///Users/admin/case/t1.csv as datalink)) | ++--------------------------------------------------------------+ +| this is a test message + | ++--------------------------------------------------------------+ +1 row in set (0.00 sec) + +mysql> select load_file(cast('stage://stage1//t1.csv' as datalink)); ++-----------------------------------------------------+ +| load_file(cast(stage://stage1//t1.csv as datalink)) | ++-----------------------------------------------------+ +| this is a test message + | ++-----------------------------------------------------+ +1 row in set (0.00 sec) +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Other/save_file.md b/docs/MatrixOne/Reference/Functions-and-Operators/Other/save_file.md new file mode 100644 index 000000000..e3b68fe19 --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Other/save_file.md @@ -0,0 +1,49 @@ +# **SAVE_FILE()** + +## **Function description** + +The `SAVE_FILE()` function is used to write content to the file pointed to by datalink, and the reference line returns the byte length of the written content. + +## **Function syntax** + +``` +>SAVE_FILE(datalink_type_data,content); +``` + +## **Parameter explanation** + +| Parameters | Description | +| ----| ----| +| datalink_type_data | datalink type data can be converted using the [cast()](../../../Reference/Operators/operators/cast-functions-and-operators/cast/) function | +| content | Need to write the content of the file pointed to by datalink | + +## Example + +``` +drop stage if exists tab1; +create stage stage01 url='file:///Users/admin/case/'; +mysql> select save_file(cast('stage://stage01/test.csv' as datalink), 'this is a test message'); ++-------------------------------------------------------------------------------+ +| save_file(cast(stage://stage01/test.csv as datalink), this is a test message) | ++-------------------------------------------------------------------------------+ +| 22 | ++-------------------------------------------------------------------------------+ +1 row in set (0.00 sec) + +mysql> select save_file(cast('file:///Users/admin/case/test1.csv' as datalink), 'this is another test message'); ++-----------------------------------------------------------------------------------------------+ +| save_file(cast(file:///Users/admin/case/test1.csv as datalink), this is another test message) | ++-----------------------------------------------------------------------------------------------+ +| 28 | ++-----------------------------------------------------------------------------------------------+ +1 row in set (0.01 sec) + +``` + +```bash +(base) admin@192 case % cat test.csv +this is a test message + +(base) admin@192 case % cat test1.csv +this is another test message +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/Other/stage_list.md b/docs/MatrixOne/Reference/Functions-and-Operators/Other/stage_list.md new file mode 100644 index 000000000..7a7585545 --- /dev/null +++ b/docs/MatrixOne/Reference/Functions-and-Operators/Other/stage_list.md @@ -0,0 +1,57 @@ +# **STAGE_LIST()** + +## **Function description** + +The `STAGE_LIST()` function is used to view the directories and files in the stage. + +## **Function syntax** + +``` +>STAGE_LIST(stage:////) as f; +``` + +## **Parameter explanation** + +| Parameters | Description | +| ----| ----| +| stage_name | Required. The name of the stage. | +| path | Optional. The directory name under stage. | +| file_name | Optional. The file name under stage can use wildcard character *| + +## **Example** + +There are the following files and directories under the directory `/Users/admin/case`: + +```bash +(base) admin@192 case % ls +customer student.csv t1.csv t2.txt t3 user.txt +``` + +```sql +create stage stage_test url = 'file:///Users/admin/case'; + +mysql> select * from stage_list('stage://stage_test') as f; ++-------------------------------+ +| file | ++-------------------------------+ +| /Users/admin/case/customer | +| /Users/admin/case/student.csv | +| /Users/admin/case/t1.csv | +| /Users/admin/case/t2.txt | +| /Users/admin/case/t3 | +| /Users/admin/case/user.txt | ++-------------------------------+ +6 rows in set (0.00 sec) + +mysql> select * from stage_list('stage://stage_test/t*') as f; ++--------------------------+ +| file | ++--------------------------+ +| /Users/admin/case/t1.csv | +| /Users/admin/case/t2.txt | ++--------------------------+ +2 rows in set (0.01 sec) + +mysql> select * from stage_list('stage://stage_test/customer') as f; +Empty set (0.00 sec) +``` diff --git a/docs/MatrixOne/Reference/Functions-and-Operators/matrixone-function-list.md b/docs/MatrixOne/Reference/Functions-and-Operators/matrixone-function-list.md index a51b3e222..a5d669a59 100644 --- a/docs/MatrixOne/Reference/Functions-and-Operators/matrixone-function-list.md +++ b/docs/MatrixOne/Reference/Functions-and-Operators/matrixone-function-list.md @@ -164,19 +164,24 @@ This document lists the functions supported in the latest version of MatrixOne. ## Window function -| Function name | effect | +| Function name | effect | | --------------------------------------------------- | --------------------------------------- | | [DENSE_RANK()](./Window-Functions/dense_rank.md) | Provide a unique ranking for each row in the dataset| | [RANK()](./Window-Functions/rank.md) | Provide a unique ranking for each row in the dataset| | [ROW_UNMBER()](./Window-Functions/row_number.md) | Provide a unique serial number for each row in the data set| -## JSON functions +## JSON function -| Function name | effect | -| --------------------------------------------- | --------------------------------------- | -| [JSON_EXTRACT()](./Json/json-functions.md) | Returning data from a JSON document| -| [JSON_QUOTE()](./Json/json-functions.md) | Referencing JSON Documents| -| [JSON_UNQUOTE()](./Json/json-functions.md) | Dereferencing JSON documents| +| Function name | effect| +|------------------------------------------------| -------------------------------------| +| [JQ()](./Json/jq.md) | Used to parse and transform JSON data based on jq expressions | +| [JSON_EXTRACT()](./Json/json_extract.md) | Return data from a JSON document | +| [JSON_EXTRACT_FLOAT64()](./Json/json_extract_float64.md) | Extract the value of the specified path from JSON data | +| [JSON_EXTRACT_STRING()](./Json/json_extract_string.md) | Extract the value of a string with a specified path from JSON data | +| [JSON_QUOTE()](./Json/json_quote.md) | Quote JSON document | +| [JSON_ROW()](./Json/json_row.md) | Used to convert each row into a json array | +| [JSON_UNQUOTE()](./Json/json_unquote.md) | Unquote a JSON document | +| [TRY_JQ()](./Json/try_jq.md) | Used to parse and convert JSON data based on jq expressions, and provide a fault-tolerant mechanism | ## system operation and maintenance function @@ -190,9 +195,12 @@ This document lists the functions supported in the latest version of MatrixOne. ## Other functions -| Function name | effect | -| ------------------------------ | --------------------------------------- | -| [SAMPLE()](./Other/sample.md) | Primarily used for quick query narrowing| -| [SERIAL_EXTRACT()](./Other/serial_extract.md) | Used to extract individual elements of a sequence/tuple of values| -| [SLEEP()](./Other/sleep.md) | Pause (sleep) the current query for the specified number of seconds.| -| [UUID()](./Other/uuid.md) | Returns the generation of an internationally recognized unique identifier according to RFC 4122.| \ No newline at end of file +| Function name | effect | +| ----------------------------------| --------------------------------------------| +| [LOAD_FILE()](./Other/load_file.md) | Used to read the contents of the file pointed to by the datalink type. | +| [SAVE_FILE()](./Other/save_file.md) | Used to save the content of the file pointed to by the datalink type. | +| [SAMPLE()](./Other/sample.md) | Mainly used to quickly reduce the query scope | +| [SERIAL_EXTRACT()](./Other/serial_extract.md) | Used to extract individual elements in a sequence/tuple value | +| [SLEEP()](./Other/sleep.md) | Pause (sleep) the current query for the specified number of seconds | +| [STAGE_LIST()](./Other/stage_list.md) | Used to view directories and files in stage. | +| [UUID()](./Other/uuid.md) | Returns an internationally unique identifier generated according to RFC 4122 | \ No newline at end of file diff --git a/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/cast-functions-and-operators-overview.md b/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/cast-functions-and-operators-overview.md index e0abc4cc4..d83dd1102 100644 --- a/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/cast-functions-and-operators-overview.md +++ b/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/cast-functions-and-operators-overview.md @@ -5,5 +5,7 @@ | [BINARY()](binary.md) | convert a value to a binary string | | [CAST()](cast.md) | Cast a value as a certain type | | [CONVERT()](convert.md) | Cast a value as a certain type | +| [DECODE()](decode.md) | Used to decrypt ENCODE() encoded data | +| [ENCODE()](encode.md) | Used for symmetric encryption of strings | | [SERIAL()](serial.md) | Serialize concatenation strings without handling NULL values | | [SERIAL_FULL()](serial_full.md) | Serialize concatenation strings and handle NULL values | diff --git a/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/decode.md b/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/decode.md new file mode 100644 index 000000000..fea76adb8 --- /dev/null +++ b/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/decode.md @@ -0,0 +1,47 @@ +# **DECODE()** + +## **Function description** + +The `DECODE()` function is used to decrypt data encoded by [`ENCODE()`](./encode.md). + +## **Function syntax** + +``` +> DECODE(crypt_str, pass_str) +``` + +## **Parameter explanation** + +| Parameters | Description | +| ---------------| ----------------------------------| +| crypt_str | Encrypted string encoded by ENCODE(). | +| pass_str | Password string used for decryption, must be the same as the key used for encryption. | + +## **Example** + +```SQL +mysql> SELECT DECODE(ENCODE('hello', 'mysecretkey'), 'mysecretkey'); ++-------------------------------------------------+ +| DECODE(ENCODE(hello, mysecretkey), mysecretkey) | ++-------------------------------------------------+ +| hello | ++-------------------------------------------------+ +1 row in set (0.00 sec) + +CREATE TABLE users ( + id INT AUTO_INCREMENT PRIMARY KEY, + username VARCHAR(255), + encrypted_password BLOB +); + +INSERT INTO users (username, encrypted_password) +VALUES ('john', ENCODE('password123', 'mysecretkey')); + +mysql> SELECT username, DECODE(encrypted_password, 'mysecretkey') AS decrypted_password FROM users WHERE username = 'john'; ++----------+--------------------+ +| username | decrypted_password | ++----------+--------------------+ +| john | password123 | ++----------+--------------------+ +1 row in set (0.00 sec) +``` diff --git a/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/encode.md b/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/encode.md new file mode 100644 index 000000000..9741336c3 --- /dev/null +++ b/docs/MatrixOne/Reference/Operators/operators/cast-functions-and-operators/encode.md @@ -0,0 +1,30 @@ +# **ENCODE()** + +## **Function description** + +The `ENCODE()` function is used to symmetrically encrypt a string. It encodes a string by combining a secret key, and the same key is required for decoding. Need to cooperate with [`DECODE()`](./decode.md) to decrypt. + +## **Function syntax** + +``` +> ENCODE (str, pass_str); +``` + +## **Parameter explanation** + +| Parameters | Description | +| ---------------| ----------------------------------| +| str | The raw string to encode. | +| pass_str | Password string (key) used for encryption. | + +## **Example** + +```SQL +mysql> SELECT ENCODE('hello', 'mysecretkey'); ++----------------------------+ +| ENCODE(hello, mysecretkey) | ++----------------------------+ +| ?;? | ++----------------------------+ +1 row in set (0.00 sec) +``` \ No newline at end of file diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-pitr.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-pitr.md new file mode 100644 index 000000000..ac2e490c7 --- /dev/null +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-pitr.md @@ -0,0 +1,52 @@ +# **ALTER PITR** + +## **Syntax description** + +`ALTER PITR` is used to change PITR. + +## **Grammar structure** + +``` +> ALTER PITR RANGE +``` + +## **Example** + +```sql +drop table if exists t1; +create table t1 (a int); +insert into t1 values(1),(2),(3),(4); +create view v5 as select * from t1; + +mysql> select * from v5; ++------+ +| a | ++------+ +| 1 | +| 2 | +| 3 | +| 4 | ++------+ +4 rows in set (0.01 sec) + +alter view v5 as select * from t1 where a=1; + +mysql> select * from v5; ++------+ +| a | ++------+ +| 1 | ++------+ +1 row in set (0.01 sec) + +alter view v5 as select * from t1 where a > 2; + +mysql> select * from v5; ++------+ +| a | ++------+ +| 3 | +| 4 | ++------+ +2 rows in set (0.00 sec) +``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-publication.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-publication.md index b9d423051..7aa879aca 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-publication.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-publication.md @@ -1,8 +1,8 @@ # **ALTER PUBLICATION** -## **Grammar description** +## **Syntax description** -`ALTER PUBLICATION` Modify the publication. +`ALTER PUBLICATION` modifies the publication content. ## **Grammar structure** @@ -15,15 +15,16 @@ ALTER PUBLICATION pubname | DROP account_name, [, ... ]] [COMMENT 'string'] [DATABASE database_name] + [TAble table_name] ``` -## Interpretation of grammar +## Grammar explanation -- pubname: The name of the publication that already exists. -- account_name: Gets the tenant name for this publication. -- database_name: The name of the release library to modify. +-pubname: Existing publication name. +-account_name: The tenant name of the publication can be obtained. +-database_name: The name of the publishing library to be modified. -## **Examples** +## **示例** ```sql create account acc0 admin_name 'root' identified by '111'; @@ -32,39 +33,39 @@ create account acc2 admin_name 'root' identified by '111'; create database t; create publication pub3 database t account acc0,acc1; -alter publication pub3 account add accx;--Modification of the scope of publication +alter publication pub3 account add acc2;--Modify release scope mysql> show create publication pub3; -+-------------+-----------------------------------------------------------------------+ -| Publication | Create Publication | -+-------------+-----------------------------------------------------------------------+ -| pub3 | CREATE PUBLICATION `pub3` DATABASE `t` ACCOUNT `acc0`, `acc1`, `accx` | -+-------------+-----------------------------------------------------------------------+ ++-------------+-----------------------------------------------------------+ +| Publication | Create Publication | ++-------------+-----------------------------------------------------------+ +| pub3 | CREATE PUBLICATION pub3 DATABASE t ACCOUNT acc0,acc1,acc2 | ++-------------+-----------------------------------------------------------+ 1 row in set (0.01 sec) mysql> show publications; -+-------------+----------+---------------------+---------------------+----------------+----------+ -| publication | database | create_time | update_time | sub_account | comments | -+-------------+----------+---------------------+---------------------+----------------+----------+ -| pub3 | t | 2024-04-24 11:17:37 | 2024-04-24 11:17:44 | acc0,acc1,accx | | -+-------------+----------+---------------------+---------------------+----------------+----------+ -1 row in set (0.01 sec) ++-------------+----------+--------+----------------+---------------------+---------------------+---------------------+----------+ +| publication | database | tables | sub_account | subscribed_accounts | create_time | update_time | comments | ++-------------+----------+--------+----------------+---------------------+---------------------+---------------------+----------+ +| pub3 | t | * | acc0,acc1,acc2 | | 2024-10-28 11:20:20 | 2024-10-28 11:20:43 | | ++-------------+----------+--------+----------------+---------------------+---------------------+---------------------+----------+ +1 row in set (0.00 sec) -alter publication pub3 comment "this is pubs";--修改发布备注 +alter publication pub3 comment "this is pubs";--Modify release notes mysql> show publications; -+-------------+----------+---------------------+---------------------+----------------+--------------+ -| publication | database | create_time | update_time | sub_account | comments | -+-------------+----------+---------------------+---------------------+----------------+--------------+ -| pub3 | t | 2024-04-24 11:17:37 | 2024-04-24 11:41:43 | acc0,acc1,accx | this is pubs | -+-------------+----------+---------------------+---------------------+----------------+--------------+ ++-------------+----------+--------+----------------+---------------------+---------------------+---------------------+--------------+ +| publication | database | tables | sub_account | subscribed_accounts | create_time | update_time | comments | ++-------------+----------+--------+----------------+---------------------+---------------------+---------------------+--------------+ +| pub3 | t | * | acc0,acc1,acc2 | | 2024-10-28 11:20:20 | 2024-10-28 11:21:28 | this is pubs | ++-------------+----------+--------+----------------+---------------------+---------------------+---------------------+--------------+ 1 row in set (0.00 sec) create database new_pub3; -alter publication pub3 database new_pub3;--修改发布数据库 +alter publication pub3 database new_pub3;--Modify publishing database mysql> show publications; -+-------------+----------+---------------------+---------------------+----------------+--------------+ -| publication | database | create_time | update_time | sub_account | comments | -+-------------+----------+---------------------+---------------------+----------------+--------------+ -| pub3 | new_pub3 | 2024-04-24 11:17:37 | 2024-04-24 11:43:36 | acc0,acc1,accx | this is pubs | -+-------------+----------+---------------------+---------------------+----------------+--------------+ ++-------------+----------+--------+----------------+---------------------+---------------------+---------------------+--------------+ +| publication | database | tables | sub_account | subscribed_accounts | create_time | update_time | comments | ++-------------+----------+--------+----------------+---------------------+---------------------+---------------------+--------------+ +| pub3 | new_pub3 | * | acc0,acc1,acc2 | | 2024-10-28 11:20:20 | 2024-10-28 11:21:44 | this is pubs | ++-------------+----------+--------+----------------+---------------------+---------------------+---------------------+--------------+ 1 row in set (0.00 sec) ``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-stage.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-stage.md index f6ffd977d..dbb615b47 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-stage.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-stage.md @@ -1,93 +1,81 @@ # **ALTER STAGE** -## **Description** +## **Syntax description** -`ALTER STAGE` is used to modify the attributes of an existing named internal or external stage. +`ALTER STAGE` is used to modify the properties of an existing stage. !!! note - Cluster administrators (i.e., root users) and account administrators can modify the data stage. + `ALTER STAGE` can only modify one parameter at a time. Therefore, if you need to update multiple parameters at the same time, such as `URL` and `COMMENT`, you need to execute multiple `ALTER STAGE` statements separately, modifying one parameter each time. -## **Syntax** +## **Grammar structure** ``` -> ALTER STAGE [ IF EXISTS ] { stage_name } +> ALTER STAGE [ IF EXISTS ] { stage_name } SET { StageParams } - [ directoryTableParams ] [ COMMENT = '' ] - + StageParams (for Amazon S3) : -URL = "endpoint"='' CREDENTIALS = {"access_key_id"='', "secret_access_key"=''} - -StageParams (for Aliyun OSS) : -URL = "endpoint"='' CREDENTIALS = {"access_key_id"='', "secret_access_key"=''} - +URL = "s3://[//]" CREDENTIALS = {"AWS_KEY_ID"='', "AWS_SECRET_KEY"='', "AWS_ROLE"='', "AWS_TOKEN"='', "AWS_REGION"='', "COMPRESSION"='', 'PROVIDER'='', 'ENDPOINT'=''} + StageParams (for File System) : -URL= 'filepath' +URL= 'file://[/path/]' -directoryTableParams : -ENABLE = { TRUE | FALSE } +StageParams (for sub-stage): +URL= "stage://[/path/]" ``` -### Explanations +## **Grammar explanation** -- `IF NOT EXISTS`: An optional parameter used to check whether a stage with the same name already exists when modifying a stage, avoiding duplicate creations. +- `IF NOT EXISTS`: optional parameter, used to check whether a Stage with the same name already exists when creating a Stage to avoid repeated creation. -- `stage_name`: The name of the stage to be modified. +- `stage_name`: The name of the Stage to be created. -- `StageParams`: This parameter group is used to specify the stage's configuration parameters. +- `StageParams (for MinIO/Amazon S3)`: Configuration parameters used to specify the stage where the object is stored as MinIO or S3. - - `endpoint`: The connection URL for the stage, indicating the location of the object storage service. This URL's content may vary for object storage services like Amazon S3, Aliyun OSS, or a file system. For example s3.us-west-2.amazonaws.com + - `URL`: Specify a file path or directory in S3 storage + - `CREDENTIALS`: This is a JSON object containing the credential information required to connect to the object storage service. - - `CREDENTIALS`: This JSON object contains the credentials required to connect to the object storage service, such as `access_key_id`, `secret_access_key`, etc. + + `access_key_id`: Access key ID used for authentication. + + `secret_access_key`: The secret associated with the access key ID. + + `aws_role`: optional, used to specify the role name if an IAM role is used. Roles can be configured on AWS to assign different permissions. + + `aws_token`: Optional, security token used for temporary access to AWS services. -- `directoryTableParams`: This parameter group is used to specify the configuration of a directory table associated with the stage. ++ `aws_region`: Specifies the AWS region where Amazon S3 storage is located. + + `compression`: optional, specifies the compression type of the file. + + `provider`: Specify the cloud storage provider. + + `endpint`: Specifies to connect to a custom or third-party S3 API-compatible service. - - `ENABLE`: Indicates whether the directory table is enabled, with values `TRUE` or `FALSE` values. +- `StageParams (for File System)`: used to specify the configuration parameters of the stage stored in the file system. -## **Examples** + - `URL`: Specifies the file path or directory in the file storage. -```sql -CREATE TABLE `user` (`id` int(11) ,`user_name` varchar(255) ,`sex` varchar(255)); -INSERT INTO user(id,user_name,sex) values('1', 'weder', 'man'), ('2', 'tom', 'man'), ('3', 'wederTom', 'man'); - --- Create internal data stage -mysql> CREATE STAGE stage1 URL='/tmp' ENABLE = TRUE; - --- Export data from the table to data stage -mysql> SELECT * FROM user INTO OUTFILE 'stage1:/user.csv'; --- You can see your exported table in your local directory - -mysql> SHOW STAGES; -+------------+-----------------------------+---------+---------+ -| STAGE_NAME | URL | STATUS | COMMENT | -+------------+-----------------------------+---------+---------+ -| stage1 | /Users/Prinz/03testrepo/csv | ENABLED | | -+------------+-----------------------------+---------+---------+ -1 row in set (0.01 sec) - --- modify the stage1 -mysql> ALTER STAGE stage1 SET COMMENT 'user stage'; - -mysql> SHOW STAGES; -+------------+-----------------------------+---------+------------+ -| STAGE_NAME | URL | STATUS | COMMENT | -+------------+-----------------------------+---------+------------+ -| stage1 | /Users/Prinz/03testrepo/csv | ENABLED | user stage | -+------------+-----------------------------+---------+------------+ -1 row in set (0.00 sec) +- `StageParams (for sub-stage)`: Configuration parameters for sub-stage. + + - URL`: Specifies the file path or directory in the file storage. --- disable the data stage named 'stage1' -mysql> ALTER STAGE stage1 SET ENABLE = FALSE; -Query OK, 0 rows affected (0.00 sec) +- `COMMENT`: Comment. --- Try to export the data of the user table to the data stage named 'stage1:/user.csv', but stage1 has been disabled, so it is no longer available, and an error is reported -mysql> SELECT * FROM user INTO OUTFILE 'stage1:/user.csv'; -ERROR 20101 (HY000): internal error: stage 'stage1' is invalid, please check +## **Example** --- Re-enable with a data stage named 'stage1' -mysql> ALTER STAGE stage1 SET ENABLE = TRUE; -Query OK, 0 rows affected (0.00 sec) +```sql +create stage stage_fs url = 'file:///Users/admin/test' comment='this is a stage'; + +mysql> select * from mo_catalog.mo_stages where stage_name='stage_fs'; ++----------+------------+--------------------------+-------------------+--------------+---------------------+-----------------+ +| stage_id | stage_name | url | stage_credentials | stage_status | created_time | comment | ++----------+------------+--------------------------+-------------------+--------------+---------------------+-----------------+ +| 1 | stage_fs | file:///Users/admin/test | | disabled | 2024-10-09 03:46:00 | this is a stage | ++----------+------------+--------------------------+-------------------+--------------+---------------------+-----------------+ +1 row in set (0.00 sec) --- The export can be executed successfully again -mysql> SELECT * FROM user INTO OUTFILE 'stage1:/user.csv'; +alter stage stage_fs set url = 'file:///Users/admin/test1'; +alter stage stage_fs set comment='stage_fs has been changed'; + +mysql> select * from mo_catalog.mo_stages where stage_name='stage_fs'; ++----------+------------+---------------------------+-------------------+--------------+---------------------+---------------------------+ +| stage_id | stage_name | url | stage_credentials | stage_status | created_time | comment | ++----------+------------+---------------------------+-------------------+--------------+---------------------+---------------------------+ +| 1 | stage_fs | file:///Users/admin/test1 | | disabled | 2024-10-09 03:46:00 | stage_fs has been changed | ++----------+------------+---------------------------+-------------------+--------------+---------------------+---------------------------+ +1 row in set (0.00 sec) ``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-pitr.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-pitr.md new file mode 100644 index 000000000..fb600ce2a --- /dev/null +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-pitr.md @@ -0,0 +1,96 @@ +# CREATEPITR + +## Syntax description + +The `CREATE PITR` command is used to create a recovery point for Point-in-Time Recovery (PITR). Cluster administrators can create cluster-level or tenant-level pitrs, while tenant administrators can create tenant/database/table-level pitrs for the current tenant. The information of each pitr is only visible to the tenant that created the pitr, ensuring data isolation and security. + +## Grammar structure + +```sql +create pitr for + [cluster]|[account ]|[database ]|[table ] + range +``` + +### Grammar explanation + +**range**: int, time range value, 1-100. +**unit**: string time range unit, optional range h (hour), d (day, default), mo (month), y (year) + +## Example + +**Example 1: Cluster administrator creates pitr for cluster** + +```sql +create pitr cluster_pitr1 for cluster range 1 "d"; +mysql> show pitr; ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| PITR_NAME | CREATED_TIME | MODIFIED_TIME | PITR_LEVEL | ACCOUNT_NAME | DATABASE_NAME | TABLE_NAME | PITR_LENGTH | PITR_UNIT | ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| cluster_pitr1 | 2024-10-18 14:07:10 | 2024-10-18 14:07:10 | cluster | * | * | * | 1 | d | ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.01 sec) +``` + +**Example 2: Cluster administrator creates pitr for tenant** + +```sql +create account acc1 admin_name 'root' identified by '111'; +create account acc2 admin_name 'root' identified by '111'; +mysql> create pitr account_pitr1 for account acc1 range 1 "d"; +mysql> show pitr where pitr_name='account_pitr1'; ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| pitr_name | created_time | modified_time | pitr_level | account_name | database_name | table_name | pitr_length | pitr_unit | ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| account_pitr1 | 2024-10-18 14:11:57 | 2024-10-18 14:11:57 | account | acc1 | * | * | 1 | d | ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.00 sec) +``` + +**Example 3: Tenant administrator creates pitr for tenant** + +```sql +create pitr account_pitr1 range 2 "h"; + +mysql> show pitr where pitr_name='account_pitr1'; ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| pitr_name | created_time | modified_time | pitr_level | account_name | database_name | table_name | pitr_length | pitr_unit | ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| account_pitr1 | 2024-10-18 14:23:12 | 2024-10-18 14:23:12 | account | acc1 | * | * | 2 | h | ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.00 sec) +``` + +**Example 4: Tenant administrator creates pitr for database** + +```sql +mysql> create pitr db_pitr1 for database db1 range 1 'y'; +Query OK, 0 rows affected (0.01 sec) + +mysql> show pitr where pitr_name='db_pitr1'; ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| pitr_name | created_time | modified_time | pitr_level | account_name | database_name | table_name | pitr_length | pitr_unit | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| db_pitr1 | 2024-10-18 14:26:02 | 2024-10-18 14:26:02 | database | acc1 | db1 | * | 1 | y | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.01 sec) +``` + +**Example 5: Tenant administrator creates pitr for table** + +```sql +mysql> create pitr tab_pitr1 for database db1 table t1 range 1 'y'; +Query OK, 0 rows affected (0.02 sec) + +mysql> show pitr where pitr_name='tab_pitr1'; ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| pitr_name | created_time | modified_time | pitr_level | account_name | database_name | table_name | pitr_length | pitr_unit | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| tab_pitr1 | 2024-10-18 14:28:53 | 2024-10-18 14:28:53 | table | acc1 | db1 | t1 | 1 | y | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.01 sec) +``` + +## limit + +- Cluster administrators can only create tenant-level pitrs when creating pitrs for other tenants. \ No newline at end of file diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-publication.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-publication.md index 83bdb4a61..ce2418a04 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-publication.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-publication.md @@ -1,35 +1,42 @@ # **CREATE PUBLICATION** -## **Grammar description** +## **Syntax description** -`CREATE PUBLICATION` Adds a new publication to the current database. +`CREATE PUBLICATION` adds a new publication to the current database. ## **Grammar structure** ``` -CREATE PUBLICATION pubname - DATABASE database_name ACCOUNT +CREATE PUBLICATION + DATABASE [] ACCOUNT [ { ALL | account_name, [, ... ] }] - [ COMMENT 'string'] + [COMMENT 'string'] ``` -## Interpretation of grammar +## Grammar explanation -- pubname: The publication name. The publication name must be different from the name of any existing publication in the current database. +- pubname: Publish name. The publication name must be different from the name of any existing publication in the current database. - database_name: The name of a database that already exists under the current tenant. -- account_name: Gets the tenant name for this publication. +- account_name: The tenant name of the publication can be obtained. -## **Examples** +## **Example** ```sql -create database t; -create account acc0 admin_name 'root' identified by '111'; -create account acc1 admin_name 'root' identified by '111'; -mysql> create publication pub1 database t account acc0,acc1; -Query OK, 0 rows affected (0.01 sec) +create account acc01 admin_name 'root' identified by '111'; +create account acc02 admin_name 'root' identified by '111'; +create database db1; +use db1; +create table t1(n1 int); +create table t2(n1 int); + +--Database level publishing +create publication db_pub1 database db1 account acc01,acc02; + +--Table level publishing +create publication tab_pub1 database db1 table t1,t2 account acc01,acc02; ``` -## Limitations +## limit -MatrxiOne currently only supports publishing one database data at a time. +-Database-level publishing currently only supports publishing one database data at a time. \ No newline at end of file diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-snapshot.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-snapshot.md index c591dd9a8..dc3b42080 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-snapshot.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-snapshot.md @@ -1,46 +1,60 @@ # CREATE SNAPSHOT -## Syntax Description +## Syntax description -The `CREATE SNAPSHOT` command is used to create a snapshot. System tenants can create snapshots for themselves or for regular tenants, but regular tenants can only create snapshots for themselves. Snapshots created by a tenant are visible only to this tenant. +The `CREATE SNAPSHOT` command is used to create a snapshot of the database. Cluster administrators can create cluster-level or tenant-level snapshots, while ordinary tenant administrators can create tenant-level snapshots for the current tenant. Each snapshot is only visible to the tenant that created it, ensuring data isolation and security. -## Syntax structure +## Grammar structure ```sql -> CREATE SNAPSHOT snapshot_name FOR ACCOUNT account_name +create snapshot for [cluster]|[account ] ``` -## Examples +## Example + +**Example 1: Cluster administrator creates a cluster snapshot for the cluster** ```sql ---Execute under system tenant sys -create snapshot sp1 for account sys; -create snapshot sp2 for account acc1; +create snapshot cluster_sp for cluster; +mysql> show snapshots; ++---------------+----------------------------+----------------+--------------+---------------+------------+ +| SNAPSHOT_NAME | TIMESTAMP | SNAPSHOT_LEVEL | ACCOUNT_NAME | DATABASE_NAME | TABLE_NAME | ++---------------+----------------------------+----------------+--------------+---------------+------------+ +| cluster_sp | 2024-10-10 10:40:14.487655 | cluster | | | | ++---------------+----------------------------+----------------+--------------+---------------+------------+ +1 row in set (0.00 sec) +``` + +**Example 2: Cluster administrator creates tenant snapshot for tenant** +```sql +mysql> create snapshot account_sp1 for account acc1; mysql> show snapshots; +---------------+----------------------------+----------------+--------------+---------------+------------+ | SNAPSHOT_NAME | TIMESTAMP | SNAPSHOT_LEVEL | ACCOUNT_NAME | DATABASE_NAME | TABLE_NAME | +---------------+----------------------------+----------------+--------------+---------------+------------+ -| sp2 | 2024-05-10 09:49:08.925908 | account | acc1 | | | -| sp1 | 2024-05-10 09:48:50.271707 | account | sys | | | +| account_sp1 | 2024-10-10 10:58:53.946829 | account | acc1 | | | +---------------+----------------------------+----------------+--------------+---------------+------------+ -2 rows in set (0.00 sec) +1 row in set (0.00 sec) +``` + +**Example 3: A normal tenant administrator creates a tenant snapshot for the tenant** ---Executed under tenant acc1 -mysql> create snapshot sp3 for account acc2;--Regular tenants can only create snapshots for themselves -ERROR 20101 (HY000): internal error: only sys tenant can create tenant level snapshot for other tenant +```sql +create snapshot account_sp2 for account acc1; -create snapshot sp3 for account acc1; +mysql> create snapshot account_sp2 for account acc2; +ERROR 20101 (HY000): internal error: only sys tenant can create tenant level snapshot for other tenant--Tenant administrators can only create snapshots for this tenant mysql> show snapshots; +---------------+----------------------------+----------------+--------------+---------------+------------+ | SNAPSHOT_NAME | TIMESTAMP | SNAPSHOT_LEVEL | ACCOUNT_NAME | DATABASE_NAME | TABLE_NAME | +---------------+----------------------------+----------------+--------------+---------------+------------+ -| sp3 | 2024-05-10 09:53:09.948762 | account | acc1 | | | +| account_sp2 | 2024-10-10 11:19:12.699093 | account | acc1 | | | +---------------+----------------------------+----------------+--------------+---------------+------------+ 1 row in set (0.00 sec) ``` -## Limitations +## limit -- Currently only tenant-level snapshots are supported, not cluster-level, database-level, and table-level snapshots. \ No newline at end of file +- Currently, only cluster-level and tenant-level snapshots are supported. Database-level and table-level snapshots are not supported. \ No newline at end of file diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-stage.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-stage.md index bbe879ed3..ccebdb435 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-stage.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-stage.md @@ -1,81 +1,72 @@ # **CREATE STAGE** -## **Description** +## **Syntax description** -The `CREATE STAGE` command is used in the MatrixOne database to create a named internal or external data stage for **data export**. By creating a data stage and exporting data to it, you can download data files to your local system or store them in cloud storage services. +`CREATE STAGE` is used to provide an efficient and secure way to interact with data from external storage (such as Amazon S3, file systems). By creating an external stage, MatrixOne can read files from external storage and quickly load them into MatrixOne database tables. For example, load a CSV file on Amazon S3 into a table. -- **Internal Stage**: Internal stages store data files within the MatrixOne system. Internal stages can be either permanent or temporary. +The following external storages are currently supported: -- **External Stage**: External stages reference data files stored outside the MatrixOne environment. Currently, the following cloud storage services are supported: +- Amazon S3 +- MinIO +- File System - - Amazon S3 buckets - - Aliyun buckets - -The storage location can be private/protected or public—however, data held in archival cloud storage classes that require restoration before retrieval cannot be accessed. - -An internal or external stage can include a directory table. Directory tables maintain a catalog of staged file directories in cloud storage. - -- Configure a specified path to control the write permissions for user `SELECT INTO` operations. After creation, users can only write to the set `STAGE` path. - -- If no `STAGE` is created or all `STAGE` instances are `DISABLED`, users can write to any path permitted by the operating system or object storage permissions. - -- If not using a `STAGE`, users must forcefully include `credential` information during `SELECT INTO` operations. - -!!! note - 1. Cluster administrators (i.e., root users) and tenant administrators can create data stages. - 2. Once created, data tables can only be imported to the paths specified in the STAGE. - -## **Syntax** +## **Grammar structure** ``` > CREATE STAGE [ IF NOT EXISTS ] { stage_name } - { StageParams } - [ directoryTableParams ] - [ COMMENT = '' ] - -StageParams (for Amazon S3) : -URL = "endpoint"='' CREDENTIALS = {"access_key_id"='', "secret_access_key"=''} + { StageParams } + [ COMMENT = '' -StageParams (for Aliyun OSS) : -URL = "endpoint"='' CREDENTIALS = {"access_key_id"='', "secret_access_key"=''} +StageParams (for MinIO/Amazon S3) : +URL = "s3://[//]" CREDENTIALS = {"AWS_KEY_ID"='', "AWS_SECRET_KEY"='', "AWS_ROLE"='', "AWS_TOKEN"='', "AWS_REGION"='', "COMPRESSION"='', 'PROVIDER'='', 'ENDPOINT'=''} StageParams (for File System) : -URL= 'filepath' +URL= 'file://[/path/]' -directoryTableParams : -ENABLE = { TRUE | FALSE } +StageParams (for sub-stage): +URL= "stage://[/path/]" ``` -### Explanations +## **Grammar explanation** -- `IF NOT EXISTS`: An optional parameter used to check whether a stage with the same name already exists when creating a stage, avoiding duplicate creations. +- `IF NOT EXISTS`: optional parameter, used to check whether a Stage with the same name already exists when creating a Stage to avoid repeated creation. -- `stage_name`: The name of the stage to be created. +- `stage_name`: The name of the Stage to be created. -- `StageParams`: This parameter group is used to specify the stage's configuration parameters. +- `StageParams (for MinIO/Amazon S3)`: Configuration parameters used to specify the stage where the object is stored as MinIO or S3. - - `endpoint`: The connection URL for the stage, indicating the location of the object storage service. This URL's content may vary for object storage services like Amazon S3, Aliyun OSS, or a file system. For example s3.us-west-2.amazonaws.com + - `URL`: Specify a file path or directory in S3 storage + - `CREDENTIALS`: This is a JSON object containing the credential information required to connect to the object storage service. - - `CREDENTIALS`: This JSON object contains the credentials required to connect to the object storage service, such as `access_key_id`, `secret_access_key`, etc. + + `access_key_id`: Access key ID used for authentication. + + `secret_access_key`: The secret associated with the access key ID. + + `aws_role`: optional, used to specify the role name if an IAM role is used. Roles can be configured on AWS to assign different permissions. + + `aws_token`: Optional, security token used for temporary access to AWS services. -- `directoryTableParams`: This parameter group is used to specify the configuration of a directory table associated with the stage. ++ `aws_region`: Specifies the AWS region where Amazon S3 storage is located. + + `compression`: optional, specifies the compression type of the file. + + `provider`: Specify the cloud storage provider. + + `endpint`: Specifies to connect to a custom or third-party S3 API-compatible service. - - `ENABLE`: Indicates whether the directory table is enabled, with values `TRUE` or `FALSE` values. +- `StageParams (for File System)`: used to specify the configuration parameters of the stage stored in the file system. -## **Examples** + - `URL`: Specifies the file path or directory in the file storage. -```sql -CREATE TABLE `user` (`id` int(11) ,`user_name` varchar(255) ,`sex` varchar(255)); -INSERT INTO user(id,user_name,sex) values('1', 'weder', 'man'), ('2', 'tom', 'man'), ('3', 'wederTom', 'man'); +- `StageParams (for sub-stage)`: Configuration parameters for sub-stage. + + - `URL`: Specifies the file path or directory in the file storage. --- Create internal data stage -mysql> CREATE STAGE stage1 URL='/tmp' ENABLE = TRUE; +- `COMMENT`: Comment. + +## **Example** + +```sql +#file system +mysql> create stage stage_fs url = 'file:///Users/admin/test'; --- Export data from the table to data stage -mysql> SELECT * FROM user INTO OUTFILE 'stage1:/user.csv'; --- You can see your exported table in your local directory +#substage +mysql> create stage sub_stage url = 'stage://fs_stage/test1/'; --- After setting the data stage, the data table can only be exported to the specified path, and an error will be reported when exporting to other paths -mysql> SELECT * FROM user INTO OUTFILE '~/tmp/csv2/user.txt'; -ERROR 20101 (HY000): internal error: stage exists, please try to check and use a stage instead +#s3 +mysql>create stage stage01 url = 's3://bucket1/test' credentials = {"aws_key_id"='AKIAYOFAMAB7FM7Axxxx',"aws_secret_key"='UjuSDmekK6uXK6CrUs9YhZzY27VOk9W3qMwYxxxx',"AWS_REGION"='us-west-2','PROVIDER'='Amazon', 'ENDPOINT'='s3.us-west-2.amazonaws.com'}; ``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-subscription.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-subscription.md index 7fcb63201..6fdc43197 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-subscription.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-subscription.md @@ -1,8 +1,8 @@ # **CREATE...FROM...PUBLICATION...** -## **Grammar description** +## **Syntax description** -`CREATE...FROM...PUBLICATION...` is a subscription by a subscriber to a publication created by the publisher to get the publisher's shared data. +`CREATE...FROM...PUBLICATION...` means the subscriber subscribes to a publication created by the publisher to obtain the publisher's shared data. ## **Grammar structure** @@ -12,91 +12,47 @@ FROM account_name PUBLICATION pubname; ``` -## Interpretation of grammar +## Grammar explanation - database_name: The name of the database created by the subscriber. -- pubname: The published name of the publisher. -- account_name: Gets the tenant name for this publication. +- pubname: The publication name published by the publisher. +- account_name: The tenant name of the publication can be obtained. -## **Examples** +## **Example** ```sql --- Suppose the system administrator creates a tenant, acc1, as a subscriber. -create account acc1 admin_name 'root' identified by '111'; - --- Assuming session 1 is the publisher, the publisher first publishes a database to the tenant -create database sys_db_1; -use sys_db_1; -create table sys_tbl_1(a int primary key ); -insert into sys_tbl_1 values(1),(2),(3); -create view v1 as (select * from sys_tbl_1); -create publication sys_pub_1 database sys_db_1; -mysql> show publications; -+-------------+----------+---------------------+-------------+-------------+----------+ -| publication | database | create_time | update_time | sub_account | comments | -+-------------+----------+---------------------+-------------+-------------+----------+ -| sys_pub_1 | sys_db_1 | 2024-04-24 11:54:36 | NULL | * | | -+-------------+----------+---------------------+-------------+-------------+----------+ -1 row in set (0.01 sec) - --- A new session is opened, assuming that session 2 is a subscriber who subscribes to the published database -mysql -h 127.0.0.1 -P 6001 -u acc1:root -p --Login to Tenant Account - -create database sub1 from sys publication sys_pub_1; -mysql> show databases; -+--------------------+ -| Database | -+--------------------+ -| information_schema | -| mo_catalog | -| mysql | -| sub1 | -| system | -| system_metrics | -+--------------------+ -6 rows in set (0.01 sec) +create account acc01 admin_name 'root' identified by '111'; +create database db1; +use db1; +create table t1(n1 int); +create table t2(n1 int); + +--Create a publication +create publication db_pub1 database db1 account acc01; +create publication tab_pub1 database db1 table t1,t2 account acc01; + +--Connect tenant acc01 +mysql> show subscriptions all; ++----------+-------------+--------------+------------+-------------+---------------------+----------+----------+--------+ +| pub_name | pub_account | pub_database | pub_tables | pub_comment | pub_time | sub_name | sub_time | status | ++----------+-------------+--------------+------------+-------------+---------------------+----------+----------+--------+ +| tab_pub1 | sys | db1 | t1,t2 | | 2024-10-25 17:06:06 | NULL | NULL | 0 | +| db_pub1 | sys | db1 | * | | 2024-10-25 17:05:54 | NULL | NULL | 0 | ++----------+-------------+--------------+------------+-------------+---------------------+----------+----------+--------+ +2 rows in set (0.00 sec) + +create database db_sub1 from sys publication db_pub1; +create database tab_sub1 from sys publication tab_pub1; mysql> show subscriptions; -+-----------+-------------+--------------+---------------------+----------+---------------------+ -| pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | -+-----------+-------------+--------------+---------------------+----------+---------------------+ -| sys_pub_1 | sys | sys_db_1 | 2024-04-24 11:54:36 | sub1 | 2024-04-24 11:56:05 | -+-----------+-------------+--------------+---------------------+----------+---------------------+ -1 row in set (0.01 sec) - -mysql> use sub1; -Reading table information for completion of table and column names -You can turn off this feature to get a quicker startup with -A - -Database changed -mysql> show tables; -+----------------+ -| Tables_in_sub1 | -+----------------+ -| sys_tbl_1 | -| v1 | -+----------------+ -2 rows in set (0.01 sec) - -mysql> desc sys_tbl_1; -+-------+---------+------+------+---------+-------+---------+ -| Field | Type | Null | Key | Default | Extra | Comment | -+-------+---------+------+------+---------+-------+---------+ -| a | INT(32) | NO | PRI | NULL | | | -+-------+---------+------+------+---------+-------+---------+ -1 row in set (0.01 sec) - -mysql> select * from sys_tbl_1 order by a; -+------+ -| a | -+------+ -| 1 | -| 2 | -| 3 | -+------+ -3 rows in set (0.01 sec) --- Subscription Success ++----------+-------------+--------------+------------+-------------+---------------------+----------+---------------------+--------+ +| pub_name | pub_account | pub_database | pub_tables | pub_comment | pub_time | sub_name | sub_time | status | ++----------+-------------+--------------+------------+-------------+---------------------+----------+---------------------+--------+ +| tab_pub1 | sys | db1 | t1,t2 | | 2024-10-25 17:06:06 | tab_sub1 | 2024-10-25 17:09:24 | 0 | +| db_pub1 | sys | db1 | * | | 2024-10-25 17:05:54 | db_sub1 | 2024-10-25 17:09:23 | 0 | ++----------+-------------+--------------+------------+-------------+---------------------+----------+---------------------+--------+ +2 rows in set (0.00 sec) ``` !!! note - If you need to unsubscribe, you can simply delete the subscribed database name and use [`DROP DATABASE`](drop-database.md). + If you need to cancel the subscription, you can directly delete the subscribed database name and use [`DROP DATABASE`](drop-database.md). \ No newline at end of file diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-pitr.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-pitr.md new file mode 100644 index 000000000..3fbd0cb0e --- /dev/null +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-pitr.md @@ -0,0 +1,31 @@ +# DROP PITR + +## Syntax description + +`DROP PITR` is used to delete the pitr created under the current tenant. + +## Grammar structure + +``` +> DROP PITR pitr_name; +``` + +## Example + +```sql +create pitr db_pitr1 for database db1 range 1 'y'; + +mysql> show pitr where pitr_name='db_pitr1'; ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| pitr_name | created_time | modified_time | pitr_level | account_name | database_name | table_name | pitr_length | pitr_unit | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| db_pitr1 | 2024-10-18 14:26:02 | 2024-10-18 14:26:02 | database | acc1 | db1 | * | 1 | y | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.01 sec) + +mysql> drop pitr db_pitr1; +Query OK, 0 rows affected (0.01 sec) + +mysql> show pitr where pitr_name='db_pitr1'; +Empty set (0.01 sec) +``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-stage.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-stage.md index 842532fd1..8d1b17edd 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-stage.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-stage.md @@ -1,40 +1,19 @@ # **DROP STAGE** -## **Description** +## **Syntax description** -`DROP STAGE` drops a named internal or external data stage from the database. The data stage is a temporary storage area that loads data from a file into a database table or exports data from a database table to a file. Use the `DROP STAGE` command to remove stages of data that are no longer needed from the database, freeing storage space and avoiding additional storage charges. +`DROP STAGE` is used to delete the specified stage in MatrixOne. It should be noted that after deleting a stage, the files in the external storage location associated with the stage will not be removed, only the mapping relationship with the stage will be deleted. -!!! note - The cluster administrator (i.e., the root user) and account administrators can delete data stages. - -## **Syntax** +## **Grammar structure** ``` --- Delete internal stage -> DROP STAGE [IF EXISTS] {internal_stage_name}; - --- Delete external stage -> DROP STAGE [IF EXISTS] {external_stage_name}; +> DROP STAGE [IF EXISTS] {stage_name}; ``` -## **Examples** +## **Example** ```sql -CREATE TABLE `user` (`id` int(11) ,`user_name` varchar(255) ,`sex` varchar(255)); -INSERT INTO user(id,user_name,sex) values('1', 'weder', 'man'), ('2', 'tom', 'man'), ('3', 'wederTom', 'man'); - --- Create internal data stage -mysql> CREATE STAGE stage1 URL='/tmp' ENABLE = TRUE; - --- Export data from the table to data stage -mysql> SELECT * FROM user INTO OUTFILE 'stage1:/user.csv'; --- You can see your exported table in your local directory - --- delete stage1 -mysql> drop stage stage1; -Query OK, 0 rows affected (0.01 sec) +mysql> create stage stage_fs url = 'file:///Users/admin/test'; --- stage1 has been deleted; the data stage is not available -mysql> SELECT * FROM user INTO OUTFILE 'stage1:/user.csv'; -ERROR 20101 (HY000): internal error: stage 'stage1' is not exists, please check +mysql>drop stage stage_fs; ``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/rename-table.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/rename-table.md new file mode 100644 index 000000000..10c7ba453 --- /dev/null +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/rename-table.md @@ -0,0 +1,40 @@ +# **Rename Table** + +## **Syntax description** + +In MatrixOne, the `RENAME TABLE` statement is used to change the name of a table. You can change the names of multiple tables at once. + +Things to note: + +- RENAME TABLE is an atomic operation. If any rename fails, all rename operations are rolled back. +- Tables cannot be renamed across different databases. If you want to rename a table across databases, you can first copy the table to the target database and then delete the original table. +- Before renaming a table, make sure there are no active transactions or locks using the table. + +## **Grammar structure** + +``` +> RENAME TABLE + tbl_name TO new_tbl_name + [, tbl_name2 TO new_tbl_name2] ... +``` + +## **Example** + +```sql +create table old_table1(n1 int); +create table old_table2(n1 int); +create table old_table3(n1 int); + +RENAME TABLE old_table1 TO new_table1; +RENAME TABLE old_table2 TO new_table2,old_table3 TO new_table3; + +mysql> show tables; ++---------------+ +| Tables_in_db1 | ++---------------+ +| new_table1 | +| new_table2 | +| new_table3 | ++---------------+ +3 rows in set (0.00 sec) +``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-pitr.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-pitr.md new file mode 100644 index 000000000..450fdd667 --- /dev/null +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-pitr.md @@ -0,0 +1,262 @@ +# RESTORE ... FROM PITR + +## Syntax description + +`RESTORE ... FROM PITR` is used to restore data from a previously created PITR. + +## Grammar structure + +``` +> RESTORE [CLUSTER]|[[ACCOUNT ] [DATABASE database_name [TABLE table_name]]]FROM PITR [TO ACCOUNT ]; +``` + +## Example + +### Example 1: Restoring the cluster + +```sql + +--Execute under sys tenant +create account acc1 admin_name 'root' identified by '111'; +create account acc2 admin_name 'root' identified by '111'; + +--Executed under tenants acc1, acc2 +create database db1; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| db1 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +6 rows in set (0.01 sec) + +--Create pitr +mysql> create pitr clu_pitr1 for cluster range 1 'h'; +Query OK, 0 rows affected (0.01 sec) + +mysql> show pitr where pitr_name='clu_pitr1'; ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| pitr_name | created_time | modified_time | pitr_level | account_name | database_name | table_name | pitr_length | pitr_unit | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| clu_pitr1 | 2024-10-18 17:06:49 | 2024-10-18 17:06:49 | cluster | * | * | * | 1 | h | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.00 sec) + +--Executed under tenants acc1, acc2 +drop database db1;--Delete database db1 + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +6 rows in set (0.01 sec) + +--recover +mysql> restore cluster from pitr clu_pitr1 "2024-10-18 17:06:50"; +Query OK, 0 rows affected (1.84 sec) + +--Executed under tenants acc1 and acc2, you can see that the recovery is successful. +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| db1 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +6 rows in set (0.01 sec) +``` + +### Example 2: Restoring a tenant + +```sql +CREATE database db1; +CREATE database db2; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| db1 | +| db2 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +7 rows in set (0.00 sec) + +--Create pitr +mysql> create pitr acc_pitr1 range 1 "h"; +Query OK, 0 rows affected (0.02 sec) + +mysql> show pitr; ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| PITR_NAME | CREATED_TIME | MODIFIED_TIME | PITR_LEVEL | ACCOUNT_NAME | DATABASE_NAME | TABLE_NAME | PITR_LENGTH | PITR_UNIT | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| acc_pitr1 | 2024-10-18 16:09:34 | 2024-10-18 16:09:34 | account | acc1 | * | * | 1 | h | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.01 sec) + +--Delete database db1,db2 after a while +drop database db1; +drop database db2; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +5 rows in set (0.01 sec) + +--recover +mysql> restore from pitr acc_pitr1 "2024-10-18 16:09:35"; +Query OK, 0 rows affected (0.78 sec) + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| db1 | +| db2 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +7 rows in set (0.00 sec) +``` + +### Example 3: Restoring the database + +```sql +CREATE database db1; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| db1 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +7 rows in set (0.00 sec) + + +--Create pitr +mysql> create pitr db_pitr1 for database db1 range 2 "d"; +Query OK, 0 rows affected (0.01 sec) + +mysql> show pitr where pitr_name='db_pitr1'; ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| pitr_name | created_time | modified_time | pitr_level | account_name | database_name | table_name | pitr_length | pitr_unit | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| db_pitr1 | 2024-10-18 16:16:03 | 2024-10-18 16:16:03 | database | acc1 | db1 | * | 2 | d | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.01 sec) + +--delete db1 +drop database db1; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +6 rows in set (0.01 sec) + +--recover db1 +mysql> restore database db1 from pitr db_pitr1 "2024-10-18 16:16:05"; +Query OK, 0 rows affected (0.02 sec) + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| db1 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +7 rows in set (0.00 sec) +``` + +### Example 4: Restore table + +```sql +--Executed under tenant acc1 +CREATE TABLE t1(n1 int); +INSERT INTO t1 values(1); + +mysql> SELECT * FROM t1; ++------+ +| n1 | ++------+ +| 1 | ++------+ +1 row in set (0.00 sec) + +create pitr tab_pitr for database db1 table t1 range 1 'd';--create pitr + +mysql> show pitr where pitr_name='tab_pitr'; ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| pitr_name | created_time | modified_time | pitr_level | account_name | database_name | table_name | pitr_length | pitr_unit | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| tab_pitr | 2024-10-23 14:32:17 | 2024-10-23 14:32:17 | table | sys | db1 | t1 | 1 | d | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.01 sec) + +truncate TABLE t1;--truncate t1 + +mysql> SELECT * FROM t1; +Empty set (0.01 sec) + +restore database db1 table t1 from pitr tab_pitr "2024-10-23 14:32:18";--恢复 pitr + +mysql> SELECT * FROM t1;--Recovery successful ++------+ +| n1 | ++------+ +| 1 | ++------+ +1 row in set (0.00 sec) +``` + +## 限制 + +- Deleted tenants do not currently support recovery. +- Cluster-level pitr can restore cluster basic and tenant levels. +- System tenant restoration from a normal tenant to a new tenant only allows tenant level restoration. +- Only the system tenant can perform restore data to the new tenant, and only tenant-level restores are allowed. New tenants need to be created in advance. In order to avoid object conflicts, it is best to create new tenants. \ No newline at end of file diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-account.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-snapshot.md similarity index 58% rename from docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-account.md rename to docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-snapshot.md index f3ad98330..640fb7b39 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-account.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-snapshot.md @@ -1,19 +1,74 @@ -# RESTORE ACCOUNT +# RESTORE ... FROM SNAPSHOT -## Syntax Description +## Syntax description -`RESTORE ACCOUNT` Restores a tenant/database/table to a state corresponding to a timestamp based on a snapshot created under the current tenant. +`RESTORE ... FROM SNAPSHOT` is used to restore data at the cluster/tenant/database/table level from a previously created cluster/tenant level snapshot. -## Syntax structure +## Grammar structure ``` -> RESTORE ACCOUNT account_name [DATABASE database_name [TABLE table_name]] FROM SNAPSHOT snapshot_name [TO ACCOUNT account_name]; +> RESTORE [CLUSTER]|[[ACCOUNT ] [DATABASE database_name [TABLE table_name]]]FROM SNAPSHOT [TO ACCOUNT ]; ``` -## Examples +## Example -- Example 1: Restore tenant to this tenant +### Example 1: Restoring the cluster +```sql +--Executed under tenants acc1, acc2 +create database db1; + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| db1 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +6 rows in set (0.01 sec) + +--Execute under system tenant sys +create snapshot cluster_sp1 for cluster;--Create a snapshot of the cluster + +--Executed under tenants acc1, acc2 +drop database db1;--Delete database db1 + +mysql> show databases; ++--------------------+ +| Database | ++--------------------+ +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +6 rows in set (0.01 sec) + +--Execute under system tenant sys +restore cluster FROM snapshot cluster_sp1;--Perform snapshot recovery on the cluster under the system tenant + +--Executed under tenants acc1, acc2 +mysql> show databases;--Recovery successful ++--------------------+ +| Database | ++--------------------+ +| db1 | +| information_schema | +| mo_catalog | +| mysql | +| system | +| system_metrics | ++--------------------+ +6 rows in set (0.01 sec) +``` + +### Example 2: Restoring a tenant + ```sql --Executed under tenant acc1 CREATE database db1; @@ -33,8 +88,8 @@ mysql> show databases; +--------------------+ 7 rows in set (0.00 sec) -create snapshot acc1_snap1 for account acc1;--Creating a Snapshot -drop database db1;--Delete databases db1,db2 +create snapshot acc1_snap1 for account acc1;--Create snapshot +drop database db1;--Delete database db1,db2 drop database db2; mysql> show databases; @@ -49,9 +104,9 @@ mysql> show databases; +--------------------+ 5 rows in set (0.01 sec) -restore account acc1 FROM snapshot acc1_snap1;--Restore tenant-level snapshots +restore account acc1 FROM snapshot acc1_snap1;--Restoring a tenant-level snapshot -mysql> show databases;--Successful recovery +mysql> show databases;--Recovery successful +--------------------+ | Database | +--------------------+ @@ -66,7 +121,7 @@ mysql> show databases;--Successful recovery 7 rows in set (0.01 sec) ``` -- Example 2: Restore database to this tenant +### Example 3: Restoring the database ```sql --Executed under tenant acc1 @@ -85,7 +140,7 @@ mysql> show databases; +--------------------+ 7 rows in set (0.00 sec) -create snapshot acc1_db_snap1 for account acc1;--Creating a Snapshot +create snapshot acc1_db_snap1 for account acc1;--Create snapshot drop database db1;--Delete database db1 mysql> show databases; @@ -100,9 +155,9 @@ mysql> show databases; +--------------------+ 6 rows in set (0.01 sec) -restore account acc1 database db1 FROM snapshot acc1_db_snap1;--Recovering database-level snapshots +restore account acc1 database db1 FROM snapshot acc1_db_snap1;--Restoring a database-level snapshot -mysql> show databases;--Successful recovery +mysql> show databases;--Recovery successful +--------------------+ | Database | +--------------------+ @@ -116,10 +171,10 @@ mysql> show databases;--Successful recovery 7 rows in set (0.00 sec) ``` -- Example 3: Restore table to this tenant +### Example 4: Restoring a table ```sql ---Executed under tenant acc1 +--在租户 acc1 下执行 CREATE TABLE t1(n1 int); INSERT INTO t1 values(1); @@ -131,15 +186,15 @@ mysql> SELECT * FROM t1; +------+ 1 row in set (0.00 sec) -create snapshot acc1_tab_snap1 for account acc1;--Creating a Snapshot +create snapshot acc1_tab_snap1 for account acc1;--Create snapshot truncate TABLE t1;--Clear t1 mysql> SELECT * FROM t1; Empty set (0.01 sec) -restore account acc1 database db1 TABLE t1 FROM snapshot acc1_tab_snap1;--Restore Snapshot +restore account acc1 database db1 TABLE t1 FROM snapshot acc1_tab_snap1;--Restore snapshot -mysql> SELECT * FROM t1;--Successful recovery +mysql> SELECT * FROM t1;--Recovery successful +------+ | n1 | +------+ @@ -148,7 +203,7 @@ mysql> SELECT * FROM t1;--Successful recovery 1 row in set (0.00 sec) ``` -- Example 4: System tenant restores normal tenant to normal tenant This tenant +### Example 5: System tenant restores ordinary tenant to ordinary tenant ```sql --Executed under tenant acc1 @@ -168,7 +223,7 @@ mysql> show databases; 6 rows in set (0.01 sec) --Execute under system tenant sys -create snapshot acc1_snap1 for account acc1;--Creating a snapshot for acc1 +create snapshot acc1_snap1 for account acc1;--Create a snapshot for acc1 --Executed under tenant acc1 drop database db1;--Delete database db1 @@ -186,10 +241,10 @@ mysql> show databases; 6 rows in set (0.01 sec) --Execute under system tenant sys -restore account acc1 FROM snapshot acc1_snap1 TO account acc1;--Snapshot recovery of acc1 under system tenant +restore account acc1 FROM snapshot acc1_snap1 TO account acc1;--在系统租户下对 acc1 进行快照恢复 --Executed under tenant acc1 -mysql> show databases;--Successful recovery +mysql> show databases;--Recovery successful +--------------------+ | Database | +--------------------+ @@ -203,10 +258,10 @@ mysql> show databases;--Successful recovery 6 rows in set (0.01 sec) ``` -- Example 5: System tenant restores normal tenant to new tenant +### Example 6: System tenant restores normal tenant to new tenant ```sql ---Executed under tenant acc1 +--Execute under tenant acc1 create database db1; mysql> show databases; @@ -223,7 +278,7 @@ mysql> show databases; 6 rows in set (0.01 sec) --Execute under system tenant sys -create snapshot acc1_snap1 for account acc1;--Creating a snapshot for acc1 +create snapshot acc1_snap1 for account acc1;--Create a snapshot for acc1 --Executed under tenant acc1 drop database db1;--Delete db1 @@ -241,8 +296,8 @@ mysql> show databases; 6 rows in set (0.01 sec) --Execute under system tenant sys -create account acc2 ADMIN_NAME admin IDENTIFIED BY '111';--Need to create new tenants to be targeted in advance -restore account acc1 FROM snapshot acc1_snap1 TO account acc2;--Snapshot recovery of acc1 under system tenant to acc2 +create account acc2 ADMIN_NAME admin IDENTIFIED BY '111';--New tenants to be targeted need to be created in advance +restore account acc1 FROM snapshot acc1_snap1 TO account acc2;--Perform snapshot recovery on acc1 under the system tenant and restore it to acc2 --Executed under tenant acc1 mysql> show databases; @@ -272,10 +327,10 @@ mysql> show databases;--Revert to acc2 6 rows in set (0.01 sec) ``` -## Limitations +## limit -- Currently only tenant/database/table level recovery is supported, not clustered. +- Only tenant-level snapshots can be restored at the database/table level. -- System tenant recovery normal tenant to new tenant allows only tenant level recovery. +- System tenant restoration from a normal tenant to a new tenant only allows tenant level restoration. -- Only system tenants can perform restore data to a new tenant, and only tenant-level restores are allowed. New tenants need to be created in advance, and in order to avoid object conflicts, it is best to have a new tenant. \ No newline at end of file +- Only the system tenant can perform restore data to the new tenant, and only tenant-level restores are allowed. New tenants need to be created in advance. In order to avoid object conflicts, it is best to create new tenants. \ No newline at end of file diff --git a/docs/MatrixOne/Reference/SQL-Reference/Data-Manipulation-Language/load-data-infile.md b/docs/MatrixOne/Reference/SQL-Reference/Data-Manipulation-Language/load-data-infile.md index ccec82753..f50decd17 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Data-Manipulation-Language/load-data-infile.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Data-Manipulation-Language/load-data-infile.md @@ -8,7 +8,7 @@ The LOAD DATA statement reads rows from a text file into a table at a very high ``` > LOAD DATA [LOCAL] - INFILE 'file_name' + INFILE '|' INTO TABLE tbl_name [CHARACTER SET charset_name] [{FIELDS | COLUMNS} @@ -24,7 +24,6 @@ The LOAD DATA statement reads rows from a text file into a table at a very high [SET column_name_1=nullif(column_name_1, expr1), column_name_2=nullif(column_name_2, expr2)...] [PARALLEL {'TRUE' | 'FALSE'}] [STRICT {'TRUE' | 'FALSE'}] - ``` ### Input File Location @@ -705,6 +704,63 @@ As you can see, the query result ignores the first line. For more information on loding *JSONLines*, see [Import the JSONLines data](../../../Develop/import-data/bulk-load/load-jsonline.md). +### Example 3: LOAD Stage + +#### Simple import example + +There is a file `t1.csv` in the `/Users/admin/test` directory: + +```bash +(base) admin@192 test % cat t1.csv +1 a +2 b +3 c +``` + +```sql +create table t1(n1 int,n2 varchar(10)); +create stage stage_fs url = 'file:///Users/admin/test'; +load data infile 'stage://stage_fs/t1.csv' into table t1; + +mysql> select * from t1; ++------+------+ +| n1 | n2 | ++------+------+ +| 1 | a | +| 2 | b | +| 3 | c | ++------+------+ +3 rows in set (0.01 sec) +``` + +#### Add conditional import example + +You can add IGNORE 1 LINES at the end of the LOAD DATA statement to skip the first line of the data file. + +There is a file `t1.csv` in the `/Users/admin/test` directory: + +```bash +(base) admin@192 test % cat t1.csv +1 a +2 b +3 c +``` + +```sql +create table t2(n1 int,n2 varchar(10)); +create stage stage_fs1 url = 'file:///Users/admin/test'; +load data infile 'stage://stage_fs1/t1.csv' into table t2 ignore 1 lines; + +mysql> select * from t2; ++------+------+ +| n1 | n2 | ++------+------+ +| 2 | b | +| 3 | c | ++------+------+ +2 rows in set (0.00 sec) +``` + ## **Constraints** 1. The `REPLACE` and `IGNORE` modifiers control handling of new (input) rows that duplicate existing table rows on unique key values (`PRIMARY KEY` or `UNIQUE index` values) are not supported in MatrixOne yet. diff --git a/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-pitrs.md b/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-pitrs.md new file mode 100644 index 000000000..96b882580 --- /dev/null +++ b/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-pitrs.md @@ -0,0 +1,37 @@ +# SHOW PITR + +## Syntax description + +`SHOW PITR` returns information about PITR created under the current tenant. + +## Grammar structure + +``` +> SHOW PITR[WHERE expr] +``` + +## Example + +```sql +create pitr account_pitr1 range 2 "h"; +create pitr db_pitr1 for database db1 range 1 'y'; +create pitr tab_pitr1 for database db1 table t1 range 1 'y'; + +mysql> show pitr; ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| PITR_NAME | CREATED_TIME | MODIFIED_TIME | PITR_LEVEL | ACCOUNT_NAME | DATABASE_NAME | TABLE_NAME | PITR_LENGTH | PITR_UNIT | ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| tab_pitr1 | 2024-10-18 14:28:53 | 2024-10-18 14:28:53 | table | acc1 | db1 | t1 | 1 | y | +| db_pitr1 | 2024-10-18 14:26:02 | 2024-10-18 14:26:02 | database | acc1 | db1 | * | 1 | y | +| account_pitr1 | 2024-10-18 14:23:12 | 2024-10-18 14:23:12 | account | acc1 | * | * | 2 | h | ++---------------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +3 rows in set (0.01 sec) + +mysql> show pitr where pitr_name='tab_pitr1'; ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| pitr_name | created_time | modified_time | pitr_level | account_name | database_name | table_name | pitr_length | pitr_unit | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +| tab_pitr1 | 2024-10-18 14:28:53 | 2024-10-18 14:28:53 | table | acc1 | db1 | t1 | 1 | y | ++-----------+---------------------+---------------------+------------+--------------+---------------+------------+-------------+-----------+ +1 row in set (0.00 sec) +``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-publications.md b/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-publications.md index 8b9c012d9..5a1dcd0d1 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-publications.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-publications.md @@ -1,18 +1,18 @@ # **SHOW PUBLICATIONS** -## **Description** +## **Syntax description** -Returns a list of all publish names, the database name of the publish, the publish creation time, the publish last modification time, a list of the tenant names specified by the publish (show "*" if all), and notes. +Returns information such as all publication names, published database/table names, publication creation time, publication latest modification time, and a list of tenant names specified by the publication (if all, "*" is displayed). -For more information, you need to have tenant administrator privileges and view the system table mo_pubs to see more parameters. +To view more information, you need to have tenant administrator rights and view the system table mo_pubs to view more parameters. -## **Syntax** +## **Grammar structure** ``` SHOW PUBLICATIONS; ``` -## **Examples** +## **Example** ```sql create account acc0 admin_name 'root' identified by '111'; @@ -22,10 +22,10 @@ create database t; create publication pub3 database t account acc0,acc1; mysql> show publications; -+-------------+----------+---------------------+-------------+-------------+----------+ -| publication | database | create_time | update_time | sub_account | comments | -+-------------+----------+---------------------+-------------+-------------+----------+ -| pub3 | t | 2024-04-23 10:10:59 | NULL | acc0,acc1 | | -+-------------+----------+---------------------+-------------+-------------+----------+ ++-------------+----------+--------+-------------+---------------------+---------------------+-------------+----------+ +| publication | database | tables | sub_account | subscribed_accounts | create_time | update_time | comments | ++-------------+----------+--------+-------------+---------------------+---------------------+-------------+----------+ +| pub3 | t | * | acc0,acc1 | | 2024-10-25 16:36:04 | NULL | | ++-------------+----------+--------+-------------+---------------------+---------------------+-------------+----------+ 1 row in set (0.00 sec) ``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-stage.md b/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-stage.md index 9cb9d6fe3..70202b17f 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-stage.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-stage.md @@ -1,36 +1,26 @@ # **SHOW STAGES** -## **Description** +## **Syntax description** -Display the data stage of the current database creation as a list. +`SHOW STAGES` returns stage specific information. -MatrixOne users use `SHOW STAGES` to view all the data stages of the current database and can choose a valid path to export the data to. - -## **Syntax** +## **Grammar structure** ``` > SHOW STAGES [LIKE 'pattern'] ``` -## **Examples** +## **Example** ```sql -CREATE TABLE `user` (`id` int(11) ,`user_name` varchar(255) ,`sex` varchar(255)); -INSERT INTO user(id,user_name,sex) values('1', 'weder', 'man'), ('2', 'tom', 'man'), ('3', 'wederTom', 'man'); - --- Create internal data stage -mysql> CREATE STAGE stage1 URL='/tmp' ENABLE = TRUE; - --- Export data from table to the data stage -mysql> SELECT * FROM user INTO OUTFILE 'stage1:/user.csv'; --- You can see your exported table in your local directory - --- Display the data stage of the current database creation as a list. -mysql> SHOW STAGES; -+------------+-----------------------------+---------+---------+ -| STAGE_NAME | URL | STATUS | COMMENT | -+------------+-----------------------------+---------+---------+ -| stage1 | /tmp | ENABLED | | -+------------+-----------------------------+---------+---------+ +mysql> create stage stage_fs url = 'file:///Users/admin/test'; +Query OK, 0 rows affected (0.03 sec) + +mysql> show stages; ++------------+--------------------------+----------+---------+ +| STAGE_NAME | URL | STATUS | COMMENT | ++------------+--------------------------+----------+---------+ +| stage_fs | file:///Users/admin/test | DISABLED | | ++------------+--------------------------+----------+---------+ 1 row in set (0.00 sec) ``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-subscriptions.md b/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-subscriptions.md index 4d06a90d0..7a9e23cb4 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-subscriptions.md +++ b/docs/MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-subscriptions.md @@ -1,50 +1,69 @@ # **SHOW SUBSCRIPTIONS** -## **Description** +## **Syntax description** -Returns all publish names, publish tenant names, published database names, times published to that tenant, subscription names, and times when that subscription was created. +Returns all publication names, publishing tenant names, published database names, published table names, comments, time published to the tenant, subscription name, time when the subscription was created, and subscription status (0: normal subscription possible; 1: publication exists , but no subscription rights; 2: Originally subscribed to the publication, but deleted the publication). -## **Syntax** +## **Grammar structure** ``` SHOW SUBSCRIPTIONS [ALL]; ``` -## **Syntax explanation** +## **Grammar explanation** -- The **ALL** option allows you to see all subscriptions with permissions, unsubscribed sub_time, sub_name is null, without **ALL** you can only see the published information of subscribed ones. +-The **ALL**option allows you to see all subscriptions with permissions. For unsubscribed sub_time, sub_name is null. Without **ALL**, you can only see subscribed publication information. -## **Examples** +## **Example** ```sql + +--Execute in sys tenant + +create account acc01 admin_name 'root' identified by '111'; +create account acc02 admin_name 'root' identified by '111'; +create database db1; +use db1; +create table t1(n1 int); +create table t2(n1 int); + +--Database level publishing +create publication db_pub1 database db1 account acc01,acc02; + +--Table level publishing +create publication tab_pub1 database db1 table t1,t2 account acc01,acc02; + +-- Executed in acc01 tenant mysql> show subscriptions all; -+----------+-------------+--------------+---------------------+----------+----------+ -| pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | -+----------+-------------+--------------+---------------------+----------+----------+ -| pub3 | sys | t | 2024-04-23 11:11:06 | NULL | NULL | -+----------+-------------+--------------+---------------------+----------+----------+ -1 row in set (0.01 sec) ++----------+-------------+--------------+------------+-------------+---------------------+----------+----------+--------+ +| pub_name | pub_account | pub_database | pub_tables | pub_comment | pub_time | sub_name | sub_time | status | ++----------+-------------+--------------+------------+-------------+---------------------+----------+----------+--------+ +| tab_pub1 | sys | db1 | t1,t2 | | 2024-10-14 19:00:21 | NULL | NULL | 0 | +| db_pub1 | sys | db1 | * | | 2024-10-14 19:00:16 | NULL | NULL | 0 | ++----------+-------------+--------------+------------+-------------+---------------------+----------+----------+--------+ +2 rows in set (0.00 sec) mysql> show subscriptions; Empty set (0.00 sec) -mysql> create database sub3 from sys publication pub3; -Query OK, 1 row affected (0.02 sec) +create database db_sub1 from sys publication db_pub1; +create database tab_sub1 from sys publication tab_pub1; mysql> show subscriptions all; -+----------+-------------+--------------+---------------------+----------+---------------------+ -| pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | -+----------+-------------+--------------+---------------------+----------+---------------------+ -| pub3 | sys | t | 2024-04-23 11:11:06 | sub3 | 2024-04-23 11:12:11 | -+----------+-------------+--------------+---------------------+----------+---------------------+ -1 row in set (0.00 sec) ++----------+-------------+--------------+------------+-------------+---------------------+----------+---------------------+--------+ +| pub_name | pub_account | pub_database | pub_tables | pub_comment | pub_time | sub_name | sub_time | status | ++----------+-------------+--------------+------------+-------------+---------------------+----------+---------------------+--------+ +| tab_pub1 | sys | db1 | t1,t2 | | 2024-10-14 19:00:21 | tab_sub1 | 2024-10-14 19:01:41 | 0 | +| db_pub1 | sys | db1 | * | | 2024-10-14 19:00:16 | db_sub1 | 2024-10-14 19:01:30 | 0 | ++----------+-------------+--------------+------------+-------------+---------------------+----------+---------------------+--------+ +2 rows in set (0.00 sec) mysql> show subscriptions; -+----------+-------------+--------------+---------------------+----------+---------------------+ -| pub_name | pub_account | pub_database | pub_time | sub_name | sub_time | -+----------+-------------+--------------+---------------------+----------+---------------------+ -| pub3 | sys | t | 2024-04-23 11:11:06 | sub3 | 2024-04-23 11:12:11 | -+----------+-------------+--------------+---------------------+----------+---------------------+ -1 row in set (0.01 sec) - -``` \ No newline at end of file ++----------+-------------+--------------+------------+-------------+---------------------+----------+---------------------+--------+ +| pub_name | pub_account | pub_database | pub_tables | pub_comment | pub_time | sub_name | sub_time | status | ++----------+-------------+--------------+------------+-------------+---------------------+----------+---------------------+--------+ +| tab_pub1 | sys | db1 | t1,t2 | | 2024-10-14 19:00:21 | tab_sub1 | 2024-10-14 19:01:41 | 0 | +| db_pub1 | sys | db1 | * | | 2024-10-14 19:00:16 | db_sub1 | 2024-10-14 19:01:30 | 0 | ++----------+-------------+--------------+------------+-------------+---------------------+----------+---------------------+--------+ +2 rows in set (0.00 sec) +``` diff --git a/docs/MatrixOne/Reference/SQL-Reference/SQL-Type.md b/docs/MatrixOne/Reference/SQL-Reference/SQL-Type.md index 134e1c73f..cb0da8144 100644 --- a/docs/MatrixOne/Reference/SQL-Reference/SQL-Type.md +++ b/docs/MatrixOne/Reference/SQL-Reference/SQL-Type.md @@ -12,8 +12,11 @@ Data Definition Language (DDL) is a subset of DBMS language used for defining da - [CREATE INDEX](Data-Definition-Language/create-index.md) - [CREATE TABLE](Data-Definition-Language/create-table.md) - [CREATE EXTERNAL TABLE](Data-Definition-Language/create-external-table.md) +- [CREATE FUNCTION](Data-Definition-Language/create-function-sql.md) +- [CREATE PITR](Data-Definition-Language/create-pitr.md) - [CREATE PUBLICATION](Data-Definition-Language/create-publication.md) - [CREATE SEQUENCE](Data-Definition-Language/create-sequence.md) +- [CREATE SNAPSHOT](Data-Definition-Language/create-snapshot.md) - [CREATE STAGE](Data-Definition-Language/create-stage.md) - [CREATE...FROM...PUBLICATION...](Data-Definition-Language/create-subscription.md) - [CREATE VIEW](Data-Definition-Language/create-view.md) @@ -22,8 +25,11 @@ Data Definition Language (DDL) is a subset of DBMS language used for defining da - [DROP INDEX](Data-Definition-Language/drop-index.md) - [DROP TABLE](Data-Definition-Language/drop-table.md) +- [DROP FUNCTION](Data-Definition-Language/drop-function.md) +- [DROP PITR](Data-Definition-Language/drop-pitr.md) - [DROP PUBLICATION](Data-Definition-Language/drop-publication.md) - [DROP SEQUENCE](Data-Definition-Language/drop-sequence.md) +- [DROP SNAPSHOT](Data-Definition-Language/drop-snapshot.md) - [DROP STAGE](Data-Definition-Language/drop-stage.md) - [DROP VIEW](Data-Definition-Language/drop-view.md) @@ -31,6 +37,9 @@ Data Definition Language (DDL) is a subset of DBMS language used for defining da - [ALTER TABLE](Data-Definition-Language/alter-table.md) - [ALTER PUBLICATION](Data-Definition-Language/alter-publication.md) +- [ALTER PITR](Data-Definition-Language/alter-pitr.md) +- [ALTER REINDEX](Data-Definition-Language/alter-reindex.md) +- [ALTER SEQUENCE](Data-Definition-Language/alter-sequence.md) - [ALTER STAGE](Data-Definition-Language/alter-stage.md) - [ALTER VIEW](Data-Definition-Language/alter-view.md) @@ -38,6 +47,15 @@ Data Definition Language (DDL) is a subset of DBMS language used for defining da - [TRUNCATE TABLE](Data-Definition-Language/truncate-table.md) +### RENAME 语句 + +- [RENAME TABLE](Data-Definition-Language/rename-table.md) + +### RESTORE 语句 + +- [RESTORE SNAPSHOT](Data-Definition-Language/restore-snapshot.md) +- [RESTORE PITR](Data-Definition-Language/restore-pitr.md) + ## DML - Data Manipulation Language Data Manipulation Language (DML) is used for database operations, including programming statements to work with database objects and data. In MatrixOne, DML is categorized as follows: diff --git a/docs/MatrixOne/Reference/System-tables.md b/docs/MatrixOne/Reference/System-tables.md index 2c948e1c4..f3db71737 100644 --- a/docs/MatrixOne/Reference/System-tables.md +++ b/docs/MatrixOne/Reference/System-tables.md @@ -168,22 +168,37 @@ The concept of multi-tenancy was introduced with MatrixOne version 0.6, and the | variable_value | VARCHAR(5000) | The value of the variable | | system_variables | BOOL(0) | if it is a system variable (compatibility variables are added in addition to system variables) | -### mo_pubs table +### `mo_pubs` table -| column | type | comments | -| -----------------| --------------- | ----------------- | -| pub_name | VARCHAR(64) | publication name| -| database_name | VARCHAR(5000) | The name of the published data | -| database_id | BIGINT UNSIGNED(64) | ID of the publishing database, corresponding to dat_id in the mo_database table | +| Column Properties | Type | Description | +| ------------------| ---------------| -----------------| +| pub_name | VARCHAR(64) | Publish name | +| database_name | VARCHAR(5000) | Name of published data | +| database_id | BIGINT UNSIGNED(64) | The ID of the publishing database, corresponding to the dat_id in the mo_database table | | all_table | BOOL(0) | Whether the publishing library contains all tables in the database corresponding to database_id | -| all_account | BOOL(0) | Whether all accounts can subscribe to the library | -| table_list | TEXT(0) | When it is not all table, publish the list of tables contained in the library, and the table name corresponds to the table under the database corresponding to database_id| -| account_list | TEXT(0) |Account list that is allowed to subscribe to the publishing library when it is not all accounts| -| created_time | TIMESTAMP(0) | Time when the release repository was created | -| owner | INT UNSIGNED(32) | The role ID corresponding to the creation of the release library | -| creator | INT UNSIGNED(32) | The ID of the user who created the release library | +| table_list | TEXT(0) | When it is not all table, publish the list of tables contained in the library. The table name corresponds one-to-one with the table under the database corresponding to database_id | +| account_list | TEXT(0) |When non-all accounts are used, the list of accounts allowed to subscribe to the publication library| +| created_time | TIMESTAMP(0) |The time when the release library was created | +| updated_time | TIMESTAMP(0) |Time to update the release library | +| owner | INT UNSIGNED(32) | Create the role ID corresponding to the release library | +| creator | INT UNSIGNED(32) | Create the user ID corresponding to the release library | | comment | TEXT(0) | Remarks for creating a release library | +### `mo_subs` table + +| Column Properties | Type | Description | +| ---------------------| ---------------| -----------------| +| sub_account_id | INT(32) | Subscription tenant id| +| sub_name | VARCHAR(5000) | subscription name | +| sub_time | TIMESTAMP(0) | Subscription time | +| pub_account_name |VARCHAR(300) | The name of the publishing tenant | +| pub_name | VARCHAR(64) | Publish name | +| pub_database | VARCHAR(5000) |Published database| +| pub_tables | TEXT(0) |Published tables | +| pub_time | TIMESTAMP(0) | Published time | +| pub_comment | TEXT(0) | Comments from the publishing library | +| status | TINYINT(8) | Subscription status, 0 means normal subscription, 1 means the publication exists but does not have subscription permission, 2 means the publication was originally subscribed but was deleted | + ### mo_stages table | column | type | comments | diff --git a/docs/MatrixOne/Reference/mo-tools/mo_ctl_standalone.md b/docs/MatrixOne/Reference/mo-tools/mo_ctl_standalone.md index aae251b83..67e2d1fb7 100644 --- a/docs/MatrixOne/Reference/mo-tools/mo_ctl_standalone.md +++ b/docs/MatrixOne/Reference/mo-tools/mo_ctl_standalone.md @@ -150,12 +150,12 @@ mo_ctl deploy help Usage : mo_ctl deploy [mo_version] [force] # deploy mo onto the path configured [mo_version]: optional, specify an mo version to deploy [force] : optional, if specified will delete all content under MO_PATH and deploy from beginning - e.g. : mo_ctl deploy # default, same as mo_ctl deploy 1.2.4 + e.g. : mo_ctl deploy # default, same as mo_ctl deploy 2.0.0 : mo_ctl deploy main # deploy development latest version : mo_ctl deploy d29764a # deploy development version d29764a - : mo_ctl deploy 1.2.4 # deploy stable verson 1.2.4 - : mo_ctl deploy force # delete all under MO_PATH and deploy verson 1.2.4 - : mo_ctl deploy 1.2.4 force # delete all under MO_PATH and deploy stable verson 1.2.4 from beginning + : mo_ctl deploy 2.0.0 # deploy stable verson 2.0.0 + : mo_ctl deploy force # delete all under MO_PATH and deploy verson 2.0.0 + : mo_ctl deploy 2.0.0 force # delete all under MO_PATH and deploy stable verson 2.0.0 from beginning ``` ### start - Starts the MatrixOne service @@ -295,7 +295,7 @@ Using `mo_ctl get_conf` will print a list of all parameters used by the current | GCC_VERSION | The version of gcc that precheck checks |default 8.5.0 | | GO_VERSION | The go version of the precheck check |default 1.22.3 | | MO_GIT_URL | MatrixOne source code pulling address | default | -| MO_DEFAULT_VERSION | The version of MatrixOne that is pulled by default | default 1.2.4 | +| MO_DEFAULT_VERSION | The version of MatrixOne that is pulled by default | default 2.0.0 | | GOPROXY | GOPROXY address, generally used for domestic accelerated pull golang dependencies | default ,direct | | STOP_INTERVAL | Stop interval, wait time to detect service status after stopping service | default 5 seconds | | START_INTERVAL | Startup interval, wait time to detect service status after starting the service | default 2 seconds | @@ -348,11 +348,11 @@ MatrixOne 0.8 and later can use `mo_ctl upgrade version` or `mo_ctl upgrade comm ``` mo_ctl upgrade help Usage : mo_ctl upgrade [version_commitid] # upgrade or downgrade mo from current version to a target commit id or stable version - [commitid] : a commit id such as '38888f7', or a stable version such as '1.2.4' + [commitid] : a commit id such as '38888f7', or a stable version such as '2.0.0' : use 'latest' to upgrade to latest commit on main branch if you don't know the id e.g. : mo_ctl upgrade 38888f7 # upgrade/downgrade to commit id 38888f7 on main branch : mo_ctl upgrade latest # upgrade/downgrade to latest commit on main branch - : mo_ctl upgrade 1.2.4 # upgrade/downgrade to stable version 1.2.4 + : mo_ctl upgrade 2.0.0 # upgrade/downgrade to stable version 2.0.0 ``` ### watchdog - Keep MatrixOne alive diff --git a/docs/MatrixOne/Release-Notes/release-timeline.md b/docs/MatrixOne/Release-Notes/release-timeline.md index 4e36a1c01..ff6e89b1b 100644 --- a/docs/MatrixOne/Release-Notes/release-timeline.md +++ b/docs/MatrixOne/Release-Notes/release-timeline.md @@ -4,6 +4,7 @@ This document shows all the released MatrixOne versions in reverse chronological | **Version** | **Release Date** | | :-------------------------- | :--------------- | +| [v24.2.0.0](v24.2.0.0.md) | 2024/10/31 | | [v24.1.2.4](v24.1.2.4.md) | 2024/09/23 | | [v24.1.2.3](v24.1.2.3.md) | 2024/09/11 | | [v24.1.2.2](v24.1.2.2.md) | 2024/07/12 | diff --git a/docs/MatrixOne/Release-Notes/v24.2.0.0.md b/docs/MatrixOne/Release-Notes/v24.2.0.0.md new file mode 100644 index 000000000..3d7206d5d --- /dev/null +++ b/docs/MatrixOne/Release-Notes/v24.2.0.0.md @@ -0,0 +1,129 @@ +# MatrixOne v24.2.0.0 Release Notes + +We are thrilled to announce the release of MatrixOne v24.2.0.0 on 2024/10/31! + +## What is Matrixone? + +MatrixOne is a hyper-converged cloud-native database. It is designed to provide a cloud-native, high-performance, highly scalable, MySQL-compatible HTAP database. MatrixOne enables users to handle mixed workloads such as transactions, analytics, time-series, and streaming processing through a one-stop data processing solution. + +

+ MatrixOne +

+ +## Feature overview + +In this new version, MatrixOne provides additional features for AIGC applications, along with significant improvements in enterprise-grade high availability and disaster recovery. + +This iteration brings substantial enhancements to MatrixOne's support for generative AI applications, disaster recovery, and system stability. Key features include access to data on external storage, capabilities for handling unstructured data, full-text search functionality, improved vector search performance, support for snapshot backups and point-in-time recovery (PiTR), CDC, and disaster recovery through log-based replication for primary-standby clusters. Additionally, compatibility with MySQL has been further enhanced. With these new features, MatrixOne is steadily becoming an ideal choice for enterprises building AI-driven intelligent data management platforms. + +## Application scenarios + +MatrixOne is suitable for the following application scenarios. We warmly invite users with the following business challenges and needs to contact us for trial testing. + +### Generative artificial intelligence scenarios + +MatrixOne hyper-converged database offers robust multimodal data support, real-time retrieval, and intelligent data processing capabilities for Generative AI, establishing a core infrastructure for such applications. In multimodal scenarios like text and image generation, MatrixOne ensures rapid response and high-quality generation on large datasets through efficient data management, vector and hybrid search, data cleansing and preprocessing supported by Python UDFs, and GPU-accelerated real-time inference. Whether for large-scale data access and storage, online inference, or dynamic feedback, MatrixOne provides stable, low-latency support for Generative AI applications, enabling enterprises to swiftly implement, iterate, and optimize AI-driven innovation. + +### Time series data application scenarios + +In modern IoT applications, billions of devices and sensors continuously collect and transmit data from industrial production lines, smart grids, smart city infrastructure, autonomous vehicles, and more, generating TBs of real-time data daily. MatrixOne hyper-converged database provides efficient real-time data processing for IoT scenarios, supporting millisecond-level high-concurrency writes and rapid retrieval with excellent scalability to handle peak loads. Its real-time analytics enable enterprises to quickly derive critical insights from vast IoT data, seamlessly integrating with machine learning models to directly stream real-time data into models for prediction and anomaly detection. This makes it ideal for applications such as predictive maintenance, energy efficiency optimization, and intelligent monitoring, fully meeting IoT application requirements for high throughput, low latency, and intelligent data management. + +### Mixed load type application support + +In enterprise systems like OA, ERP, and CRM, the growth in data volume and business complexity often makes it challenging for traditional single-node databases to meet peak performance demands, especially during critical periods like month-end or quarter-end. At these times, frequent analysis and real-time statistical reporting are essential for decision-making support. Many companies address this by configuring independent analytical databases or adopting database sharding to alleviate query load on the primary database. However, MatrixOne's support for hybrid workloads enables enterprises to fulfill both operational and analytical needs within a single database, eliminating the need for additional systems. Real-time data analytics allows for quick responses even under high concurrency, while MatrixOne's scalability enables seamless expansion as business needs grow, maintaining efficient real-time querying and reporting even with large data volumes. This ensures that data-driven decision-making remains real-time, continuous, and efficient, significantly enhancing flexibility in data management. + +### Enterprise-level SaaS scenario + +With the rapid growth of enterprise-level SaaS applications, SaaS development must address the needs of a multi-tenant model. Traditional solutions typically choose between a shared database instance for multiple tenants and a dedicated instance per tenant, leading to trade-offs between management costs and tenant isolation. MatrixOne natively supports a multi-tenant architecture, providing load isolation and independent scalability for each tenant along with unified management capabilities. This architecture effectively reduces management costs, ensures data isolation, and improves operational efficiency, fully meeting SaaS applications' demands for cost control, simplified management, and isolation. This makes MatrixOne an ideal database choice for SaaS applications. + +## Key new features + +### Multi-mode data management capabilities + +MatrixOne supports direct access to external object storage, remote file systems, and local file systems through the Stage object, as well as direct file access on storage systems via the datalink type. This capability is highly beneficial for building data pipelines in generative AI applications with MatrixOne, significantly improving development efficiency and reducing application maintenance costs. + +### Full-text indexing for text or JSON type data + +Creating full-text indexes on JSON or TEXT columns in tables can effectively enhance MatrixOne's performance in AIoT applications. Leveraging MatrixOne's JSON data type can further reduce data redundancy, thereby strengthening MatrixOne's competitiveness in AIoT scenarios. + +### Vector search + +In this iteration, MatrixOne has optimized its vector search capabilities, enabling rapid vector distance-based retrieval within large-scale vector datasets. This efficient retrieval is especially critical for generative AI applications based on large language models (LLMs) and retrieval-augmented generation (RAG) technologies. + +### Snapshot-based backup and recovery + +Creating data snapshots for clusters or tenants allows the database's state at a specific moment to be quickly captured, ensuring rapid recovery in case of failure or emergency. Snapshot technology has minimal impact on system performance while guaranteeing data consistency for complete recovery. It also supports cross-tenant restoration, enhancing the system's disaster recovery capabilities. + +### Disaster recovery of active and standby clusters based on log replication + +Using a log replication mechanism to synchronize transaction logs from the primary database to the standby database enables high availability and disaster recovery for the primary-standby cluster. When the primary database encounters a failure, the standby database can quickly take over, ensuring uninterrupted business operations. + +### Point-in-time recovery capabilities + +By recording all data changes after the initial snapshot, this feature allows users to restore the database to a precise historical moment in the event of failure, accidental operations, or data corruption, preventing the loss of important information. Compared to traditional full backups, it significantly reduces storage costs and improves recovery efficiency. This feature provides flexibility and security for critical business scenarios, enabling rapid recovery to meet business continuity and compliance requirements. + +### MatrixOne to MySQL CDC + +By capturing changes in the MatrixOne database and synchronizing them in real-time to downstream MySQL, data disaster recovery from MatrixOne to MySQL is achieved. This allows users migrating from MySQL to MatrixOne to maintain a disaster recovery link. + +### Table-level publish-subscribe functionality + +In previous iterations, we supported database-level publish/subscribe, and this iteration further implements more granular table-level publish/subscribe. When data changes occur, table-level publish/subscribe can real-time sync data changes from specific tables to subscribers without exposing information from other tables, providing greater flexibility and control compared to database-level publish/subscribe. + +## Other new features + +### SQL statements + +- Support rename table +- Support create pitr +- Support drop pitr +- Support alter pitr +- Support restore pitr +- Support show pitrs +- Optimized show publications +- Optimized show subscriptions +- The load data infile command supports loading data in the order of user-specified column names. + +### Data type + +- Support datalink data type + +### Indexes and constraints + +- Add full text index (fulltext Index) + +### Functions and operators + +- Support json_row, jq, try_jq, json_extract_string, json_extract_float64 functions for json data type +- Support addition or subtraction of the date returned by the now() function + +### Tools + +- mo-backup: supports management of pitr +- mo_cdc: supports management of cdc tasks + +### MySQL Compatibility + +- Support Encode()/Decode() function + +## Quick start + +Community users and enterprise developers can deploy MatrixOne for trial use with the following command. + +```bash +docker pull matrixorigin/matrixone:2.0.0 +``` + +This documentation site also provides detailed architectural descriptions, installation guides, and development tutorials to help you explore the capabilities of MatrixOne. Additionally, you are welcome to ask questions, discuss, or provide feedback on our website at and in our community WeChat group. + +## Known Issues + +- In the current disaster recovery solution for the active and standby clusters, the standby cluster does not support synchronization of data in external tables and stages. +- In the current disaster recovery solution for the active and standby clusters, the standby cluster only supports cold backup and cannot be opened in read-only mode. +- CDC only supports table-level data synchronization. +- Snapshot backup currently only supports cluster and tenant level backup, but can be restored to the cluster, tenant, database or table level. +- Snapshots and PITR backups cannot recover deleted tenant data. + +## More detailed update log + + diff --git a/docs/MatrixOne/Security/TLS-introduction.md b/docs/MatrixOne/Security/TLS-introduction.md index bca485578..591228077 100644 --- a/docs/MatrixOne/Security/TLS-introduction.md +++ b/docs/MatrixOne/Security/TLS-introduction.md @@ -109,7 +109,7 @@ After completing the configuration of these two main steps, a TLS secure connect Current pager: stdout Using outfile: '' Using delimiter: ; - Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne + Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Protocol version: 10 Connection: 127.0.0.1 via TCP/IP Server characterset: utf8mb4 @@ -134,7 +134,7 @@ After completing the configuration of these two main steps, a TLS secure connect Current pager: stdout Using outfile: '' Using delimiter: ; - Server version: 8.0.30-MatrixOne-v1.2.4 MatrixOne + Server version: 8.0.30-MatrixOne-v2.0.0 MatrixOne Protocol version: 10 Connection: 127.0.0.1 via TCP/IP Server characterset: utf8mb4 diff --git a/docs/MatrixOne/Tutorial/rag-demo.md b/docs/MatrixOne/Tutorial/rag-demo.md index a9cb44b19..373d817ea 100644 --- a/docs/MatrixOne/Tutorial/rag-demo.md +++ b/docs/MatrixOne/Tutorial/rag-demo.md @@ -124,7 +124,7 @@ documents = [ "MatrixOne touts significant features, including real-time HTAP, multi-tenancy, stream computation, extreme scalability, cost-effectiveness, enterprise-grade availability, and extensive MySQL compatibility. MatrixOne unifies tasks traditionally performed by multiple databases into one system by offering a comprehensive ultra-hybrid data solution. This consolidation simplifies development and operations, minimizes data fragmentation, and boosts development agility.", "MatrixOne is optimally suited for scenarios requiring real-time data input, large data scales, frequent load fluctuations, and a mix of procedural and analytical business operations. It caters to use cases such as mobile internet apps, IoT data applications, real-time data warehouses, SaaS platforms, and more.", "Matrix is a collection of complex or real numbers arranged in a rectangular array.", -"The lastest version of MatrixOne is v24.1.2.4,released on September 23th, 2024." +"The lastest version of MatrixOne is v24.2.0.0,released on Octomber 31th, 2024." "We are excited to announce MatrixOne v22.0.8.0 release on 2023/6/30." ] @@ -194,7 +194,7 @@ print(output['response']) Console output related answer: ``` -Based on the provided data, the latest version of MatrixOne is v24.1.2.4, which was released on September 23th, 2024. +Based on the provided data, the latest version of MatrixOne is v24.2.0.0, which was released on Octomber 31th, 2024. ``` After enhancement, the model generates the correct answer. diff --git a/mkdocs.yml b/mkdocs.yml index ab77a54f3..9d3a71e00 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -241,6 +241,11 @@ nav: - mo_br guidelines for use: MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr.md - mo_br regular physical backup recovery: MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-physical-backup-restore.md - mo_br snapshot backup recovery: MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-snapshot-backup-restore.md + - mo_br pitr: MatrixOne/Maintain/backup-restore/mobr-backup-restore/mobr-pitr-backup-restore.md + - MatrixOne active/standby disaster recovery: MatrixOne/Maintain/backup-restore/active-standby.md + - cdc: + - mo_cdc guidelines for use: MatrixOne/Maintain/cdc/mocdc.md + - From MatrixOne to MySQL: MatrixOne/Maintain/cdc/mo-mysql.md - Mount Data: - Mount directory to Docker container: MatrixOne/Maintain/mount-data-by-docker.md - Migrating: @@ -305,6 +310,7 @@ nav: - YEAR Type: MatrixOne/Reference/Data-Types/date-time-data-types/year-type.md - JSON Data Type: MatrixOne/Reference/Data-Types/json-type.md - BLOB and TEXT Type: MatrixOne/Reference/Data-Types/blob-text-type.md + - DATALINK Type: MatrixOne/Reference/Data-Types/datalink-type.md - ENUM Type: MatrixOne/Reference/Data-Types/enum-type.md - UUID Type: MatrixOne/Reference/Data-Types/uuid-type.md - VECTOR Type: MatrixOne/Reference/Data-Types/vector-type.md @@ -320,6 +326,7 @@ nav: - CREATE TABLE ... LIKE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-table-like.md - CREATE EXTERNAL TABLE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-external-table.md - CREATE CLUSTER TABLE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-cluster-table.md + - CREATE PITR: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-pitr.md - CREATE PUBLICATION: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-publication.md - CREATE SEQUENCE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-sequence.md - CREATE STAGE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-stage.md @@ -332,6 +339,7 @@ nav: - CREATE SNAPSHOT: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/create-snapshot.md - ALTER TABLE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-table.md - ALTER TABLE ... ALTER REINDEX: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-reindex.md + - ALTER PITR: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-pitr.md - ALTER PUBLICATION: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-publication.md - ALTER SEQUENCE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-sequence.md - ALTER STAGE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/alter-stage.md @@ -339,6 +347,7 @@ nav: - DROP DATABASE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-database.md - DROP INDEX: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-index.md - DROP TABLE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-table.md + - DROP PITR: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-pitr.md - DROP PUBLICATION: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-publication.md - DROP SEQUENCE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-sequence.md - DROP STAGE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-stage.md @@ -346,7 +355,9 @@ nav: - DROP VIEW: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-view.md - DROP FUNCTION: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/drop-function.md - TRUNCATE TABLE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/truncate-table.md - - RESTORE ACCOUNT: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-account.md + - RENAME TABLE: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/rename-table.md + - RESTORE PITR: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-pitr.md + - RESTORE SNAPSHOT: MatrixOne/Reference/SQL-Reference/Data-Definition-Language/restore-snapshot.md - Data Manipulation Language: - INSERT: MatrixOne/Reference/SQL-Reference/Data-Manipulation-Language/insert.md - INSERT INTO SELECT: MatrixOne/Reference/SQL-Reference/Data-Manipulation-Language/insert-into-select.md @@ -413,6 +424,7 @@ nav: - SHOW GRANT: MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-grants.md - SHOW PROCESSLIST: MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-processlist.md - SHOW PUBLICATIONS: MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-publications.md + - SHOW PITRS: MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-pitrs.md - SHOW ROLES: MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-roles.md - SHOW SEQUENCES: MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-sequences.md - SHOW STAGE: MatrixOne/Reference/SQL-Reference/Other/SHOW-Statements/show-stage.md @@ -463,6 +475,8 @@ nav: - BINARY: MatrixOne/Reference/Operators/operators/cast-functions-and-operators/binary.md - CAST: MatrixOne/Reference/Operators/operators/cast-functions-and-operators/cast.md - CONVERT: MatrixOne/Reference/Operators/operators/cast-functions-and-operators/convert.md + - DECODE: MatrixOne/Reference/Operators/operators/cast-functions-and-operators/decode.md + - ENCODE: MatrixOne/Reference/Operators/operators/cast-functions-and-operators/encode.md - SERIAL: MatrixOne/Reference/Operators/operators/cast-functions-and-operators/serial.md - SERIAL_FULL: MatrixOne/Reference/Operators/operators/cast-functions-and-operators/serial_full.md - Comparison Functions and Operators: @@ -645,11 +659,22 @@ nav: - DENSE_RANK(): MatrixOne/Reference/Functions-and-Operators/Window-Functions/dense_rank.md - RANK(): MatrixOne/Reference/Functions-and-Operators/Window-Functions/rank.md - ROW_NUMBER(): MatrixOne/Reference/Functions-and-Operators/Window-Functions/row_number.md - - JSON Functions: MatrixOne/Reference/Functions-and-Operators/Json/json-functions.md + - JSON Functions: + - JQ(): MatrixOne/Reference/Functions-and-Operators/Json/jq.md + - JSON_EXTRACT(): MatrixOne/Reference/Functions-and-Operators/Json/json_extract.md + - JSON_EXTRACT_FLOAT64(): MatrixOne/Reference/Functions-and-Operators/Json/json_extract_float64.md + - JSON_EXTRACT_STRING(): MatrixOne/Reference/Functions-and-Operators/Json/json_extract_string.md + - JSON_QUOTE(): MatrixOne/Reference/Functions-and-Operators/Json/json_quote.md + - JSON_ROW(): MatrixOne/Reference/Functions-and-Operators/Json/json_row.md + - JSON_UNQUOTE(): MatrixOne/Reference/Functions-and-Operators/Json/json_unquote.md + - TRY_JQ(): MatrixOne/Reference/Functions-and-Operators/Json/try_jq.md - Other Functions: - - SAMPLE: MatrixOne/Reference/Functions-and-Operators/Other/sample.md - - SERIAL_EXTRACT: MatrixOne/Reference/Functions-and-Operators/Other/serial_extract.md - - SLEEP: MatrixOne/Reference/Functions-and-Operators/Other/sleep.md + - LOAD_FILE(): MatrixOne/Reference/Functions-and-Operators/Other/load_file.md + - SAVE_FILE(): MatrixOne/Reference/Functions-and-Operators/Other/save_file.md + - SAMPLE(): MatrixOne/Reference/Functions-and-Operators/Other/sample.md + - SERIAL_EXTRACT(): MatrixOne/Reference/Functions-and-Operators/Other/serial_extract.md + - SLEEP(): MatrixOne/Reference/Functions-and-Operators/Other/sleep.md + - STAGE_LIST(): MatrixOne/Reference/Functions-and-Operators/Other/stage_list.md - UUID(): MatrixOne/Reference/Functions-and-Operators/Other/uuid.md - System OPS Functions: - CURRENT_ROLE_NAME(): MatrixOne/Reference/Functions-and-Operators/system-ops/current_role_name.md @@ -685,6 +710,7 @@ nav: - SQL FAQs: MatrixOne/FAQs/sql-faqs.md - Release Notes: - Release Timeline: MatrixOne/Release-Notes/release-timeline.md + - v24.2.0.0: MatrixOne/Release-Notes/v24.2.0.0.md - v24.1.2.4: MatrixOne/Release-Notes/v24.1.2.4.md - v24.1.2.3: MatrixOne/Release-Notes/v24.1.2.3.md - v24.1.2.2: MatrixOne/Release-Notes/v24.1.2.2.md