Skip to content

Practices

Jump to bottom
Shannon Data AI Lab edited this page Sep 14, 2024 · 10 revisions

The practice cases of ShannonBase.

0: Compiling & Installation

Getting Started

  • How to build ShannonBase from source code

step 0: Requirement Installation

1: compiling tools:

apt-get install -y g++
apt-get install -y libbison-dev
apt-get install -y flex
apt-get install -y clang-format
apt-get install -y lcov
apt-get install -y pkg-config

2: Cmake

apt-get install -y cmake

3: tools

apt-get install -y git
apt-get install -y wget
apt-get install -y tar
apt-get install -y bzip2
apt-get install -y unzip

4: openssl etc.

apt-get install -y libssl-dev
apt-get install -y libncurses-dev
apt-get install -y  libudev-dev
apt-get install -y libgsasl-dev
apt-get install -y libldap-dev

5: Boost [from source code]

wget https://boostorg.jfrog.io/artifactory/main/release/1.77.0/source/boost_1_77_0.tar.bz2
tar -xvf boost_1_77_0.tar.bz2 && cd boost_1_77_0 && ./bootstrap.sh && ./b2  && ./b2 install

step 1: clone the source code from Shannonbase Repo

  git clone [email protected]:Shannon-Data/ShannonBase.git

step 2: makes a directory where we build the source code.

  cd ShannonBase && mkdir cmake_build && cd cmake_build

step 3: run cmake

 cmake ../ \
  -DWITH_BOOST=/path-to-boost-include-files/ \
  -DCMAKE_BUILD_TYPE=[Release|Debug]  \
  -DCMAKE_INSTALL_PREFIX=/path-to-shannon-bin \
  -DMYSQL_DATADIR=/home/path-to-shannon-bin/data \
  -DSYSCONFDIR=. \
  -DMYSQL_UNIX_ADDR=/home/path-to-shannon-bin/tmp/mysql.sock \
  -DWITH_EMBEDDED_SERVER=OFF \
  -DWITH_MYISAM_STORAGE_ENGINE=1 \
  -DWITH_INNOBASE_STORAGE_ENGINE=1 \
  -DWITH_PARTITION_STORAGE_ENGINE=1 \
  -DMYSQL_TCP_PORT=3306 \
  -DENABLED_LOCAL_INFILE=1 \
  -DEXTRA_CHARSETS=all \
  -DWITH_PROTOBUF=bundled \
  -DWITH_SSL_PATH=/path-to-open-ssl/ \
  -DDEFAULT_SET=community \
  -DWITH_UNIT_TESTS=OFF \
  [-DENABLE_GCOV=1 \ |
  -DWITH_ASAN=1 \    | 
  ]
  -DCOMPILATION_COMMENT="MySQL Community Server, and Shannon Data AI Alpha V.- (GPL)"

step 4: compile and install

  make -j5 && make install

step 5: initialize the database

  /path-to-shannbase-bin/bin/mysqld --defaults-file=/path-to-shannonbase-bin/my.cnf --initialize  --user=xxx

step 6: run shannonbase

 /path-to-shannbase-bin/bin//mysqld --defaults-file=/path-to-shannonbase-bin/my.cnf   --user=xxx &

1: Using TPC-H

the script of creating table and loading data can be found here. https://github.com/Shannon-Data/heatwave-tpch/tree/main/TPCH.

After we have loaded the data into rapid engine, then, start to run queries.

Taking NATION as an instance.

create database tpch_1024;
alter database tpch_1024 CHARACTER SET ascii COLLATE ascii_bin; 
use tpch_1024;

CREATE TABLE NATION  ( N_NATIONKEY  INTEGER NOT NULL,
                       N_NAME       CHAR(25) NOT NULL,
                       N_REGIONKEY  INTEGER NOT NULL,
                       N_COMMENT    VARCHAR(152),
                       PRIMARY KEY (N_NATIONKEY));

2: Using ML functions

3: Using automatic changes population

Load the data into rapid engine.

alter table NATION secondary_load;

and if you want to use rapid mandatorily.

set use_secondary_engine=forced;

or you can use it according to cost

set secondary_engine_cost_threshold = xxx;

then do query, it will use rapid to do query.

  select * from NATION;

4: Using master-replica

Takes the following steps to build up ShannonBase master-replica.

1. Prepare the Master Instance

Enable Binary Logging on the master server:

Edit the MySQL configuration file (/etc/my.cnf or /etc/mysql/my.cnf) to add or enable the following options:

[mysqld]
log_bin = mysql-bin  # Enable binary logging
server-id = 1        # Unique server ID for the master[1 is an example]
binlog_format = ROW  # Use row-based replication

  1. Restart MySQL servito apply the changes.

  2. Create a Replication User on the master:

CREATE USER 'replica_user'@'%' IDENTIFIED BY 'replica_password';
GRANT REPLICATION SLAVE ON *.* TO 'replica_user'@'%';
FLUSH PRIVILEGES;
  1. Get the Master Log File and Position:
FLUSH TABLES WITH READ LOCK;
SHOW MASTER STATUS;

2. Configure the Replica Instance

  1. Prepare the Replica Server: On the replica server, configure the MySQL configuration file with a unique server ID:
[mysqld]
server-id = 2        # Unique server ID for the replica[2 is an example]
relay_log = relay-bin  # Optional: to enable relay logging

  1. Restart Replica Service.

  2. Set Up Replication on the replica: Run the following SQL on the replica to link it to the master:

CHANGE MASTER TO
MASTER_HOST = 'master_ip_address',
MASTER_USER = 'replica_user',
MASTER_PASSWORD = 'replica_password',
MASTER_LOG_FILE = 'mysql-bin.xxxxx',  # Use the file from the master
MASTER_LOG_POS = xxxxx;               # Use the position from the master
  1. Start Replica.
START SLAVE;
  1. Monitor and Maintain Replication.
SHOW SLAVE STATUS\G

If there're some errors, the replica relationship will be broken.

*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: localhost
                  Master_User: replica_user
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: binlog.000234
          Read_Master_Log_Pos: 3866
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 511
        Relay_Master_Log_File: binlog.000234
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 1007
                   Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction 'ANONYMOUS' at source log binlog.000234, end_log_pos 3266. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 3081
              Relay_Log_Space: 1500
              Until_Condition: None
Clone this wiki locally