Skip to content

ShannonBase Introduction

Jump to bottom
Shannon Data AI Lab edited this page Jan 29, 2024 · 15 revisions

Welcome to the ShannonBase

Introduction

Shannonbase is the only fully managed database service that combines transactions, analytics, and machine learning services into one MySQL Database, without the complexity, latency, and cost of ETL duplication.Shannonbase will include Lakehouse, letting users query half a petabyte of data in object storage—in a variety of file formats, such as CSV, Parquet, and export files from other databases. Customers can leverage the benefits of HTAP even when their data is stored outside a MySQL database.

ShannonBase is an integrated HTAP database that adopts hybrid row-column store and in-memory computing. It is fully compatible with MySQL version 8.1.

image

Features

  • Transaction workload process

    It keeps all TP(Transaction workloads Process) features which MySQL has.

  • Analytical workloads process

    In shannonbase, it employes a novel engnine, named IMCS(in-memory column store) to process all AP workloads.

  • rapid secondary engine.

    An in-memory column store, named rapid secondary engine be incorporated. it's used to process AP workloads.

mysql> show engines; 
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine             | Support | Comment                                                        | Transactions | XA   | Savepoints |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| ndbcluster         | NO      | Clustered, fault-tolerant tables                               | NULL         | NULL | NULL       |
| FEDERATED          | NO      | Federated MySQL storage engine                                 | NULL         | NULL | NULL       |
| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables      | NO           | NO   | NO         |
| InnoDB             | DEFAULT | Supports transactions, row-level locking, and foreign keys     | YES          | YES  | YES        |
| PERFORMANCE_SCHEMA | YES     | Performance Schema                                             | NO           | NO   | NO         |
| Rapid              | YES     | Shannon Rapid storage engine                                   | NO           | NO   | NO         |
| MyISAM             | YES     | MyISAM storage engine                                          | NO           | NO   | NO         |
| ndbinfo            | NO      | MySQL Cluster system information storage engine                | NULL         | NULL | NULL       |
| MRG_MYISAM         | YES     | Collection of identical MyISAM tables                          | NO           | NO   | NO         |
| BLACKHOLE          | YES     | /dev/null storage engine (anything you write to it disappears) | NO           | NO   | NO         |
| CSV                | YES     | CSV storage engine                                             | NO           | NO   | NO         |
| ARCHIVE            | YES     | Archive storage engine                                         | NO           | NO   | NO         |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+

Getting Started

  • How to build ShannonBase from source code

step 1: clone the source code from Shannonbase Repo

  git clone [email protected]:Shannon-Data/ShannonBase.git

step 2: makes a directory where we build the source code.

  cd ShannonBase && mkdir cmake_build && cd cmake_build

step 3: run cmake

 cmake ../ \
  -DWITH_BOOST=/path-to-boost-include-files/ \
  -DCMAKE_BUILD_TYPE=[Release|Debug]  \
  -DCMAKE_INSTALL_PREFIX=/path-to-shannon-bin \
  -DMYSQL_DATADIR=/home/path-to-shannon-bin/data \
  -DSYSCONFDIR=. \
  -DMYSQL_UNIX_ADDR=/home/path-to-shannon-bin/tmp/mysql.sock \
  -DWITH_EMBEDDED_SERVER=OFF \
  -DWITH_MYISAM_STORAGE_ENGINE=1 \
  -DWITH_INNOBASE_STORAGE_ENGINE=1 \
  -DWITH_PARTITION_STORAGE_ENGINE=1 \
  -DMYSQL_TCP_PORT=3306 \
  -DENABLED_LOCAL_INFILE=1 \
  -DEXTRA_CHARSETS=all \
  -DWITH_PROTOBUF=bundled \
  -DWITH_SSL_PATH=/path-to-open-ssl/ \
  -DDEFAULT_SET=community \
  -DWITH_UNIT_TESTS=OFF \
  [-DENABLE_GCOV=1 \ |
  -DWITH_ASAN=1 \    | 
  ]
  -DCOMPILATION_COMMENT="MySQL Community Server, and Shannon Data AI Alpha V.- (GPL)"

step 4: compile and install

  make -j5 && make install

step 5: initialize the database

  /path-to-shannbase-bin/bin/mysqld --defaults-file=/path-to-shannonbase-bin/my.cnf --initialize  --user=xxx

step 6: run shannonbase

 /path-to-shannbase-bin/bin//mysqld --defaults-file=/path-to-shannonbase-bin/my.cnf   --user=xxx &
  • How to use Rapid engine step 1: create a table with secondary engine.
CREATE TABLE `NATION` (
  `N_NATIONKEY` int NOT NULL,
  `N_NAME` char(25) COLLATE ascii_bin NOT NULL COMMENT 'RAPID_COLUMN=ENCODING=SORTED',
  `N_REGIONKEY` int NOT NULL,
  `N_COMMENT` varchar(152) COLLATE ascii_bin DEFAULT NULL COMMENT 'RAPID_COLUMN=ENCODING=SORTED',
  PRIMARY KEY (`N_NATIONKEY`)
) ENGINE=InnoDB DEFAULT CHARSET=ascii COLLATE=ascii_bin SECONDARY_ENGINE=RAPID

step 2: Load data into rapid engine;

 alter table NATION secondary_load;

step 3: Queries.

  select * from NATION;

Releases

ver 0.1, 2024-01-29

1: Rapid Supported

2: Supported Type The following types has been supported in shannon rapid engine. (will be updated in future)

  • Numeric

1: Integer

2: double

  • String

1: char

2: varchar

3: text

*Datetime

1: date

2: time

3: datetime

Clone this wiki locally