[KYUUBI #5327][BATCH] Add priority field in metadata table definition

### _Why are the changes needed?_

Add `priority` field in `metadata` table definition

In the current batch v2 implementation, Kyuubi supports the FIFO schedule strategy, we are planning the enhance it with `priority` support, those with a higher integer value of `priority` have a better opportunity to be scheduled.

It refers priority definition in [Apache Hadoop YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html#Setup_for_application_priority) and [Apache YuniKorn](https://yunikorn.apache.org/docs/user_guide/priorities/#application-priority) to define the `priority` as:
1. use a signed 4 bits integer to represent priority with the default value of 10
2. a higher integer value of `priority` has a better opportunity to be scheduled.

In practice, we face some issues when use MySQL as metastore:

MySQL 5.7 could not benefit from index in query mixes `ASC` and `DESC`, for example: `SELECT * FROM metadata ORDER BY priority DESC, create_time ASC`, which is mentioned in [MySQL ORDER BY optimization](https://dev.mysql.com/doc/refman/5.7/en/order-by-optimization.html)
> In some cases, MySQL cannot use indexes to resolve the `ORDER BY`, although it may still use indexes to find the rows that match the `WHERE` clause. Examples:
> - ...
> - The query mixes `ASC` and `DESC`
> - ...

Fortunately, it was enhanced in MySQL 8.0
> Two columns in an ORDER BY can sort in the same direction (both ASC, or both DESC) or in opposite directions (one ASC, one DESC). A condition for index use is that the index must have the same homogeneity, but need not have the same actual direction.
> If a query mixes ASC and DESC, the optimizer can use an index on the columns if the index also uses corresponding mixed ascending and descending columns:

Thus, we create a mixed `ORDER BY` index `ALTER TABLE metadata ADD INDEX priority_create_time_index(priority DESC, create_time ASC)` to improve the query efficiency and avoid slowness in the case of large data volumes.

According to [MySQL 5.7 Index](https://dev.mysql.com/doc/refman/5.7/en/create-index.html), this won't cause SQL error when creating this index in MySQL 5.7 but not take effect.

> A key_part specification can end with ASC or DESC. These keywords are permitted for future extensions for specifying ascending or descending index value storage. Currently, they are parsed but ignored; index values are always stored in ascending order.

Close #5327

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

Test MySQL & Derby with DBeaver.

MySQL:
![截屏2023-09-25 19 33 51](https://github.com/apache/kyuubi/assets/52876270/cd8b7015-1fad-45bd-add6-8c327ae9461b)
![截屏2023-09-25 19 33 57](https://github.com/apache/kyuubi/assets/52876270/7e10e5b6-2aa9-405a-a4ef-ec61e42fb210)

Derby:
![截屏2023-09-25 19 33 34](https://github.com/apache/kyuubi/assets/52876270/a8a9e3fc-cb2b-4012-a912-2536d271d38f)
![截屏2023-09-25 19 33 38](https://github.com/apache/kyuubi/assets/52876270/4bf986ec-8190-4fda-ad15-494aca70814d)

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5329 from zwangsheng/KYUUBI#5327.

Closes #5327

8688d6b [zwangsheng] fix comments
107221b [zwangsheng] fix comments
3e4083d [zwangsheng] fix comments
313b42b [zwangsheng] EOF
eb6dc9c [zwangsheng] [KYUUBI #5327][Umbrella][V2] Kyuubi server pick task from metadata db with priority

Authored-by: zwangsheng <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>

kyuubi-server/src/main/resources/sql/derby/005-KYUUBI-5327.derby.sql

@@ -0,0 +1,3 @@
+ALTER TABLE metadata ADD COLUMN priority int NOT NULL DEFAULT 10;
+
+CREATE INDEX metadata_priority_create_time_index ON metadata(priority, create_time);

kyuubi-server/src/main/resources/sql/derby/metadata-store-schema-1.8.0.derby.sql

@@ -26,6 +26,7 @@ CREATE TABLE metadata(
     engine_state varchar(32), -- the engine application state
     engine_error clob, -- the engine application diagnose
     end_time bigint,  -- the metadata end time
+    priority int NOT NULL DEFAULT 10, -- the application priority, high value means high priority
     peer_instance_closed boolean default FALSE -- closed by peer kyuubi instance
 );
 
@@ -36,3 +37,5 @@ CREATE INDEX metadata_user_name_index ON metadata(user_name);
 CREATE INDEX metadata_engine_type_index ON metadata(engine_type);
 
 CREATE INDEX metadata_create_time_index ON metadata(create_time);
+
+CREATE INDEX metadata_priority_create_time_index ON metadata(priority, create_time);

kyuubi-server/src/main/resources/sql/mysql/005-KYUUBI-5327.mysql.sql

@@ -0,0 +1,13 @@
+SELECT '< KYUUBI-5327: Introduce priority in metadata' AS ' ';
+
+ALTER TABLE metadata ADD COLUMN priority int NOT NULL DEFAULT 10 COMMENT 'the application priority, high value means high priority';
+
+-- In MySQL 5.7, A key_part specification can end with ASC or DESC.
+-- These keywords are permitted for future extensions for specifying ascending or descending index value storage.
+-- Currently, they are parsed but ignored; index values are always stored in ascending order.
+-- In MySQL 8 this can take effect and this index will be hit if query order by priority DESC, create_time ASC.
+-- See more detail in:
+-- https://dev.mysql.com/doc/refman/8.0/en/index-hints.html
+-- https://dev.mysql.com/doc/refman/8.0/en/create-index.html
+-- https://dev.mysql.com/doc/refman/5.7/en/create-index.html
+ALTER TABLE metadata ADD INDEX priority_create_time_index(priority DESC, create_time ASC);

kyuubi-server/src/main/resources/sql/mysql/metadata-store-schema-1.8.0.mysql.sql

@@ -24,9 +24,12 @@ CREATE TABLE IF NOT EXISTS metadata(
     engine_state varchar(32) COMMENT 'the engine application state',
     engine_error mediumtext COMMENT 'the engine application diagnose',
     end_time bigint COMMENT 'the metadata end time',
+    priority int NOT NULL DEFAULT 10 COMMENT 'the application priority, high value means high priority',
     peer_instance_closed boolean default '0' COMMENT 'closed by peer kyuubi instance',
     UNIQUE INDEX unique_identifier_index(identifier),
     INDEX user_name_index(user_name),
     INDEX engine_type_index(engine_type),
-    INDEX create_time_index(create_time)
+    INDEX create_time_index(create_time),
+    -- See more detail about this index in ./005-KYUUBI-5327.mysql.sql
+    INDEX priority_create_time_index(priority DESC, create_time ASC)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

kyuubi-server/src/main/resources/sql/mysql/upgrade-1.7.0-to-1.8.0.mysql.sql

@@ -1,4 +1,5 @@
 SELECT '< Upgrading MetaStore schema from 1.7.0 to 1.8.0 >' AS ' ';
 SOURCE 003-KYUUBI-5078.mysql.sql;
 SOURCE 004-KYUUBI-5131.mysql.sql;
+SOURCE 005-KYUUBI-5327.mysql.sql;
 SELECT '< Finished upgrading MetaStore schema from 1.7.0 to 1.8.0 >' AS ' ';

kyuubi-server/src/main/resources/sql/sqlite/metadata-store-schema-1.8.0.sqlite.sql

@@ -24,6 +24,7 @@ CREATE TABLE IF NOT EXISTS metadata(
     engine_state varchar(32), -- the engine application state
     engine_error mediumtext, -- the engine application diagnose
     end_time bigint, -- the metadata end time
+    priority INTEGER NOT NULL DEFAULT 10, -- the application priority, high value means high priority
     peer_instance_closed boolean default '0' -- closed by peer kyuubi instance
 );
 
@@ -34,3 +35,5 @@ CREATE INDEX IF NOT EXISTS metadata_user_name_index ON metadata(user_name);
 CREATE INDEX IF NOT EXISTS metadata_engine_type_index ON metadata(engine_type);
 
 CREATE INDEX IF NOT EXISTS metadata_create_time_index ON metadata(create_time);
+
+CREATE INDEX IF NOT EXISTS metadata_priority_create_time_index ON metadata(priority, create_time);