Skip to content

Commit

Permalink
Add Interval implementation of InlineExpressionParser SPI
Browse files Browse the repository at this point in the history
  • Loading branch information
linghengqian committed Dec 6, 2023
1 parent a0519cf commit 1ebdf52
Show file tree
Hide file tree
Showing 9 changed files with 447 additions and 2 deletions.
1 change: 1 addition & 0 deletions docs/document/content/dev-manual/sharding.cn.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,4 +96,5 @@ chapter = true
|----------|----------------------------------------------------|--------------------------------------------------------------------------------|
| GROOVY | 使用 Groovy 语法的行表达式 | `org.apache.shardingsphere.infra.expr.groovy.GroovyInlineExpressionParser` |
| LITERAL | 使用标准列表的行表达式 | `org.apache.shardingsphere.infra.expr.literal.LiteralInlineExpressionParser` |
| INTERVAL | 基于固定时间范围的 Key-Value 语法的行表达式 | `org.apache.shardingsphere.infra.expr.interval.IntervalInlineExpressionParser` |
| ESPRESSO | 基于 GraalVM Truffle 的 Espresso 实现的使用 Groovy 语法的行表达式 | `org.apache.shardingsphere.infra.expr.espresso.EspressoInlineExpressionParser` |
1 change: 1 addition & 0 deletions docs/document/content/dev-manual/sharding.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,4 +96,5 @@ Row Value Expressions definition
|----------------------|------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------|
| GROOVY | Row Value Expressions that uses the Groovy syntax | `org.apache.shardingsphere.infra.expr.groovy.GroovyInlineExpressionParser` |
| LITERAL | Row Value Expressions that uses a standard list | `org.apache.shardingsphere.infra.expr.literal.LiteralInlineExpressionParser` |
| INTERVAL | Row Value Expressions based on fixed interval that uses the Key-Value syntax | `org.apache.shardingsphere.infra.expr.interval.IntervalInlineExpressionParser` | |
| ESPRESSO | Row Value Expressions that uses the Groovy syntax based on GraalVM Truffle's Espresso implementation | `org.apache.shardingsphere.infra.expr.espresso.EspressoInlineExpressionParser` |
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,40 @@ weight = 10
- `<LITERAL>t_order_1, t_order_2, t_order_3` 将被转化为 `t_order_1, t_order_2, t_order_3`
- `<LITERAL>t_order_${1..3}` 将被转化为 `t_order_${1..3}`

## 基于固定时间范围的 Key-Value 语法的行表达式

`INTERVAL` 实现引入了 Key-Value 风格的属性语法,来通过单行字符串定义一组时间范围内的字符串。这通常用于简化对 `数据分片` 功能的 `actualDataNodes` 的定义。

`INTERVAL` 实现定义多个属性的方式为 `Key1=Value1;Key2=Value2`,通过 `;` 号分割键值对,通过 `=` 号分割 `Key` 值和 `Value` 值。

此实现主动忽视了 `SP` 的时区信息,这意味着当 `DL``DU` 含有时区信息时,不会因为时区不一致而发生时区转换。

此实现键值对的顺序不敏感, 行表达式尾部不携带 `;` 号。

`INTERVAL` 实现引入了如下 Key 的键值:

1. `P` 代表 prefix 的缩写,意为结果列表单元的前缀, 通常代表真实表的前缀格式。
2. `SP` 代表 suffix pattern 的缩写,意为结果列表单元的后缀的时间戳格式, 通常代表真实表的后缀格式,必须遵循 Java DateTimeFormatter 的格式。
例如:yyyyMMdd,yyyyMM 或 yyyy 等。但不支持与 `java.time.chrono.JapaneseDate` 相关的 Gy_MM 等。
3. `DIA` 代表 datetime interval amount 的缩写,意为结果列表单元的时间间隔。存在默认值为 `1`
4. `DIU` 代表 datetime interval unit 的缩写,意为分片键时间间隔单位,大小写不敏感,必须遵循 Java ChronoUnit 的枚举值。例如:`Months`
存在默认值为 `Days`
5. `DL` 代表 datetime lower 的缩写,意为时间上界值,格式与 `SP` 定义的时间戳格式一致。
6. `DU` 代表 datetime upper 的缩写,意为时间上界值,格式与 `SP` 定义的时间戳格式一致。

类型:INTERVAL

用例:

- `<INTERVAL>P=t_order_;SP=yyyy_MMdd;DIA=1;DIU=Days;DL=2023_1202;DU=2023_1204` 将被转化为 `t_order_2023_1202, t_order_2023_1203, t_order_2023_1204`
- `<INTERVAL>P=t_order_;SP=yyyy_MM;DIA=1;DIU=Months;DL=2023_10;DU=2023_12` 将被转化为 `t_order_2023_10, t_order_2023_11, t_order_2023_12`
- `<INTERVAL>P=ds-0.t_order_;SP=yyyy;DIA=1;DIU=Years;DL=2021;DU=2023` 将被转化为 `t_order_2021, t_order_2022, t_order_2023`
- `<INTERVAL>P=ds-0.t_order_;SP=HH_mm_ss_SSS;DIA=1;DIU=Millis;DL=22_48_52_131;DU=22_48_52_133` 将被转化为 `t_order_22_48_52_131, t_order_22_48_52_132, t_order_22_48_52_133`
- `<INTERVAL>P=ds-0.t_order_;SP=yyyy_MM_dd_HH_mm_ss_SSS;DIA=1;DIU=Days;DL=2023_12_04_22_48_52_131;DU=2023_12_06_22_48_52_131` 将被转化为 `t_order_2023_12_04_22_48_52_131, t_order_2023_12_05_22_48_52_131, t_order_2023_12_06_22_48_52_131`

## 基于 GraalVM Truffle 的 Espresso 实现的使用 Groovy 语法的行表达式

此为可选实现,你需要在自有项目的 `pom.xml` 主动声明如下依赖。并且请确保自有项目通过 GraalVM CE 23.0.1 For JDK17 编译。
此为可选实现,你需要在自有项目的 `pom.xml` 主动声明如下依赖。并且请确保自有项目通过 GraalVM CE 23.0.1 For JDK 17.0.9 编译。

```xml
<dependencies>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,45 @@ Example:
- `<LITERAL>t_order_1, t_order_2, t_order_3` will be converted to `t_order_1, t_order_2, t_order_3`
- `<LITERAL>t_order_${1..3}` will be converted to `t_order_${1..3}`

## Row Value Expressions based on fixed interval that uses the Key-Value syntax

The `INTERVAL` implementation introduces a Key-Value style property syntax to define a set of time ranges of strings via a single line string.
This is often used to simplify the definition of `actualDataNodes` for `Sharding` feature.

`INTERVAL` implements the method of defining multiple attributes as `Key1=Value1;Key2=Value2`, using `;` to separate key-value pairs, and `=` to separate `Key` values and `Value` values.

This implementation actively ignores the time zone information of `SP`, which means that when `DL` and `DU` contain time zone information,
no time zone conversion will occur due to inconsistent time zones.

This implementation is not sensitive to the order of key-value pairs, and the line expression does not carry the `;` sign at the end.

The `INTERVAL` implementation introduces the following Key values:

1. `P` stands for the abbreviation of prefix, which means the prefix of the result list unit, usually representing the prefix format of the real table.
2. `SP` stands for the abbreviation of suffix pattern, which means the timestamp format of the suffix of the result list unit.
It usually represents the suffix format of the real table and must follow the format of Java DateTimeFormatter.
For example: yyyyMMdd, yyyyMM or yyyy etc. But Gy_MM etc. related to `java.time.chrono.JapaneseDate` are not supported.
3. `DIA` stands for the abbreviation of datetime interval amount, which means the time interval of the result list unit. There is a default value of `1`.
4. `DIU` stands for the abbreviation of datetime interval unit, which means the shard key time interval unit.
It is not case-sensitive and must follow the enumeration value of Java ChronoUnit. For example: `Months`.
There is a default value of `Days`.
5. `DL` stands for the abbreviation of datetime lower, which means the upper bound of time. The format is consistent with the timestamp format defined by `SP`.
6. `DU` stands for the abbreviation of datetime upper, which means the upper bound value of time. The format is consistent with the timestamp format defined by `SP`.

Type: INTERVAL

Example:

- `<INTERVAL>P=t_order_;SP=yyyy_MMdd;DIA=1;DIU=Days;DL=2023_1202;DU=2023_1204` will be converted to `t_order_2023_1202, t_order_2023_1203, t_order_2023_1204`
- `<INTERVAL>P=t_order_;SP=yyyy_MM;DIA=1;DIU=Months;DL=2023_10;DU=2023_12` will be converted to `t_order_2023_10, t_order_2023_11, t_order_2023_12`
- `<INTERVAL>P=ds-0.t_order_;SP=yyyy;DIA=1;DIU=Years;DL=2021;DU=2023` will be converted to `t_order_2021, t_order_2022, t_order_2023`
- `<INTERVAL>P=ds-0.t_order_;SP=HH_mm_ss_SSS;DIA=1;DIU=Millis;DL=22_48_52_131;DU=22_48_52_133` will be converted to `t_order_22_48_52_131, t_order_22_48_52_132, t_order_22_48_52_133`
- `<INTERVAL>P=ds-0.t_order_;SP=yyyy_MM_dd_HH_mm_ss_SSS;DIA=1;DIU=Days;DL=2023_12_04_22_48_52_131;DU=2023_12_06_22_48_52_131` will be converted to `t_order_2023_12_04_22_48_52_131, t_order_2023_12_05_22_48_52_131, t_order_2023_12_06_22_48_52_131`

## Row Value Expressions that uses the Groovy syntax based on GraalVM Truffle's Espresso implementation

This is an optional implementation, and you need to actively declare the following dependencies in the `pom.xml` of your own project.
And make sure your own project is compiled with GraalVM CE 23.0.1 For JDK17.
And make sure your own project is compiled with GraalVM CE 23.0.1 For JDK 17.0.9.

```xml
<dependencies>
Expand Down
43 changes: 43 additions & 0 deletions infra/expr/type/interval/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-infra-expr-type</artifactId>
<version>5.4.2-SNAPSHOT</version>
</parent>
<artifactId>shardingsphere-infra-expr-interval</artifactId>
<name>${project.artifactId}</name>

<dependencies>
<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-infra-expr-spi</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.apache.shardingsphere</groupId>
<artifactId>shardingsphere-test-util</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.shardingsphere.infra.expr.interval;

import com.google.common.base.Strings;
import org.apache.shardingsphere.infra.exception.core.ShardingSpherePreconditions;
import org.apache.shardingsphere.infra.exception.core.external.sql.type.generic.UnsupportedSQLOperationException;
import org.apache.shardingsphere.infra.expr.spi.InlineExpressionParser;

import java.time.LocalDate;
import java.time.LocalDateTime;
import java.time.LocalTime;
import java.time.Month;
import java.time.Year;
import java.time.YearMonth;
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeParseException;
import java.time.temporal.ChronoField;
import java.time.temporal.ChronoUnit;
import java.time.temporal.Temporal;
import java.time.temporal.TemporalAccessor;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.stream.Collectors;
import java.util.stream.LongStream;

/**
* Interval inline expression parser.
*/
public class IntervalInlineExpressionParser implements InlineExpressionParser {

/**
* Abbreviation for prefix.
*/
private static final String PREFIX_KEY = "P";

/**
* Abbreviation for suffix pattern.
*/
private static final String SUFFIX_PATTERN_KEY = "SP";

/**
* Abbreviation for datetime interval amount.
*/
private static final String INTERVAL_AMOUNT_KEY = "DIA";

/**
* Abbreviation for datetime interval unit.
*/
private static final String INTERVAL_UNIT_KEY = "DIU";

/**
* Abbreviation for datetime lower.
*/
private static final String DATE_TIME_LOWER_KEY = "DL";

/**
* Abbreviation for datetime upper.
*/
private static final String DATE_TIME_UPPER_KEY = "DU";

private TemporalAccessor startTime;

private TemporalAccessor endTime;

private String prefix;

private DateTimeFormatter dateTimeFormatterForSuffixPattern;

private int stepAmount;

private ChronoUnit stepUnit;

private String inlineExpression;

@Override
public void init(final Properties props) {
inlineExpression = props.getProperty(INLINE_EXPRESSION_KEY);
Map<String, String> propsMap = Arrays.stream(inlineExpression.split(";"))
.collect(Collectors.toMap(string -> string.split("=")[0], string -> string.split("=")[1]));
prefix = getPrefix(propsMap);
dateTimeFormatterForSuffixPattern = getSuffixPattern(propsMap);
startTime = getDateTimeLower(propsMap);
endTime = getDateTimeUpper(propsMap);
stepAmount = Integer.parseInt(propsMap.getOrDefault(INTERVAL_AMOUNT_KEY, "1"));
stepUnit = propsMap.containsKey(INTERVAL_UNIT_KEY) ? getStepUnit(propsMap.get(INTERVAL_UNIT_KEY)) : ChronoUnit.DAYS;
}

@Override
public String handlePlaceHolder() {
return inlineExpression;
}

@Override
public List<String> splitAndEvaluate() {
return Strings.isNullOrEmpty(inlineExpression) ? Collections.emptyList() : split();
}

private String getPrefix(final Map<String, String> props) {
ShardingSpherePreconditions.checkState(props.containsKey(PREFIX_KEY),
() -> new RuntimeException(String.format("%s can not be null.", PREFIX_KEY)));
return props.get(PREFIX_KEY);
}

private TemporalAccessor getDateTimeLower(final Map<String, String> props) {
ShardingSpherePreconditions.checkState(props.containsKey(DATE_TIME_LOWER_KEY),
() -> new RuntimeException(String.format("%s can not be null.", DATE_TIME_LOWER_KEY)));
return getDateTime(props.get(DATE_TIME_LOWER_KEY));
}

private TemporalAccessor getDateTimeUpper(final Map<String, String> props) {
ShardingSpherePreconditions.checkState(props.containsKey(DATE_TIME_UPPER_KEY),
() -> new RuntimeException(String.format("%s can not be null.", DATE_TIME_UPPER_KEY)));
return getDateTime(props.get(DATE_TIME_UPPER_KEY));
}

private TemporalAccessor getDateTime(final String dateTimeValue) {
try {
return dateTimeFormatterForSuffixPattern.parse(dateTimeValue);
} catch (final DateTimeParseException dateTimeParseException) {
throw new RuntimeException(dateTimeParseException);
}
}

private DateTimeFormatter getSuffixPattern(final Map<String, String> props) {
String suffix = props.get(SUFFIX_PATTERN_KEY);
ShardingSpherePreconditions.checkState(!Strings.isNullOrEmpty(suffix),
() -> new RuntimeException(String.format("%s can not be null or empty.", SUFFIX_PATTERN_KEY)));
return DateTimeFormatter.ofPattern(suffix);
}

private ChronoUnit getStepUnit(final String stepUnit) {
return Arrays.stream(ChronoUnit.values())
.filter(chronoUnit -> chronoUnit.toString().equalsIgnoreCase(stepUnit))
.findFirst()
.orElseThrow(() -> new UnsupportedSQLOperationException(String.format("Cannot find step unit for specified %s property: `%s`", INTERVAL_UNIT_KEY, stepUnit)));
}

private List<String> split() {
TemporalAccessor calculateTime = startTime;
if (!calculateTime.isSupported(ChronoField.NANO_OF_DAY)) {
if (calculateTime.isSupported(ChronoField.EPOCH_DAY)) {
return convertStringFromTemporal(startTime.query(LocalDate::from), endTime.query(LocalDate::from));
}
if (calculateTime.isSupported(ChronoField.YEAR) && calculateTime.isSupported(ChronoField.MONTH_OF_YEAR)) {
return convertStringFromTemporal(startTime.query(YearMonth::from), endTime.query(YearMonth::from));
}
if (calculateTime.isSupported(ChronoField.YEAR)) {
return convertStringFromTemporal(startTime.query(Year::from), endTime.query(Year::from));
}
if (calculateTime.isSupported(ChronoField.MONTH_OF_YEAR)) {
return convertStringFromMonth();
}
}
if (!calculateTime.isSupported(ChronoField.EPOCH_DAY)) {
return convertStringFromTemporal(startTime.query(LocalTime::from), endTime.query(LocalTime::from));
}
return convertStringFromTemporal(startTime.query(LocalDateTime::from), endTime.query(LocalDateTime::from));
}

private List<String> convertStringFromMonth() {
Month startTimeAsMonth = startTime.query(Month::from);
Month endTimeAsMonth = endTime.query(Month::from);
return LongStream.iterate(0, x -> x + stepAmount)
.limit((endTimeAsMonth.getValue() - startTimeAsMonth.getValue()) / stepAmount + 1)
.parallel()
.boxed()
.map(startTimeAsMonth::plus)
.map(dateTimeFormatterForSuffixPattern::format)
.map(suffix -> prefix + suffix)
.collect(Collectors.toList());
}

private List<String> convertStringFromTemporal(final Temporal startTimeAsTemporal, final Temporal endTimeAsTemporal) {
return LongStream.iterate(0, x -> x + stepAmount)
.limit(stepUnit.between(startTimeAsTemporal, endTimeAsTemporal) / stepAmount + 1)
.parallel()
.boxed()
.map(arithmeticSequence -> startTimeAsTemporal.plus(arithmeticSequence, stepUnit))
.map(dateTimeFormatterForSuffixPattern::format)
.map(suffix -> prefix + suffix)
.collect(Collectors.toList());
}

@Override
public String getType() {
return "INTERVAL";
}
}
Loading

0 comments on commit 1ebdf52

Please sign in to comment.