Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support scientific notation representation in PPL #2827

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Support scientific notation representation in PPL #2827

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/user/dql/expressions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,13 @@ Examples

Here is an example for different type of literals::

os> SELECT 123, 'hello', false, -4.567, DATE '2020-07-07', TIME '01:01:01', TIMESTAMP '2020-07-07 01:01:01';
os> SELECT 123, 'hello', false, -4.567, 9.876E-1, DATE '2020-07-07', TIME '01:01:01', TIMESTAMP '2020-07-07 01:01:01';
fetched rows / total rows = 1/1
+-------+-----------+---------+----------+---------------------+-------------------+-----------------------------------+
| 123 | 'hello' | false | -4.567 | DATE '2020-07-07' | TIME '01:01:01' | TIMESTAMP '2020-07-07 01:01:01' |
|-------+-----------+---------+----------+---------------------+-------------------+-----------------------------------|
| 123 | hello | False | -4.567 | 2020-07-07 | 01:01:01 | 2020-07-07 01:01:01 |
+-------+-----------+---------+----------+---------------------+-------------------+-----------------------------------+
+-------+-----------+---------+----------+------------+---------------------+-------------------+-----------------------------------+
| 123 | 'hello' | false | -4.567 | 9.876E-1 | DATE '2020-07-07' | TIME '01:01:01' | TIMESTAMP '2020-07-07 01:01:01' |
|-------+-----------+---------+----------+------------+---------------------+-------------------+-----------------------------------|
| 123 | hello | False | -4.567 | 0.9876 | 2020-07-07 | 01:01:01 | 2020-07-07 01:01:01 |
+-------+-----------+---------+----------+------------+---------------------+-------------------+-----------------------------------+


os> SELECT "Hello", 'Hello', "It""s", 'It''s', "It's", '"Its"', 'It\'s', 'It\\\'s', "\I\t\s"
Expand Down
45 changes: 45 additions & 0 deletions docs/user/ppl/functions/expressions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,51 @@ Introduction

Expressions, particularly value expressions, are those which return a scalar value. Expressions have different types and forms. For example, there are literal values as atom expression and arithmetic, predicate and function expression built on top of them. And also expressions can be used in different clauses, such as using arithmetic expression in ``Filter``, ``Stats`` command.

Literal Values
==============

Description
-----------

A literal is a symbol that represents a value. The most common literal values include:

1. Numeric literals: specify numeric values such as integer and floating-point numbers.
2. String literals: specify a string enclosed by single or double quotes.
3. Boolean literals: ``true`` or ``false``.
4. Date and Time literals: DATE 'YYYY-MM-DD' represent the date, TIME 'hh:mm:ss' represent the time, TIMESTAMP 'YYYY-MM-DD hh:mm:ss' represent the timestamp.

Examples
--------

Here is an example for different type of literals::

os> source=accounts | eval `123`=123, `'hello'`='hello', `false`=false, `-4.567`=-4.567, `9.876E-1`=9.876E-1, `DATE '2020-07-07'`=DATE '2020-07-07', `TIME '01:01:01'`=TIME '01:01:01', `TIMESTAMP '2020-07-07 01:01:01'`=TIMESTAMP '2020-07-07 01:01:01' | fields `123`, `'hello'`, `false`, `-4.567`, `9.876E-1`, `DATE '2020-07-07'`, `TIME '01:01:01'`, `TIMESTAMP '2020-07-07 01:01:01'` | head 1;
fetched rows / total rows = 1/1
+-------+-----------+---------+----------+------------+---------------------+-------------------+-----------------------------------+
| 123 | 'hello' | false | -4.567 | 9.876E-1 | DATE '2020-07-07' | TIME '01:01:01' | TIMESTAMP '2020-07-07 01:01:01' |
|-------+-----------+---------+----------+------------+---------------------+-------------------+-----------------------------------|
| 123 | hello | False | -4.567 | 0.9876 | 2020-07-07 | 01:01:01 | 2020-07-07 01:01:01 |
+-------+-----------+---------+----------+------------+---------------------+-------------------+-----------------------------------+


os> source=accounts | eval `"Hello"`="Hello", `'Hello'`='Hello', `"It""s"`="It""s", `'It''s'`='It''s', `"It's"`="It's", `'"Its"'`='"Its"', `'It\'s'`='It\'s', `'It\\\'s'`='It\\\'s', `"\I\t\s"`="\I\t\s" | fields `"Hello"`, `'Hello'`, `"It""s"`, `'It''s'`, `"It's"`, `'"Its"'`, `'It\'s'`, `'It\\\'s'`, `"\I\t\s"` | head 1;
fetched rows / total rows = 1/1
+-----------+-----------+-----------+-----------+----------+-----------+-----------+-------------+------------+
| "Hello" | 'Hello' | "It""s" | 'It''s' | "It's" | '"Its"' | 'It\'s' | 'It\\\'s' | "\I\t\s" |
|-----------+-----------+-----------+-----------+----------+-----------+-----------+-------------+------------|
| Hello | Hello | It"s | It's | It's | "Its" | It's | It\'s | \I\t\s |
+-----------+-----------+-----------+-----------+----------+-----------+-----------+-------------+------------+


os> source=accounts | eval `{DATE '2020-07-07'}`={DATE '2020-07-07'}, `{TIME '01:01:01'}`={TIME '01:01:01'}, `{TIMESTAMP '2020-07-07 01:01:01'}`={TIMESTAMP '2020-07-07 01:01:01'} | fields `{DATE '2020-07-07'}`, `{TIME '01:01:01'}`, `{TIMESTAMP '2020-07-07 01:01:01'}` | head 1;
fetched rows / total rows = 1/1
+-----------------------+---------------------+-------------------------------------+
| {DATE '2020-07-07'} | {TIME '01:01:01'} | {TIMESTAMP '2020-07-07 01:01:01'} |
|-----------------------+---------------------+-------------------------------------|
| 2020-07-07 | 01:01:01 | 2020-07-07 01:01:01 |
+-----------------------+---------------------+-------------------------------------+


Arithmetic Operators
====================

Expand Down
28 changes: 28 additions & 0 deletions integ-test/src/test/java/org/opensearch/sql/ppl/DataTypeIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@
import static org.opensearch.sql.legacy.SQLIntegTestCase.Index.DATA_TYPE_NUMERIC;
import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_DATATYPE_NONNUMERIC;
import static org.opensearch.sql.legacy.TestsConstants.TEST_INDEX_DATATYPE_NUMERIC;
import static org.opensearch.sql.util.MatcherUtils.rows;
import static org.opensearch.sql.util.MatcherUtils.schema;
import static org.opensearch.sql.util.MatcherUtils.verifyDataRows;
import static org.opensearch.sql.util.MatcherUtils.verifySchema;

import java.io.IOException;
Expand Down Expand Up @@ -75,4 +77,30 @@ public void test_long_integer_data_type() throws IOException {
schema("long1", "long"),
schema("long2", "long"));
}

@Test
public void test_exponent_literal_converting_to_double_type() throws IOException {
JSONObject result =
executeQuery(
String.format(
"source=%s | eval `9e1` = 9e1, `+9e+1` = +9e+1, `900e-1` = 900e-1, `9.0e1` ="
+ " 9.0e1, `9.0e+1` = 9.0e+1, `9.0E1` = 9.0E1, `.9e+2` = .9e+2, `0.09e+3` ="
+ " 0.09e+3, `900.0e-1` = 900.0e-1, `-900.0E-1` = -900.0E-1 | fields `9e1`,"
+ " `+9e+1`, `900e-1`, `9.0e1`, `9.0e+1`, `9.0E1`, `.9e+2`, `0.09e+3`,"
+ " `900.0e-1`, `-900.0E-1`",
TEST_INDEX_DATATYPE_NUMERIC));
verifySchema(
result,
schema("9e1", "double"),
schema("+9e+1", "double"),
schema("900e-1", "double"),
schema("9.0e1", "double"),
schema("9.0e+1", "double"),
schema("9.0E1", "double"),
schema(".9e+2", "double"),
schema("0.09e+3", "double"),
schema("900.0e-1", "double"),
schema("-900.0E-1", "double"));
verifyDataRows(result, rows(90.0, 90.0, 90.0, 90.0, 90.0, 90.0, 90.0, 90.0, 90.0, -90.0));
}
}
8 changes: 6 additions & 2 deletions ppl/src/main/antlr/OpenSearchPPLLexer.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,8 +385,9 @@ Y: 'Y';
//STRING_LITERAL: DQUOTA_STRING | SQUOTA_STRING | BQUOTA_STRING;
ID: ID_LITERAL;
CLUSTER: CLUSTER_PREFIX_LITERAL;
INTEGER_LITERAL: DEC_DIGIT+;
DECIMAL_LITERAL: (DEC_DIGIT+)? '.' DEC_DIGIT+;
INTEGER_LITERAL: INTEGER_NUM;
DECIMAL_LITERAL: DECIMAL_NUM;
EXPONENT_LITERAL: INTEGER_NUM EXPONENT_NUM | DECIMAL_NUM EXPONENT_NUM;

fragment DATE_SUFFIX: ([\-.][*0-9]+)+;
fragment ID_LITERAL: [@*A-Z]+?[*A-Z_\-0-9]*;
Expand All @@ -396,6 +397,9 @@ DQUOTA_STRING: '"' ( '\\'. | '""' | ~('"'| '\\') )* '"';
SQUOTA_STRING: '\'' ('\\'. | '\'\'' | ~('\'' | '\\'))* '\'';
BQUOTA_STRING: '`' ( '\\'. | '``' | ~('`'|'\\'))* '`';
fragment DEC_DIGIT: [0-9];
fragment INTEGER_NUM: DEC_DIGIT+;
fragment DECIMAL_NUM: (DEC_DIGIT+)? '.' DEC_DIGIT+;
fragment EXPONENT_NUM: 'E' [-+]? DEC_DIGIT+;


ERROR_RECOGNITION: . -> channel(ERRORCHANNEL);
6 changes: 6 additions & 0 deletions ppl/src/main/antlr/OpenSearchPPLParser.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,7 @@ percentileApproxFunction
numericLiteral
: integerLiteral
| decimalLiteral
| exponentLiteral
;

// expressions
Expand Down Expand Up @@ -690,6 +691,7 @@ literalValue
| stringLiteral
| integerLiteral
| decimalLiteral
| exponentLiteral
| booleanLiteral
| datetimeLiteral //#datetime
;
Expand All @@ -711,6 +713,10 @@ decimalLiteral
: (PLUS | MINUS)? DECIMAL_LITERAL
;

exponentLiteral
: (PLUS | MINUS)? EXPONENT_LITERAL
;

booleanLiteral
: TRUE
| FALSE
Expand Down
6 changes: 6 additions & 0 deletions ppl/src/main/java/org/opensearch/sql/ppl/parser/AstExpressionBuilder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import static org.opensearch.sql.ppl.antlr.parser.OpenSearchPPLParser.DistinctCountFunctionCallContext;
import static org.opensearch.sql.ppl.antlr.parser.OpenSearchPPLParser.EvalClauseContext;
import static org.opensearch.sql.ppl.antlr.parser.OpenSearchPPLParser.EvalFunctionCallContext;
import static org.opensearch.sql.ppl.antlr.parser.OpenSearchPPLParser.ExponentLiteralContext;
import static org.opensearch.sql.ppl.antlr.parser.OpenSearchPPLParser.FieldExpressionContext;
import static org.opensearch.sql.ppl.antlr.parser.OpenSearchPPLParser.IdentsAsQualifiedNameContext;
import static org.opensearch.sql.ppl.antlr.parser.OpenSearchPPLParser.IdentsAsTableQualifiedNameContext;
Expand Down Expand Up @@ -343,6 +344,11 @@ public UnresolvedExpression visitDecimalLiteral(DecimalLiteralContext ctx) {
return new Literal(Double.valueOf(ctx.getText()), DataType.DOUBLE);
}

@Override
public UnresolvedExpression visitExponentLiteral(ExponentLiteralContext ctx) {
return new Literal(Double.valueOf(ctx.getText()), DataType.DOUBLE);
}

@Override
public UnresolvedExpression visitBooleanLiteral(BooleanLiteralContext ctx) {
return new Literal(Boolean.valueOf(ctx.getText()), DataType.BOOLEAN);
Expand Down
22 changes: 22 additions & 0 deletions ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -417,4 +417,26 @@ public void testCanParseTimestampdiffFunction() {
new PPLSyntaxParser()
.parse("SOURCE=test | eval k = TIMESTAMPDIFF(WEEK,'2003-01-02','2003-01-02')"));
}

@Test
public void testExponentLiteralShouldPass() {
List<String> scientificNotationList =
List.of(
"9e1",
"+9e+1",
"9e-1",
"-9e1",
"9.0e1",
"9.0e+1",
"9.0E1",
".9e+2",
"0.9e+2",
"900e-1",
"900.0E-1");
for (String exponentLiteral : scientificNotationList) {
ParseTree tree =
new PPLSyntaxParser().parse("search source=t | eval scientific = " + exponentLiteral);
assertNotEquals(null, tree);
}
}
}
38 changes: 38 additions & 0 deletions ppl/src/test/java/org/opensearch/sql/ppl/parser/AstExpressionBuilderTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -502,6 +502,44 @@ public void testDoubleLiteralExpr() {
"source=t b=0.1", filter(relation("t"), compare("=", field("b"), doubleLiteral(0.1))));
}

@Test
public void testExponentLiteralExpr() {
List<String> scientificNotationList =
List.of(
"9e1",
"+9e+1",
"900e-1",
"9.0e1",
"9.0e+1",
"9.0E1",
".9e+2",
"0.09e+3",
"900.0e-1",
"+900.0E-1");
for (String scientificNotation : scientificNotationList) {
assertEqual(
"source=t b=" + scientificNotation,
filter(relation("t"), compare("=", field("b"), doubleLiteral(90.0))));
}
List<String> negativeScientificNotationList =
List.of(
"-9e1",
"-9e+1",
"-900e-1",
"-9.0e1",
"-9.0e+1",
"-9.0E1",
"-.9e+2",
"-0.09e+3",
"-900.0e-1",
"-900.0E-1");
for (String negativeScientificNotation : negativeScientificNotationList) {
assertEqual(
"source=t b=" + negativeScientificNotation,
filter(relation("t"), compare("=", field("b"), doubleLiteral(-90.0))));
}
}

@Test
public void testBooleanLiteralExpr() {
assertEqual(
Expand Down
Loading