Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds changes for sequence data type #144

Merged
merged 3 commits into from
Sep 25, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions code-gen-projects/schema/sequence.isl
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
type::{
name: sequence,
type: list,
element: string
}
236 changes: 207 additions & 29 deletions src/bin/ion/commands/generate/generator.rs

Large diffs are not rendered by default.

74 changes: 58 additions & 16 deletions src/bin/ion/commands/generate/model.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ use std::fmt::{Display, Formatter};
// _Note: This model will eventually use a map (FullQualifiedTypeReference, DataModel) to resolve some the references in container types(sequence or structure)._
// TODO: This is not yet used in the implementation, modify current implementation to use this data model.
use crate::commands::generate::context::SequenceType;
use crate::commands::generate::utils::Language;
use serde::Serialize;
use serde_json::Value;

Expand Down Expand Up @@ -70,10 +71,10 @@ impl DataModelNode {
false
}

pub fn fully_qualified_type_ref(&mut self) -> Option<FullyQualifiedTypeReference> {
pub fn fully_qualified_type_ref<L: Language>(&mut self) -> Option<FullyQualifiedTypeReference> {
self.code_gen_type
.as_ref()
.and_then(|t| t.fully_qualified_type_ref())
.and_then(|t| t.fully_qualified_type_ref::<L>())
}
}

Expand Down Expand Up @@ -170,14 +171,13 @@ impl FullyQualifiedTypeReference {
#[derive(Debug, Clone, PartialEq, Serialize)]
pub enum AbstractDataType {
// Represents a scalar type which also has a name attached to it and is nominally distinct from its base type.
#[allow(dead_code)]
WrappedScalar(WrappedScalar),
// Represents a scalar value (e.g. a string or integer or user defined type)
#[allow(dead_code)]
Scalar(Scalar),
// A series of zero or more values whose type is described by the nested `element_type`
#[allow(dead_code)]
Sequence(Sequence),
// Represents a sequence type which also has name attached to it and is nominally distinct from its enclosed type.
WrappedSequence(WrappedSequence),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to have to create corresponding Wrapped* variants for all of the non-wrapped variants? Why/why not?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. I think its in general there are only 3 types structure, sequence and scalar. Where only nested structure would have a name but scalar and sequence would not have name. Hence to separate out whether they are named or not I have them as a separate variant. I think having them separate allows us to perform specific operations for these types which will be stored as property instead of entirely different class in the generated code.

e.g. For below ISL:

type::{
   name: my_type,
   fields: {
        foo: {type: list, element: string} // this will not create a new nested class in the generated code
   }
}

Here's the generated code in Java:

class MyType {
   private java.util.ArrayList<String> foo;
   ...
}

whereas for structure type like following:

type::{
   name: my_type,
   fields: {
        foo: {
           fields: {
               bar: string
           }
        } 
   }
}

Here's the generated code in Java, which creates a new nested type for this nested structure:

class MyType {
   private NestedType1 foo;
   ...
   
   class NestedType1 {
       private String bar;
       ...
   }
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need the non-wrapped variant of sequence? We have parameterized, fully-qualified type references, so we should be able to model the non-wrapped lists as e.g. java.util.List<org.example.MyGeneratedClass>.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WrpapedSequence use FullyQualifiedTypeName (not FullyqualifiedTypeReference). And WrappedSequence have a name, for example:

type::{
   name: foo, // this is the name of the wrapped sequence
   type: list,
   element: string
}

Above wrapped sequence's name will be stored as org.example.Foo .
Whereas Sequence would not have a name, for example:

{type: list, element: string} // does not have a name

We can surely store it as fully-qualified type references and thats what we already we store the element property as FullyWualfiiedTypeReference and then when storing this type into TypeStore we use java.util.ArrayList<String> but it still wouldn't have name (like the wrapped sequence has name as foo)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but the non-wrapped version (Sequence) does not have a name and does not have a class generated for it. Therefore, why do we need to have the unnamed Sequence model? You don't need a reference to the Sequence when you can just use a parameterized reference such as java.util.ArrayList<java.lang.String>.

For example, if you have this schema, the { type: list, element: string } does not need a Sequence to be created to represent it. Instead, the field bar should have the data model equivalent of a reference to java.util.ArrayList<java.lang.String>.

type::{
  name: foo,
  fields: {
    bar: { type: list, element: string }
  }
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have created an issue for this: #145. Need to look at if removing the non-wrapped variants has any requirements for how they are rendered in templates.

// A collection of field name/value pairs (e.g. a map)
Structure(Structure),
}
Expand All @@ -192,20 +192,30 @@ impl AbstractDataType {
AbstractDataType::Scalar(Scalar { doc_comment, .. }) => {
doc_comment.as_ref().map(|s| s.as_str())
}
AbstractDataType::Sequence(Sequence { doc_comment, .. }) => Some(doc_comment.as_str()),
AbstractDataType::Sequence(Sequence { doc_comment, .. }) => {
doc_comment.as_ref().map(|s| s.as_str())
}
AbstractDataType::WrappedSequence(WrappedSequence { doc_comment, .. }) => {
doc_comment.as_ref().map(|s| s.as_str())
}
AbstractDataType::Structure(Structure { doc_comment, .. }) => {
doc_comment.as_ref().map(|s| s.as_str())
}
}
}

pub fn fully_qualified_type_ref(&self) -> Option<FullyQualifiedTypeReference> {
pub fn fully_qualified_type_ref<L: Language>(&self) -> Option<FullyQualifiedTypeReference> {
match self {
AbstractDataType::WrappedScalar(w) => {
Some(w.fully_qualified_type_name().to_owned().into())
}
AbstractDataType::Scalar(s) => Some(s.base_type.to_owned()),
AbstractDataType::Sequence(seq) => Some(seq.element_type.to_owned()),
AbstractDataType::Sequence(seq) => {
Some(L::target_type_as_sequence(seq.element_type.to_owned()))
}
AbstractDataType::WrappedSequence(seq) => {
Some(L::target_type_as_sequence(seq.element_type.to_owned()))
}
AbstractDataType::Structure(structure) => Some(structure.name.to_owned().into()),
}
}
Expand Down Expand Up @@ -289,8 +299,6 @@ impl WrappedScalar {

/// Represents series of zero or more values whose type is described by the nested `element_type`
/// and sequence type is described by nested `sequence_type` (e.g. List or SExp).
/// If there is no `element` constraint present in schema type then `element_type` will be None.
/// If there is no `type` constraint present in schema type then `sequence_type` will be None.
/// e.g. Given below ISL,
/// ```
/// type::{
Expand All @@ -308,11 +316,12 @@ impl WrappedScalar {
#[allow(dead_code)]
#[derive(Debug, Clone, Builder, PartialEq, Serialize)]
#[builder(setter(into))]
pub struct Sequence {
pub struct WrappedSequence {
// Represents the fully qualified name for this data model
name: FullyQualifiedTypeName,
// Represents doc comment for the generated code
doc_comment: String,
#[builder(default)]
doc_comment: Option<String>,
// Represents the fully qualified name with namespace where each element of vector stores a module name or class/struct name.
// _Note: that a hashmap with (FullQualifiedTypeReference, DataModel) pairs will be stored in code generator to get information on the element_type name used here._
element_type: FullyQualifiedTypeReference,
Expand All @@ -326,6 +335,41 @@ pub struct Sequence {
source: IslType,
}

/// Represents series of zero or more values whose type is described by the nested `element_type`
/// and sequence type is described by nested `sequence_type` (e.g. List or SExp).
/// e.g. Given below ISL,
/// ```
/// type::{
/// name: sequence_type,
/// element: int,
/// type: list
/// }
/// ```
/// Corresponding generated code in Rust would look like following:
/// ```
/// struct SequenceType {
/// value: Vec<i64>
/// }
/// ```
#[derive(Debug, Clone, Builder, PartialEq, Serialize)]
#[builder(setter(into))]
pub struct Sequence {
// Represents doc comment for the generated code
#[builder(default)]
pub(crate) doc_comment: Option<String>,
// Represents the fully qualified name with namespace where each element of vector stores a module name or class/struct name.
// _Note: that a hashmap with (FullQualifiedTypeReference, DataModel) pairs will be stored in code generator to get information on the element_type name used here._
pub(crate) element_type: FullyQualifiedTypeReference,
// Represents the type of the sequence which is either `sexp` or `list`.
pub(crate) sequence_type: SequenceType,
// Represents the source ISL type which can be used to get other constraints useful for this type.
// For example, getting the length of this sequence from `container_length` constraint or getting a `regex` value for string type.
// This will also be useful for `text` type to verify if this is a `string` or `symbol`.
// TODO: `IslType` does not implement `Serialize`, define a custom implementation or define methods on this field that returns values which could be serialized.
#[serde(skip_serializing)]
pub(crate) source: IslType,
}

/// Represents a collection of field name/value pairs (e.g. a map)
/// e.g. Given below ISL,
/// ```
Expand Down Expand Up @@ -448,8 +492,7 @@ mod model_tests {
#[test]
fn sequence_builder_test() {
let expected_seq = Sequence {
name: vec![],
doc_comment: "This is sequence type of strings".to_string(),
doc_comment: Some("This is sequence type of strings".to_string()),
element_type: FullyQualifiedTypeReference {
type_name: vec!["String".to_string()],
parameters: vec![],
Expand All @@ -465,8 +508,7 @@ mod model_tests {

// sets all the information about the sequence except the `element_type`
seq_builder
.name(vec![])
.doc_comment("This is sequence type of strings")
.doc_comment(Some("This is sequence type of strings".to_string()))
.sequence_type(SequenceType::List)
.source(anonymous_type(vec![
type_constraint(named_type_ref("list")),
Expand Down
9 changes: 9 additions & 0 deletions src/bin/ion/commands/generate/result.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
use crate::commands::generate::model::{
ScalarBuilderError, SequenceBuilderError, StructureBuilderError, WrappedScalarBuilderError,
WrappedSequenceBuilderError,
};
use ion_schema::result::IonSchemaError;
use thiserror::Error;
Expand Down Expand Up @@ -71,6 +72,14 @@ impl From<SequenceBuilderError> for CodeGenError {
}
}

impl From<WrappedSequenceBuilderError> for CodeGenError {
fn from(value: WrappedSequenceBuilderError) -> Self {
CodeGenError::DataModelBuilderError {
description: value.to_string(),
}
}
}

impl From<StructureBuilderError> for CodeGenError {
fn from(value: StructureBuilderError) -> Self {
CodeGenError::DataModelBuilderError {
Expand Down
10 changes: 5 additions & 5 deletions src/bin/ion/commands/generate/templates/java/class.templ
Original file line number Diff line number Diff line change
Expand Up @@ -119,13 +119,13 @@ import java.io.IOException;
{% set field_value = field_val.0 | fully_qualified_type_name %}
writer.setFieldName("{{ field_name }}");
{% if field_value | is_built_in_type == false %}
this.{{ field_name | camel }}.writeTo(writer);
{% else %}
{% if field_value is containing("ArrayList") %}
{% if field_value is containing("ArrayList") %}
{{ util_macros::write_as_sequence(field_value=field_value,field_name=field_name,type_store=type_store) }}
{% else %}
{% else %}
this.{{ field_name | camel }}.writeTo(writer);
{% endif %}
{% else %}
writer.write{{ field_value | replace(from="double", to="float") | replace(from="boolean", to="bool") | upper_camel }}(this.{{ field_name | camel }});
{% endif %}
{% endif %}
{% endfor %}
writer.stepOut();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@
{{ sequence_info["element_type"] | fully_qualified_type_name }} value
{% elif inline_type.code_gen_type is containing("Scalar") %}
{% set scalar_info = model.code_gen_type["WrappedScalar"] %}
{% set base_type = scalar_info["name"]["parameters"][0] | fully_qualified_type_name %}
{% set base_type = scalar_info["base_type"] | fully_qualified_type_name %}
{{ base_type }} value
{% endif %}
{% endmacro %}
68 changes: 35 additions & 33 deletions src/bin/ion/commands/generate/templates/java/sequence.templ
Original file line number Diff line number Diff line change
@@ -1,81 +1,83 @@
package {{ namespace }};
import java.util.ArrayList;
{% macro sequence(model) %}

{% if is_nested == false %}
{% set full_namespace = namespace | join(sep=".") %}

package {{ full_namespace }};
import com.amazon.ion.IonReader;
import com.amazon.ion.IonException;
import com.amazon.ion.IonWriter;
import com.amazon.ion.IonType;
import java.io.IOException;
{% endif %}

{# Verify that the abstract data type is a sequence type and store information for this sequence value #}
{% set sequence_info = model.code_gen_type["WrappedSequence"] %}

public class {{ target_kind_name }} {
private {{ fields[0].value_type }} value;
class {{ model.name }} {
private java.util.ArrayList<{{ sequence_info["element_type"] | fully_qualified_type_name }}> value;

public {{ target_kind_name }}() {}
public {{ model.name }}() {}

public {{ fields[0].value_type }} getValue() {
public java.util.ArrayList<{{ sequence_info["element_type"] | fully_qualified_type_name }}> getValue() {
return this.value;
}

public void setValue({{ fields[0].value_type }} value) {
public void setValue(java.util.ArrayList<{{ sequence_info["element_type"] | fully_qualified_type_name }}> value) {
this.value = value;
return;
}

/**
* Reads a {{ target_kind_name }} from an {@link IonReader}.
* Reads a {{ model.name }} from an {@link IonReader}.
*
* This method does not advance the reader at the current level.
* The caller is responsible for positioning the reader on the value to read.
*/
public static {{ target_kind_name }} readFrom(IonReader reader) {
public static {{ model.name }} readFrom(IonReader reader) {
{# Initializes all the fields of this class #}
{{ fields[0].value_type }} value =
{% if fields[0].value_type == "boolean" %}
false
{% elif fields[0].value_type == "int" or fields[0].value_type == "double" %}
0
{% else %}
null
{% endif %};
java.util.ArrayList<{{ sequence_info["element_type"] | fully_qualified_type_name }}> value = new java.util.ArrayList<{{ sequence_info["element_type"] | fully_qualified_type_name }}>();
{# Reads `Sequence` class with a single field `value` that is an `ArrayList` #}
if(reader.getType() != IonType.{{ abstract_data_type["Sequence"].sequence_type | upper }}) {
throw new IonException("Expected {{ abstract_data_type["Sequence"].sequence_type }}, found " + reader.getType() + " while reading {{ fields[0].name | camel }}.");
if(reader.getType() != IonType.{{ sequence_info["sequence_type"] | upper }}) {
throw new IonException("Expected {{ sequence_info["sequence_type"] }}, found " + reader.getType() + " while reading value.");
}
reader.stepIn();
value = new {{ fields[0].value_type }}();
{# Iterate through the `ArrayList` and read each element in it based on the data type provided in `abstract_data_type[Sequence]` #}
{# Iterate through the `ArrayList` and read each element in it based on the data type provided in `sequence_info["sequence_type"]` #}
while (reader.hasNext()) {
reader.next();
{% if abstract_data_type["Sequence"].element_type | is_built_in_type == false %}
value.add({{ abstract_data_type["Sequence"].element_type }}.readFrom(reader));
{% elif abstract_data_type["Sequence"].element_type == "bytes[]" %}
{% if sequence_info["element_type"] |fully_qualified_type_name | is_built_in_type == false %}
value.add({{ sequence_info["element_type"] | fully_qualified_type_name }}.readFrom(reader));
{% elif sequence_info["element_type"] | fully_qualified_type_name == "bytes[]" %}
value.add(reader.newBytes());
{% else %}
value.add(reader.{{ abstract_data_type["Sequence"].element_type | camel }}Value());
value.add(reader.{{ sequence_info["element_type"] | fully_qualified_type_name | camel }}Value());
{% endif %}
}
reader.stepOut();
{{ target_kind_name }} {{ target_kind_name | camel }} = new {{ target_kind_name }}();
{{ target_kind_name | camel }}.value = value;
{{ model.name }} {{ model.name | camel }} = new {{ model.name }}();
{{ model.name | camel }}.value = value;

return {{ target_kind_name | camel }};
return {{ model.name | camel }};
}

/**
* Writes a {{ target_kind_name }} as Ion from an {@link IonWriter}.
* Writes a {{ model.name }} as Ion from an {@link IonWriter}.
*
* This method does not close the writer after writing is complete.
* The caller is responsible for closing the stream associated with the writer.
*/
public void writeTo(IonWriter writer) throws IOException {
{# Writes `Sequence` class with a single field `value` that is an `ArrayList` as an Ion sequence #}
writer.stepIn(IonType.{{ abstract_data_type["Sequence"].sequence_type | upper }});
for ({{ abstract_data_type["Sequence"].element_type }} value: this.value) {
{% if abstract_data_type["Sequence"].element_type | is_built_in_type == false %}
writer.stepIn(IonType.{{ sequence_info["sequence_type"] | upper }});
for ({{ sequence_info["element_type"] | fully_qualified_type_name }} value: this.value) {
{% if sequence_info["element_type"] | fully_qualified_type_name | is_built_in_type == false %}
value.writeTo(writer);
{% else %}
writer.write{{ abstract_data_type["Sequence"].element_type | upper_camel }}(value);
writer.write{{ sequence_info["element_type"] | fully_qualified_type_name | replace(from="double", to="float") | replace(from="boolean", to="bool") | upper_camel }}(value);
{% endif %}
}
writer.stepOut();
}
}
{% endmacro %}
{{ self::sequence(model=model) }}
32 changes: 17 additions & 15 deletions src/bin/ion/commands/generate/templates/java/util_macros.templ
Original file line number Diff line number Diff line change
@@ -1,34 +1,36 @@
{# following macro defines statements to read a class field as sequence #}
{% macro read_as_sequence(field) %}
new {{ field.value_type }}();
{% macro read_as_sequence(field_name, field_value, type_store) %}
{% set field_value_model = type_store[field_value] %}
new {{ field_value }}();
{# Reads `Sequence` field that is an `ArrayList` #}
if(reader.getType() != IonType.{{ field.abstract_data_type["Sequence"].sequence_type | upper }}) {
throw new IonException("Expected {{ field.abstract_data_type["Sequence"].sequence_type }}, found " + reader.getType() + " while reading {{ field.name | camel }}.");
if(reader.getType() != IonType.{{ field_value_model.code_gen_type["Sequence"].sequence_type | upper }}) {
throw new IonException("Expected {{ field_value_model.code_gen_type["Sequence"].sequence_type }}, found " + reader.getType() + " while reading {{ field_name | camel }}.");
}
reader.stepIn();
{# Iterate through the `ArrayList` and read each element in it based on the data type provided in `field.abstract_data_type[Sequence]` #}
while (reader.hasNext()) {
reader.next();
{% if field.abstract_data_type["Sequence"].element_type | is_built_in_type == false %}
{{ field.name | camel }}.add({{ field.abstract_data_type["Sequence"].element_type }}.readFrom(reader));
{% elif field.abstract_data_type["Sequence"].element_type == "bytes[]" %}
{{ field.name | camel }}.add(reader.newBytes());
{% if field_value_model.code_gen_type["Sequence"].element_type | fully_qualified_type_name | is_built_in_type == false %}
{{ field_name | camel }}.add({{ field_value_model.code_gen_type["Sequence"].element_type }}.readFrom(reader));
{% elif field_value_model.code_gen_type["Sequence"].element_type == "bytes[]" %}
{{ field_name | camel }}.add(reader.newBytes());
{% else %}
{{ field.name | camel }}.add(reader.{{ field.abstract_data_type["Sequence"].element_type | camel }}Value());
{{ field_name | camel }}.add(reader.{{ field_value_model.code_gen_type["Sequence"].element_type | fully_qualified_type_name | camel }}Value());
{% endif %}
}
reader.stepOut();
{% endmacro %}
{# following macro defines statements to write a class field as sequence #}
{% macro write_as_sequence(field) %}
{% macro write_as_sequence(field_name, field_value, type_store) %}
{% set field_value_model = type_store[field_value] %}
{# Writes `Sequence` field that is an `ArrayList` as an Ion sequence #}
writer.stepIn(IonType.{{ field.abstract_data_type["Sequence"].sequence_type | upper }});
for ({{ field.abstract_data_type["Sequence"].element_type }} value: this.{{ field.name |camel }}) {
{% if field.abstract_data_type["Sequence"].element_type | is_built_in_type == false %}
writer.stepIn(IonType.{{ field_value_model.code_gen_type["Sequence"].sequence_type | upper }});
for ({{ field_value_model.code_gen_type["Sequence"].element_type | fully_qualified_type_name }} value: this.{{ field_name |camel }}) {
{% if field_value_model.code_gen_type["Sequence"].element_type | fully_qualified_type_name | is_built_in_type == false %}
value.writeTo(writer);
{% else %}
writer.write{{ field.abstract_data_type["Sequence"].element_type | upper_camel }}(value);
writer.write{{ field_value_model.code_gen_type["Sequence"].element_type | fully_qualified_type_name | replace(from="double", to="float") | replace(from="boolean", to="bool") | upper_camel }}(value);
{% endif %}
}
writer.stepOut();
{% endmacro %}
{% endmacro %}
4 changes: 3 additions & 1 deletion src/bin/ion/commands/generate/utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -245,7 +245,9 @@ impl TryFrom<&DataModelNode> for Template {
AbstractDataType::Scalar(_) | AbstractDataType::WrappedScalar(_) => {
Ok(Template::Scalar)
}
AbstractDataType::Sequence(_) => Ok(Template::Sequence),
AbstractDataType::Sequence(_) | AbstractDataType::WrappedSequence(_) => {
Ok(Template::Sequence)
}
AbstractDataType::Structure(_) => Ok(Template::Struct),
}
} else {
Expand Down
Loading