Implement support for primitive data types for dataflow.yaml

[meua]

Implement support for primitive data types

---

Data type comparison between yaml and many other languages

Here is a table comparing the data types supported by YAML and several programming languages:

Data Type	YAML	Python	Rust	Go	C	C++
Scalars	✓	✓	✓	✓	✓	✓
Sequences	✓	list, tuple	Vec, Array	slice, array	array	array, vector
Mappings	✓	dict	HashMap	map	struct, union	map, unordered_map
Dates	✓	datetime	chrono	time	time	chrono
Times	✓	datetime	chrono	time	time	chrono
Time Intervals	✓	timedelta	duration	duration	N/A	duration
Regular Expressions	✓	re	regex	regexp	regex	regex, boost::regex
Binary Data	✓	bytes, bytearray	Vec<u8>	[]byte	char[]	vector<char>
Ordered Maps	✓	OrderedDict	BTreeMap	N/A	N/A	map, multimap
Sets	✓	set	HashSet	map[type]bool	N/A	set, multiset

Note that this table is not exhaustive and some programming languages may support additional data types beyond those listed here. Additionally, the data types listed for each programming language may not be an exact match for those supported by YAML, but are meant to provide a rough comparison.

YAML supports multiple data types, including:

• Scalar types: including strings, integers, floating-point numbers, booleans, and null.
• Sequence types: also known as lists or arrays, used to represent a group of values of the same or different types.
• Mapping types: also known as dictionaries, hash tables, or associative arrays, used to represent a set of key-value pairs where keys and values can be any type of value, but keys must be unique.

In addition to the basic data types mentioned above, YAML also supports some extended data types, such as dates, times, regular expressions, binary data, and more. These extended data types can be identified using specific tags, such as !date, !time, !regex, !binary, and so on.

Here is a list of the data types supported by YAML:

• Strings
• Integers
• Floating-point numbers
• Booleans
• Null
• Lists/Arrays
• Dictionaries/Maps
• Dates
• Times
• Timestamps
• Time intervals
• Regular expressions
• Binary data
• Ordered maps/Lists
• Sets

Note that the support for these data types may vary depending on the implementation of the YAML parser and the specific version of the YAML specification being used.

Implementation examples of yaml parsing in different languages

1. How to parse YAML data types using Rust

To parse YAML data types in Rust, you can use YAML libraries such as serde_yaml.

First, you need to add serde and serde_yaml to your project dependencies. Add the following lines to your Cargo.toml file:

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_yaml = "0.8"

Then, you can use the serde_yaml::from_str function to parse a YAML string into a Rust struct. For example, assuming you have the following YAML data:

name: John Doe
age: 42
is_student: true

You can define a corresponding Rust struct to represent this data:

use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct Person {
    name: String,
    age: u32,
    is_student: bool,
}

Now, you can use the serde_yaml::from_str function to parse the YAML string into a Person struct:

use serde_yaml;

fn main() {
    let yaml_str = r#"
        name: John Doe
        age: 42
        is_student: true
    "#;

    let person: Person = serde_yaml::from_str(yaml_str).unwrap();
    println!("{:?}", person);
}

The output will be:

Person { name: "John Doe", age: 42, is_student: true }

You can modify the Rust struct and the YAML data as needed to fit your requirements. Note that you need to match the fields of the Rust struct with the keys of the YAML data, otherwise the parsing will fail.

2. How to parse YAML data types using Python

To parse YAML data types in Python, you can use the PyYAML library.

First, you need to install PyYAML. You can do this by running the following command:

pip install pyyaml

Then, you can use the yaml.load function to parse a YAML string into a Python data type. For example, assuming you have the following YAML data:

name: John Doe
age: 42
is_student: true

You can use the following code to parse it into a Python dictionary:

import yaml

yaml_str = """
name: John Doe
age: 42
is_student: true
"""

data = yaml.load(yaml_str, Loader=yaml.FullLoader)
print(data)

The output will be:

{'name': 'John Doe', 'age': 42, 'is_student': True}

You can modify the Python data and the YAML data as needed to fit your requirements. Note that when using the yaml.load function, you need to be careful because it can execute arbitrary code, so you should not pass unknown or untrusted YAML data to it.

3. How to parse YAML data types using C/C++

To parse YAML data types in C/C++, you can use the libyaml library.

First, you need to install the libyaml library. You can do this by running the following command on Linux:

sudo apt-get install libyaml-cpp-dev

Then, you can use the following code to parse a YAML file into C/C++ data types. Assuming you have the following YAML data:

name: John Doe
age: 42
is_student: true

You can use the following code to parse it into a C/C++ struct:

#include <yaml-cpp/yaml.h>
#include <iostream>
#include <string>

struct Person {
    std::string name;
    int age;
    bool is_student;
};

int main() {
    YAML::Node config = YAML::LoadFile("dataflow.yaml");

    Person person;
    person.name = config["name"].as<std::string>();
    person.age = config["age"].as<int>();
    person.is_student = config["is_student"].as<bool>();

    std::cout << "Name: " << person.name << std::endl;
    std::cout << "Age: " << person.age << std::endl;
    std::cout << "Is student: " << std::boolalpha << person.is_student << std::endl;

    return 0;
}

The output will be:

Name: John Doe
Age: 42
Is student: true

You can modify the C/C++ struct and the YAML data as needed to fit your requirements. Note that when using the libyaml library, you need to be careful because it can execute arbitrary code, so you should not pass unknown or untrusted YAML data to it.

4. How to parse YAML data types using Golang

Here is a simplified example program to parse YAML data and output it to the console:

package main

import (
    "fmt"
    "io/ioutil"
    "log"

    "To parse YAML data types in Golang, you can use the "gopkg.in/yaml.v3" package.

First, you need to install the package. You can do this by running the following command:

go get gopkg.in/yaml.v3

Then, you can use the following code to parse a YAML file into Golang data types. Assuming you have the following YAML data:

```yaml
name: John Doe
age: 42
is_student: true

You can use the following code to parse it into a Golang struct:

package main

import (
    "fmt"
    "io/ioutil"
    "log"

    "gopkg.in/yaml.v3"
)

type Person struct {
    Name      string `yaml:"name"`
    Age       int    `yaml:"age"`
    IsStudent bool   `yaml:"is_student"`
}

func main() {
    data, err := ioutil.ReadFile("dataflow.yaml")
    if err != nil {
        log.Fatalf("error: %v", err)
    }

    var person Person
    err = yaml.Unmarshal(data, &person)
    if err != nil {
        log.Fatalf("error: %v", err)
    }

    fmt.Printf("Name: %s\n", person.Name)
    fmt.Printf("Age: %d\n", person.Age)
    fmt.Printf("Is student: %t\n", person.IsStudent)
}

The output will be:

Name: John Doe
Age: 42
Is student: true

You can modify the Golang struct and the YAML data as needed to fit your requirements. Note that when using the "gopkg.in/yaml.v3" package, you need to be careful because it can execute arbitrary code, so you should not pass unknown or untrusted YAML data to it.

Use apache arrow to realize type conversion between yaml and multiple languages

Apache Arrow is a cross-language, in-memory data structure that enables efficient data exchange between multiple programming languages. While Apache Arrow is not itself a data serialization format, it provides a unified data model that makes it easy to convert data types between different programming languages, including YAML.

With Apache Arrow, you can convert your data into Arrow format and then share and manipulate that data across different programming languages without the need for manual type conversions or complex serialization and deserialization code. In fact, many programming languages already support the Arrow format, including C/C++, Python, Rust, Golang, and Java, among others. Therefore, if your application involves data exchange between multiple programming languages, Apache Arrow may be a good choice.

[heyong4725]

@phil-opp @haixuanTao hope this is aligned with the ROS bridge efforts ongoing such that Dora user could specify data types between dataflow nodes

[haixuanTao]

So maybe for reference,

One of the key value of dora is to be performant. One of the reason of using arrow is to use zero-copy messages as it implement a well defined data memory layout that can be used across languages without copy.

Issues of serialization deserialisation is:

• it is slow
• most of the time require a copy
• most of the time requires a compiling step ( e.g. ROS 2) which might be hard to achieve in Python as it is interpreted.

This is why we're trying to use Apache Arrow as a cross-language data format.

I think that you mention YAML as an example, but, for the sake of clarity for future readers, I don't think we can use YAML as a performant way of exchanging data.

[meua]

I have updated the diagram, the YAML dataflow declarative definition needs to be parsed by rust, here's my humble opinion on Dora.

[Euraxluo]

I don't understand the question. Doesn't Dora now use arrow for data exchange

[haixuanTao]

Yes, so, we currently have a limited number of types supported by arrow but we hope to support additional types soon.
I think that the main idea of this discussion is to support more types.