poppy is dataframe library for java, which provides common SQL operations (e.g. select, from, where, group by, order by, distinct) to process data in java.
Unlike other dataframe libraries, which keep all the data in memory, poppy process data in streaming manager. That is, it is more similar as Java8 Stream library, but relational version.
Here is a simple example. We have a Student
class
public class Student {
private int studentId;
private String name;
private int grade;
private int room;
private int height;
private int weight;
...
}
In SQL, we have a query like this
select
grade,
room,
avg(weight) as weight,
avg(height) as height
from Student
group by grade, room
order by grade, room
Here is the Poppy's version
List<Student> students = ...;
DataFrame
.from(students, Student.class)
.groupby("grade", "room")
.aggregate(
avgLong("weight").as("weight"),
avgLong("height").as("height"))
.sort("grade", "room")
.print();
Java 8 or higher
Poppy's package is managed by JCenter repository.
Maven
<dependency>
<groupId>io.tenmax</groupId>
<artifactId>poppy</artifactId>
<version>0.1.8</version>
<type>pom</type>
</dependency>
Gradle
compile 'io.tenmax:poppy:0.1.8'
- Support the most common operations in SQL. e.g. select, from, where, group by, order by, distinct
- Support the most common aggregation functions in SQL. e.g. avg(), sum(), count(), min(), max()
- Custom aggregation functions. by java.util.stream.Collector
- Partition support. Partition is the unit of parallelism. Multiple partitions allow you processing data concurrently.
- Multi-threaded support. For CPU-bound jobs, it leverages all your CPU resources for better performance; for IO-bound jobs, it reduces the waiting time, and take adventages of better concurrency.
- Suitable for both batch and streaming scenario.
- Lightweight. Comparing to Spark DataFrame API, it is much more lightweight to embed in your application.
- Stream-based design. Comparing to joinery, which keeps the whole data in memory. Poppy's streaming behaviour allows limited memory to process huge volume of data.
Please fork this project and pull request to me and any comment would be appreciated!