Xskipper is an Extensible Data Skipping Framework for Apache Spark.
To get started, see the Quick Start Guide .
See Xskipper site for more info.
To build a project using the Xskipper binaries from the Maven Central Repository, use the following Maven coordinates:
Include Xskipper in a Maven project by adding it as a dependency in the project's POM file. Xskipper should be compiled with Scala 2.12.
<dependency>
<groupId>io.xskipper</groupId>
<artifactId>xskipper-core_2.12</artifactId>
<version>1.3.0</version>
</dependency>Include Xskipper in an SBT project by adding the following line to its build.sbt file:
libraryDependencies += "io.xskipper" %% "xskipper-core" % "1.3.0"Xskipper is compiled using SBT.
To compile, run
build/sbt compile
To generate artifacts, run
build/sbt package
To execute tests, run
build/sbt test
Refer to SBT docs for more commands.
Xskipper tracks issues in GitHub and prefers to receive contributions as pull requests.
Xskipper is compatible with Spark according to the following table:
| Xskipper version | Spark Version |
|---|---|
| 1.4.x | 3.3.x |
| 1.3.x | 3.2.x |
| 1.2.x | 3.0.x |
| 1.1.x | 2.4.x |
| 1.0.x | 2.3.x |
- IEEE Big Data 2020 paper - Extensible Data Skipping (arxiv version)
Apache License 2.0, see LICENSE.
This software has been developed under the BigDataStack project, as part of the holistic solution for big data applications and operations. BigDataStack has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 779747.