Skip to content

REST API for Apache Spark on K8S or YARN

License

Notifications You must be signed in to change notification settings

exacaster/lighter

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Lighter

Lighter is an opensource application for interacting with Apache Spark on Kubernetes or Apache Hadoop YARN. It is heavily inspired by Apache Livy and has some overlapping features.

Lighter supports:

  • Interactive Python Sessions through Sparkmagic kernel
  • Batch application submissions through the REST API

You can read a breaf description on how Lighter works here.

Using Lighter

Developing Lighter

Building

To build Docker image, containing application, you need to run

docker build -t lighter .

Spark versions

Lighter will always try to support the latest patch version for all officially recommended Apache Spark releases i.e.: 3.4.3 and 3.5.3.

Contributing

See Contribution guide

License

Lighter is MIT licensed.