ndscheduler is a flexible python library for building your own cron-like system to schedule jobs, which is to run a tornado process to serve REST APIs and a web ui. It's like LLVM that provides modular and reusable components for building a compiler.
Check out our blog post - We Don’t Run Cron Jobs at Nextdoor
ndscheduler currently supports Python 2 & 3 on Mac OS X / Linux.
- Key Abstractions
- Try it NOW
- How to build Your own cron-replacement
- Contribute code to ndscheduler
- REST APIs
- Web UI
- Core: a bunch of resuable components
- Datastore: manages database connections and makes queries; could support Postgres, MySQL, and sqlite.
- Job: represents a schedule job and decides how to run a paricular job.
- Execution: represents an instance of job execution.
- AuditLog: logs when and who runs what job.
- ScheduleManager: access Datastore to manage jobs, i.e., schedule/modify/delete/pause/resume a job.
- Datastore: manages database connections and makes queries; could support Postgres, MySQL, and sqlite.
- Server: a tornado server that runs ScheduleManager and provides REST APIs and serves UI.
- Web UI: a single page HTML app; this is a default implementation.
From source code:
git clone https://github.com/Nextdoor/ndscheduler.git
cd ndscheduler
make simple
Or use docker:
docker run -it -p 8888:8888 wenbinf/ndscheduler
Open your browser and go to localhost:8888.
Demo
(Click for fullscreen play)
Using pip (from GitHub repo)
#
# Put this in requirements.txt, then run
# pip install -r requirements.txt
#
# If you want the latest build
git+https://github.com/Nextdoor/ndscheduler.git#egg=ndscheduler
# Or put this if you want a specific commit
git+https://github.com/Nextdoor/ndscheduler.git@5843322ebb440d324ca5a66ba55fea1fd00dabe8
# Or put this if you want a specific tag version
git+https://github.com/Nextdoor/[email protected]#egg=ndscheduler
#
# Run from command line
#
pip install -e git+https://github.com/Nextdoor/ndscheduler.git#egg=ndscheduler
(We'll upload the package to PyPI soon.)
You have to implement three things for your scheduler, i.e., Settings, Server, and Jobs.
Settings
In your implementation, you need to provide a settings file to override default settings (e.g., settings in simple_scheduler). You need to specify the python import path in the environment variable NDSCHEDULER_SETTINGS_MODULE before running the server.
All available settings can be found in default_settings.py file.
Server
You need to have a server file to import and run ndscheduler.server.server.SchedulerServer.
Jobs
Each job should be a standalone class that is a subclass of ndscheduler.job.JobBase and put the main logic of the job in run() function.
After you set up Settings, Server and Jobs, you can run the whole thing like this:
NDSCHEDULER_SETTINGS_MODULE=simple_scheduler.settings \
PYTHONPATH=.:$(PYTHONPATH) \
python simple_scheduler/scheduler.py
See code in the simple_scheduler/ directory for inspiration :)
Run it
make simple
Access the web ui via localhost:8888
The reference implementation also comes with several sample jobs.
- AwesomeJob: it just prints out 2 arguments you pass in.
- SlackJob: it sends a slack message periodically, for example, team standup reminder.
- ShellJob: it runs an executable command, for example, run curl to crawl web pages.
- CurlJob: it's like running curl periodically.
And it's dockerized.
Install dependencies
# Each time we introduce a new dependency in setup.py, you have to run this
make install
Run unit tests
make test
Clean everything and start from scratch
make clean
Finally, send pull request. Please make sure the CI passes for your PR.
Please see README.md in ndscheduler/server/handlers.
We provide a default implementation of web ui. You can replace the default web ui by overwriting these settings
STATIC_DIR_PATH = :static asset directory paths:
TEMPLATE_DIR_PATH = :template directory path:
APP_INDEX_PAGE = :the file name of the single page app's html:
List of jobs
List of executions
Audit Logs
Modify a job
