an anywidget for data that talks like a duck
quak is a scalable data profiler for quickly scanning large tables, capturing interactions as executable SQL queries.
- interactive 🖱️ mouse over column summaries, cross-filter, sort, and slice rows.
- fast ⚡ built with Mosaic; views are expressed as SQL queries lazily executed by DuckDB.
- flexible 🔄 supports many data types and formats via Apache Arrow, the dataframe interchange protocol, and the Arrow PyCapsule Interface.
- reproducible 📓 a UI for building complex SQL queries; materialize views in the kernel for further analysis.
Warning
quak is a prototype exploring a high-performance data profiler based on anywidget. It is not production-ready. Expect bugs. Open-sourced for SciPy 2024.
pip install quak
The easiest way to get started with quak is using the IPython cell magic.
%load_ext quak
import polars as pl
df = pl.read_parquet("https://github.com/uwdata/mosaic/raw/main/data/athletes.parquet")
df
![olympic athletes table](https://private-user-images.githubusercontent.com/24403730/352296312-83858282-8876-4b12-aeea-44eb82d3bed3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNDcxOTAsIm5iZiI6MTcyNjE0Njg5MCwicGF0aCI6Ii8yNDQwMzczMC8zNTIyOTYzMTItODM4NTgyODItODg3Ni00YjEyLWFlZWEtNDRlYjgyZDNiZWQzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDEzMTQ1MFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThkNWVhMTlmMTA1NTAwMzI3ZWZkNzZiMjJhNTczNDMwNzA5OWI3YjUwZWRkNzYzNzJkZmQ0ZGQ2NDVlMWVlODUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.IF-4JVlgc_iiM4Sw4VKTZp8H24-Kqk_BhydaPzoQ08U)
quak hooks into Jupyter's display mechanism to automatically render any
dataframe-like object (implementing the Python dataframe interchange
protocol or Arrow PyCapsule Interface)
using quak.Widget
instead of the default display.
Alternatively, you can use quak.Widget
directly:
import polars as pl
import quak
df = pl.read_parquet("https://github.com/uwdata/mosaic/raw/main/data/athletes.parquet")
widget = quak.Widget(df)
widget
quak captures all user interactions as queries.
At any point, table state can be accessed as SQL,
widget.sql # SELECT * FROM df WHERE ...
which for convenience can be executed in the kernel to materialize the view for further analysis:
widget.data() # returns duckdb.DuckDBPyRelation object
By representing UI state as SQL, quak makes it easy to generate complex queries via interactions that would be challenging to write manually, while keeping them reproducible.
quak can also be used in marimo notebooks, which provide out-of-the-box support for anywidget:
import marimo as mo
import polars as pl
import quak
df = pl.read_parquet("https://github.com/uwdata/mosaic/raw/main/data/athletes.parquet")
widget = mo.ui.anywidget(quak.Widget(df))
widget
Contributors welcome! Check the Contributors Guide to get started. Note: I'm wrapping up my PhD, so I might be slow to respond. Please open an issue before contributing a new feature.
quak pieces together many important ideas from the web and Python data science ecosystems. It serves as an example of what you can achieve by embracing these platforms for their strengths.
- Observable's data table: Inspiration for the UI design and user interactions.
- Mosaic: The foundation for linking databases and interactive table views.
- Apache Arrow: Support for various data types and efficient data interchange between JS/Python.
- DuckDB: An amazingly engineered piece of software that makes SQL go vroom.