New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: bytes read and bytes write mectirc for per lance scanner and writer #3570

Open

SaintBacchus opened this issue Mar 20, 2025 · 1 comment

Labels

Collaborator

SaintBacchus commented Mar 20, 2025

When we scan the lance dataset, it's better to provide the read metrics, especially for the io bytes and IOPS metric.

The Spark web UI has some task-level metrics, for example, the input bytes. But now Lance can't provide it.

The same metrics are required when we write a lance dataset.

SaintBacchus added the help wanted label

Contributor

westonpace commented Mar 21, 2025

#3575 adds these metrics on the read side. We might need to add some new methods for accessing these metrics. Currently they are logged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment