Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement URI scheme for command-line (and SDK) IO #157

Open
nicholasjng opened this issue Nov 11, 2024 · 0 comments
Open

Implement URI scheme for command-line (and SDK) IO #157

nicholasjng opened this issue Nov 11, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@nicholasjng
Copy link
Collaborator

We would like to support benchmark result IO to various places without having to prepare complex logic on the command line directly.

For example, an invocation like nnbench benchmarks.py -o gs://my-bucket/my-result1.json should write the results to json on disk, and then send the file to the given GCS bucket. A tested solution for this is the fsspec project.

The same works for an existing record upload, where nnbench benchmarks.py -o bq://my-proj/my-dataset/my-table could push the results into the given BQ table.

The interesting case is record fetching from databases in evaluation / comparison scenarios. For example, when comparing a local run (say 1.json) to latest results in BQ where a context value foo equals 'bar', it would be nnbench compare 1.json bq://my-proj/my-dataset/my-table, but WHERE context.foo='bar'.

This could be supported with a command-line parameter --query.

Urgent for building a comparison pipeline.

@nicholasjng nicholasjng added the enhancement New feature or request label Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant