You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We would like to support benchmark result IO to various places without having to prepare complex logic on the command line directly.
For example, an invocation like nnbench benchmarks.py -o gs://my-bucket/my-result1.json should write the results to json on disk, and then send the file to the given GCS bucket. A tested solution for this is the fsspec project.
The same works for an existing record upload, where nnbench benchmarks.py -o bq://my-proj/my-dataset/my-table could push the results into the given BQ table.
The interesting case is record fetching from databases in evaluation / comparison scenarios. For example, when comparing a local run (say 1.json) to latest results in BQ where a context value foo equals 'bar', it would be nnbench compare 1.json bq://my-proj/my-dataset/my-table, but WHERE context.foo='bar'.
This could be supported with a command-line parameter --query.
Urgent for building a comparison pipeline.
The text was updated successfully, but these errors were encountered:
We would like to support benchmark result IO to various places without having to prepare complex logic on the command line directly.
For example, an invocation like
nnbench benchmarks.py -o gs://my-bucket/my-result1.json
should write the results to json on disk, and then send the file to the given GCS bucket. A tested solution for this is the fsspec project.The same works for an existing record upload, where
nnbench benchmarks.py -o bq://my-proj/my-dataset/my-table
could push the results into the given BQ table.The interesting case is record fetching from databases in evaluation / comparison scenarios. For example, when comparing a local run (say 1.json) to latest results in BQ where a context value
foo
equals'bar'
, it would bennbench compare 1.json bq://my-proj/my-dataset/my-table
, butWHERE context.foo='bar'
.This could be supported with a command-line parameter
--query
.Urgent for building a comparison pipeline.
The text was updated successfully, but these errors were encountered: