[WIP] dandi search command #1126
Conversation
|
@djarecka - you may want to consider adopting the slurm sacct command syntax for specifying fields. also let's focus on asset based search as that's what was seen as most useful by the participants of the workshop. |
|
note: converted to draft since WiP etc |
|
|
see the field syntax of the sacct command. |
|
@satra - I did... |
yes |
… fields in select_fields, allowing for multiple values and ranges in filter fields; adding tests
Codecov ReportBase: 87.91% // Head: 65.90% // Decreases project coverage by
Additional details and impacted files@@ Coverage Diff @@
## master #1126 +/- ##
===========================================
- Coverage 87.91% 65.90% -22.02%
===========================================
Files 73 75 +2
Lines 8549 8845 +296
===========================================
- Hits 7516 5829 -1687
- Misses 1033 3016 +1983
Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
|
Looks good!
|
|
@bendichter - everything is possible, it will just require adding specific queries, see example for adding You have a good point that when searching for age you will have everything in ISO: Perhaps for every field there could be an optional wrapper function that convert it to something else. Or we can have |
|
@djarecka sorry I was unclear. I meant I wanted to filter by age. How would I filter for a mouse between the ages of 20 and 30 days? |
|
My filtering function does not work for ranges in ISO right now, I have to add it |
There was a problem hiding this comment.
Sorry @djarecka I have just got to look at it. I would love to see dandi search so would be great to finish it up, and also work it out in "collaboration" with all the other searches. When should we do it? see some comments spread out through codebase
| "--search_type", | ||
| help="Type of the search.", | ||
| type=click.Choice(["dandisets", "assets"]), | ||
| ) |
There was a problem hiding this comment.
FTR we have
download
--download [assets,dandiset.yaml,all]
Comma-separated list of elements to download
[default: all]
ls
--metadata [api|all|assets]
I wonder if we should unify somehow
| @click.option( | ||
| "-f", | ||
| "--filter_fields", | ||
| help="Field name for dandisets search", |
There was a problem hiding this comment.
help is the same as for -s , so what is the difference?
There was a problem hiding this comment.
also we have in ls
-F, --fields TEXT Comma-separated list of fields to display.
An empty value to trigger a list of
available fields to be printed out
-f, --format [auto|pyout|json|json_pp|json_lines|yaml]
Choose the format/frontend for output. If
'auto', 'pyout' will be used in case of
multiple files, and 'yaml' for a single
file.
I think here we should be uniform with that -- with dandi/dandi-archive#1659 in mind, we do adhere to MVC pattern here too, and clearly separate out the model from rendering and thus have similar -f|--format option which might be just an output of those "search result records". Well -- you already have --format below, so we are on the right track.
May be here we should even consider RF/reusing ls code -- we just need a list of records to render. they might come from different search engines etc.
| ) | ||
| @click.option( | ||
| "-t", | ||
| "--search_type", |
There was a problem hiding this comment.
here and everywhere below use --long-option, not --long_option as we do everywhere else (run git grep -A5 '@click.option')
| "-t", | ||
| "--search_type", | ||
| help="Type of the search.", | ||
| type=click.Choice(["dandisets", "assets"]), |
There was a problem hiding this comment.
familiarize with #1357 and RF similarly here
| else: | ||
| raise NotImplementedError | ||
|
|
||
| endpoint = "https://search.dandiarchive.org:5820/dandisets_new/query" |
There was a problem hiding this comment.
this would be just 1 type of search. We already have a few more (even if not exposed via API): https://llmsearch.dandiarchive.org/, https://dandiarchive.org/search etc.
I think we should generalize this dandi search [options] [query] interface into dandi search [common-options] HOW [how-specific-options-if-any] [query] or dandi search [options] [--how HOW] [query] interface, so we could expand etc.
edited
I've added draft of the
dandi searchcommand. I've added some test, but below you will show some example of usage:dandisets:--select_fields(or-s):select_fieldscan be done in two ways:--filter_fields(or-f):--filter_fields(or-f):assets:@satra @bendichter - let me know what don you think about these examples