--limit - working at intended?

[jlaura]

I am executing the following search using the CLI:

sat-search search --limit 1 --url=https://jat52qc8c0.execute-api.us-west-2.amazonaws.com/dev/ -c usgs_controlled_mosaics_voy1_voy2_galileo --print-md

My expectation is that this search would result in a single (limited) response. I see a single GeoJSON Feauture (STAC item) being returned, followed by a warning and then a print of 111 items:

<Response [200]>
{'type': 'FeatureCollection', 'stac_version': '1.0.0', 'stac_extensions': [], 'context': {'page': 1, 'limit': 1, 'matched': 111, 'returned': 1}, 'numberMatched': 111, 'numberReturned': 1, 'features': [{'type': 'Feature', 'stac_version': '1.0.0', 'id': '25ESMOTTER01_GalileoSSI_Equi-cog', 'properties': {'title': 'Absolutely Controlled Image Mosaic; Europa; 25ESMOTTER01_GalileoSSI_Equi', 'description': 'The Solid State Image
...
WARNING:satsearch.search:There are more items found (111) than the limit (1) provided.
<Response [200]>
<Response [200]>
111 items found
Items (111):
date                      id
2021-10-20                14ESVLOFOT01_GalileoSSI_Equi-cog
2021-10-20                17ESLIBLIN01_GalileoSSI_Equi-cog
2021-10-20                12ESWEDGE_02_GalileoSSI_Equi-cog
2021-10-20                25ESMOTTER01_GalileoSSI_Equi-cog

I am hitting a stat-server API. I would expect that --limit would return a single item, but that is definitely not the case. Should I be calling with a different set or arguments or is this a bug in either sat-search or stac server?

I am also not sure if and how this might be related to #115.

[matthewhanson]

@jlaura Not sure if we discussed this previously, but have you tried to use pystac-client, it's more or less a drop-in replacement for sat-search but is up to date.
https://pystac-client.readthedocs.io/en/docs/

Also, what are you using as a STAC server here, is it stac-server, or are you using stac-fastapi ?

[matthewhanson]

Oh, @jlaura I realized what the problem is. --limit is actually not the max number of items, but rather is the limit per page. sat-search will still paginate through and get all the items.

pystac-client has a --max-items option which specifies the max # of items to return, along with --limit which is items per page. I'd definitely recommend switching to using pystac-client, but let me know if there's a problem switching to it.

[jlaura]

👍 This is stac-server backed. I'll give the pystac client a try.

The pagination makes sense. I had misunderstood how limit was working. This also makes sense why I was seeing HTTP 413 errors when a ton of geoms were getting returned.

[jlaura]

@matthewhanson pystac-client was a great swap in. It also got me to update out service. The following works beautifully now:

stac-client search https://jat52qc8c0.execute-api.us-west-2.amazonaws.com/dev/ --matched

Also, I really liked the download function on sat-search, so I wrote a little bash helper (using jq) if that is ever of interest to anyone!

#!/bin/bash
infile=$1

for row in $(cat $1 | jq -r '.features[] | @base64'); do
    _jq() {
      echo ${row} | base64 --decode | jq -r ${1}
    }

    collection=$(_jq '.collection')
    dir=$(_jq '.id')

    if [ ! -d "${collection}/${dir}" ]
    then
      mkdir -p "${collection}/${dir}"
    fi

    for href in $(_jq '.assets[].href'); do
      wget $href -P "${collection}/${dir}"
    done

done