You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to create a HSDS to serve data for my application. When I make multiple parallel get's to the service from my application, I frequently get 503's. I can mitigate this problem by implementing a generous retry policy in my application, but my question is: Is this expected? Are there any ways to improve availability \ reliability of the HSDS? I see in the data owner's documentation mention of the same error I am seeing, but it isn't clear to me exactly what the problem is or how my HSDS deployment could be more robust to parallel requests.
To reproduce:
Start a HSDS container using latest image: docker-compose.yml
Then make parallel requests for data using a client, in this case h5pyd:
from concurrent.futures import ThreadPoolExecutor
import h5pyd
keys = ['ghi', 'dni', 'dhi', 'air_temperature', 'surface_albedo']
def data_worker():
with h5pyd.File(
'/nrel/nsrdb/current/nsrdb_tmy-2022.h5',
endpoint='http://localhost:5101',
bucket='nrel-pds-hsds',
retries=1,
) as h5_file:
for key in keys:
h5_file.get(key)[:, 100:200]
def main():
with ThreadPoolExecutor(5) as executor:
futures = [executor.submit(data_worker) for _ in range(10)]
for future in futures:
future.result()
if __name__ == '__main__':
main()
This will (usually) fail with:
ERROR:root:got <class 'requests.exceptions.RetryError'> exception: HTTPConnectionPool(host='localhost', port=5101): Max retries exceeded with url: /datasets/d-1d865ecb-1c251681-7021-33f71a-2c6661/value?nonstrict=1&select=%5B0%3A8760%3A1%2C100%3A200%3A1%5D&domain=%2Fnrel%2Fnsrdb%2Fcurrent%2Fnsrdb_tmy-2022.h5&bucket=nrel-pds-hsds (Caused by ResponseError('too many 503 error responses'))
Traceback (most recent call last):
File ".venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 894, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 884, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5101): Max retries exceeded with url: /datasets/d-1d865ecb-1c251681-7021-33f71a-2c6661/value?nonstrict=1&select=%5B0%3A8760%3A1%2C100%3A200%3A1%5D&domain=%2Fnrel%2Fnsrdb%2Fcurrent%2Fnsrdb_tmy-2022.h5&bucket=nrel-pds-hsds (Caused by ResponseError('too many 503 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".venv/lib/python3.11/site-packages/h5pyd/_hl/httpconn.py", line 474, in GET
rsp = s.get(
^^^^^^
File ".venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/requests/adapters.py", line 510, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPConnectionPool(host='localhost', port=5101): Max retries exceeded with url: /datasets/d-1d865ecb-1c251681-7021-33f71a-2c6661/value?nonstrict=1&select=%5B0%3A8760%3A1%2C100%3A200%3A1%5D&domain=%2Fnrel%2Fnsrdb%2Fcurrent%2Fnsrdb_tmy-2022.h5&bucket=nrel-pds-hsds (Caused by ResponseError('too many 503 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".venv/lib/python3.11/site-packages/h5pyd/_hl/dataset.py", line 1160, in __getitem__
rsp = self.GET(req, params=params, format="binary")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/h5pyd/_hl/base.py", line 962, in GET
rsp = self.id._http_conn.GET(req, params=params, headers=headers, format=format, use_cache=use_cache)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/h5pyd/_hl/httpconn.py", line 490, in GET
raise IOError("Unexpected exception")
OSError: Unexpected exception
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/bgarrett-mac/Documents/workspace/hsds/call.py", line 28, in <module>
main()
File "/Users/bgarrett-mac/Documents/workspace/hsds/call.py", line 24, in main
future.result()
File "/Users/bgarrett-mac/.asdf/installs/python/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/Users/bgarrett-mac/.asdf/installs/python/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/Users/bgarrett-mac/.asdf/installs/python/3.11.4/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/bgarrett-mac/Documents/workspace/hsds/call.py", line 17, in data_worker
h5_file.get(key)[:, 100:200]
~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File ".venv/lib/python3.11/site-packages/h5pyd/_hl/dataset.py", line 1169, in __getitem__
raise IOError(f"Error retrieving data: {ioe.errno}")
OSError: Error retrieving data: None
This indicates the client is receiving a 503 from the HSDS. The log file hs.log does not indicate any sort or error occurred in the HSDS container. Adding retries usually allows me to get the data I need from the HSDS, but I am trying to understand why the service is unavailable and if there is a way to configure the HSDS to be more robust to parallel requests.
The text was updated successfully, but these errors were encountered:
The 503 responses aren't failures as such - it's just the server telling you to lower the rate of requests and retry the given request after a bit of delay.
Backgroiund: Each HSDS container keeps track of the number or inflight requests. If it exceeds a certain limit (100 by default), it will fail the next requests with a 503 response. The idea is not to overtax the server or having issues with the container running out of memory.
The actual effective limit depends greatly on the type of load coming in (basically how memory or cpu intensive the requests are on average). You can change the default by creating a file: hsds/admin/config/override.yml with the following line: "max_tax_count: 999" where 999 is whatever you'd like to max_task_count to be. Restart the server to have it take effect.
If you see the containers regularly hitting 100% CPU, or restarting because of out of memory errors, you've probably set max_task_count to high and you'll want to scale it back a bit.
I am trying to create a HSDS to serve data for my application. When I make multiple parallel
get
's to the service from my application, I frequently get503
's. I can mitigate this problem by implementing a generous retry policy in my application, but my question is: Is this expected? Are there any ways to improve availability \ reliability of the HSDS? I see in the data owner's documentation mention of the same error I am seeing, but it isn't clear to me exactly what the problem is or how my HSDS deployment could be more robust to parallel requests.To reproduce:
Start a HSDS container using latest image:
docker-compose.yml
Then make parallel requests for data using a client, in this case
h5pyd
:This will (usually) fail with:
This indicates the client is receiving a
503
from the HSDS. The log filehs.log
does not indicate any sort or error occurred in the HSDS container. Adding retries usually allows me to get the data I need from the HSDS, but I am trying to understand why the service is unavailable and if there is a way to configure the HSDS to be more robust to parallel requests.The text was updated successfully, but these errors were encountered: