Error using client.listdir() on a canfar directory with 100s of files.

[ErikOsinga]

I'm running into a vague error using the commands

from vos import Client
client = Client()
canfar_sourcelists = client.listdir("vos://cadc.nrc.ca~arc/projects/CIRADA/polarimetry/ASKAP/PartialTiles/sourcelists/",force=True)

which returns

OSError: [Errno 400] OptionNotSupported: batch container node listing: limit ignored, resume not implemented

I'm pretty sure it's not a certificate or permission error, since I've previously been able to run this command when the directory was less populated, and I'm able to listdir() the directory(ies) above it without problems.

This makes me think it's likely some kind of processing limit, perhaps because of the amount of files in the folder? Though not overly large, the folder I'd like to listdir contains 572 files.

Full code error:

In [25]:         canfar_sourcelists = client.listdir("vos://cadc.nrc.ca~arc/projects/CIRADA/polarimetry/ASKAP/Parti
    ...: alTiles/",force=True)

In [26]:         canfar_sourcelists = client.listdir("vos://cadc.nrc.ca~arc/projects/CIRADA/polarimetry/ASKAP/Parti
    ...: alTiles/sourcelists/",force=True)
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File ~/miniconda3/envs/py311/lib/python3.11/site-packages/cadcutils/net/ws.py:1107, in RetrySession.check_status(self, response, retry)
   1106 try:
-> 1107     response.raise_for_status()
   1108 except requests.HTTPError as e:

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/requests/models.py:1024, in Response.raise_for_status(self)
   1023 if http_error_msg:
-> 1024     raise HTTPError(http_error_msg, response=self)

HTTPError: 400 Client Error: Bad Request for url: https://ws-uv.canfar.net/arc/nodes/projects/CIRADA/polarimetry/ASKAP/PartialTiles/sourcelists?uri=vos%3A%2F%2Fcadc.nrc.ca~arc%2Fprojects%2FCIRADA%2Fpolarimetry%2FASKAP%2FPartialTiles%2Fsourcelists%2Fselavy-image.i.EMU_1748-64.SB54926.cont.taylor.0.restored.conv.components.xml

During handling of the above exception, another exception occurred:

BadRequestException                       Traceback (most recent call last)
File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:1179, in VOFile.read(self, size, return_response)
   1178         self.connector.session.retry = False
-> 1179     self.resp = self.connector.session.send(self.request,
   1180                                             stream=True)
   1181 except exceptions.HttpException as http_exception:

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/cadcutils/net/ws.py:1051, in RetrySession.send(self, request, **kwargs)
   1049 response = super(RetrySession, self).send(request,
   1050                                           **kwargs)
-> 1051 self.check_status(response)
   1052 return response

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/cadcutils/net/ws.py:1116, in RetrySession.check_status(self, response, retry)
   1115 elif e.response.status_code == requests.codes.bad_request:
-> 1116     raise exceptions.BadRequestException(orig_exception=e)
   1117 elif e.response.status_code == requests.codes.precondition_failed:

BadRequestException: OptionNotSupported: batch container node listing: limit ignored, resume not implemented


During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
Cell In[26], line 1
----> 1 canfar_sourcelists = client.listdir("vos://cadc.nrc.ca~arc/projects/CIRADA/polarimetry/ASKAP/PartialTiles/sourcelists/",force=True)

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:2736, in Client.listdir(self, uri, force)
   2734 names = []
   2735 logger.debug(str(uri))
-> 2736 node = self.get_node(uri, limit=None, force=force)
   2737 while node.type == "vos:LinkNode":
   2738     uri = node.target

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:2074, in Client.get_node(self, uri, limit, force)
   2070 while next_uri != node.node_list[-1].uri:
   2071     next_uri = node.node_list[-1].uri
   2072     xml_file = StringIO(
   2073         self.open(uri, os.O_RDONLY, next_uri=next_uri,
-> 2074                   limit=limit).read().decode('UTF-8'))
   2075     xml_file.seek(0)
   2076     next_page = Node(ElementTree.parse(xml_file).getroot())

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:1192, in VOFile.read(self, size, return_response)
   1189 # restore the original retry flag of the session
   1190 self.connector.session.retry = orig_retry_flag
-> 1192 self.checkstatus()
   1194 if isinstance(http_exception,
   1195               exceptions.UnauthorizedException) or \
   1196         isinstance(http_exception,
   1197                    exceptions.BadRequestException) or \
   1198         isinstance(http_exception,
   1199                    exceptions.ForbiddenException):
   1200     raise

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:1076, in VOFile.checkstatus(self, codes)
   1073     if self.resp.status_code == 400 and \
   1074             "sorting options not supported" in msg:
   1075         exception = Exception("service does not support sorting")
-> 1076     raise exception
   1078 # Get the file size. We use this HEADER-CONTENT-LENGTH as a
   1079 # fallback to work around a server-side Java bug that limits
   1080 # 'Content-Length' to a signed 32-bit integer (~2 gig files)
   1081 try:

OSError: [Errno 400] OptionNotSupported: batch container node listing: limit ignored, resume not implemented

[andamian]

Hmm. Not sure what is going on. Is it reproducible? Sometimes these errors could be caused by intermittent networking/storage issues. I've tried to run it but I don't have access to it. Also, are you able to list other directories in the path?
572 items is pretty small so unlikely to be the cause. Also, is it always chocking on the same file (selavy-image.i.EMU_1748-64.SB54926.cont.taylor.0.restored.conv.components.xml)?

[ErikOsinga]

Can confirm that the error is reproducible, since it's still throwing the same error for me. Indeed every time on the same file.

Other directories in the path work fine. Both higher directories and a subdirectory I just created with only 1 file.

However, I think it is related to the number of files, because when I executed the following small test:

import os

for i in range(501):
    os.system(f"touch test{i}.txt")

and then try to listdir that directory in CANFAR, it breaks consistently on file test274.txt, which is identical to all other test files. So I think it's probably reproducible if you try to do it in a directory you have access to where you create 100s of files. Full error trace below for the directory with five hundred empty test{i}.txt files

In [42]: from vos import Client
    ...: client = Client()
    ...: canfar_sourcelists = client.listdir("vos://cadc.nrc.ca~arc/projects/CIRADA/polarimetry/ASKAP/PartialTil
    ...: es/sourcelists/test/",force=True)
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File ~/miniconda3/envs/py311/lib/python3.11/site-packages/cadcutils/net/ws.py:1107, in RetrySession.check_status(self, response, retry)
   1106 try:
-> 1107     response.raise_for_status()
   1108 except requests.HTTPError as e:

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/requests/models.py:1024, in Response.raise_for_status(self)
   1023 if http_error_msg:
-> 1024     raise HTTPError(http_error_msg, response=self)

HTTPError: 400 Client Error: Bad Request for url: https://ws-uv.canfar.net/arc/nodes/projects/CIRADA/polarimetry/ASKAP/PartialTiles/sourcelists/test?uri=vos%3A%2F%2Fcadc.nrc.ca~arc%2Fprojects%2FCIRADA%2Fpolarimetry%2FASKAP%2FPartialTiles%2Fsourcelists%2Ftest%2Ftest274.txt

During handling of the above exception, another exception occurred:

BadRequestException                       Traceback (most recent call last)
File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:1179, in VOFile.read(self, size, return_response)
   1178         self.connector.session.retry = False
-> 1179     self.resp = self.connector.session.send(self.request,
   1180                                             stream=True)
   1181 except exceptions.HttpException as http_exception:

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/cadcutils/net/ws.py:1051, in RetrySession.send(self, request, **kwargs)
   1049 response = super(RetrySession, self).send(request,
   1050                                           **kwargs)
-> 1051 self.check_status(response)
   1052 return response

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/cadcutils/net/ws.py:1116, in RetrySession.check_status(self, response, retry)
   1115 elif e.response.status_code == requests.codes.bad_request:
-> 1116     raise exceptions.BadRequestException(orig_exception=e)
   1117 elif e.response.status_code == requests.codes.precondition_failed:

BadRequestException: OptionNotSupported: batch container node listing: limit ignored, resume not implemented


During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
Cell In[42], line 3
      1 from vos import Client
      2 client = Client()
----> 3 canfar_sourcelists = client.listdir("vos://cadc.nrc.ca~arc/projects/CIRADA/polarimetry/ASKAP/PartialTiles/sourcelists/test/",force=True)

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:2736, in Client.listdir(self, uri, force)
   2734 names = []
   2735 logger.debug(str(uri))
-> 2736 node = self.get_node(uri, limit=None, force=force)
   2737 while node.type == "vos:LinkNode":
   2738     uri = node.target

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:2074, in Client.get_node(self, uri, limit, force)
   2070 while next_uri != node.node_list[-1].uri:
   2071     next_uri = node.node_list[-1].uri
   2072     xml_file = StringIO(
   2073         self.open(uri, os.O_RDONLY, next_uri=next_uri,
-> 2074                   limit=limit).read().decode('UTF-8'))
   2075     xml_file.seek(0)
   2076     next_page = Node(ElementTree.parse(xml_file).getroot())

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:1192, in VOFile.read(self, size, return_response)
   1189 # restore the original retry flag of the session
   1190 self.connector.session.retry = orig_retry_flag
-> 1192 self.checkstatus()
   1194 if isinstance(http_exception,
   1195               exceptions.UnauthorizedException) or \
   1196         isinstance(http_exception,
   1197                    exceptions.BadRequestException) or \
   1198         isinstance(http_exception,
   1199                    exceptions.ForbiddenException):
   1200     raise

File ~/miniconda3/envs/py311/lib/python3.11/site-packages/vos/vos.py:1076, in VOFile.checkstatus(self, codes)
   1073     if self.resp.status_code == 400 and \
   1074             "sorting options not supported" in msg:
   1075         exception = Exception("service does not support sorting")
-> 1076     raise exception
   1078 # Get the file size. We use this HEADER-CONTENT-LENGTH as a
   1079 # fallback to work around a server-side Java bug that limits
   1080 # 'Content-Length' to a signed 32-bit integer (~2 gig files)
   1081 try:

OSError: [Errno 400] OptionNotSupported: batch container node listing: limit ignored, resume not implemented

[ErikOsinga]

Steps to reproduce:

1. On a CANFAR session, go to any directory you have access to /arc/directory/you/have/access/to/
2. Execute the following python script

import os

for i in range(501):
    os.system(f"touch test{i}.txt")

3. Try to listdir

from vos import Client
client = Client()
canfar_sourcelists = client.listdir("vos://cadc.nrc.ca~arc/directory/you/have/access/to",force=True) 

Throws error OSError: [Errno 400] OptionNotSupported: batch container node listing: limit ignored, resume not implemented on file test274.txt

[andamian]

Thanks for the details - I'm now able to reproduce it. We are working on a patch, but until then if you need to list the content of that directory please use the vls command that installs with the package.

[ErikOsinga]

Thanks, the vls command does work for both directories.

I know the client.listdir() function caches results, which I don't want to use (hence force=True). Do you know if vls does a similar thing and if so, can I force it to not use cache?

[andamian]

@ErikOsinga - the latest release (3.6.2) should fix this. Please let me know if you notice any unexpected behaviour from that method.