Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ServiceX fails to report science image missing under certain circumstances #540

Open
gordonwatts opened this issue Feb 11, 2023 · 4 comments · May be fixed by #555
Open

ServiceX fails to report science image missing under certain circumstances #540

gordonwatts opened this issue Feb 11, 2023 · 4 comments · May be fixed by #555

Comments

@gordonwatts
Copy link
Collaborator

Describe the bug

This crash when running:

servicex.utils.ServiceXException: (ServiceXException(...), 'ServiceX rejected the transformation request: (400){"message": "Failed to submit transform request: Expecting value: line 1 column 1 (char 0)"}\n')

Can with some deeper debugging be tracked to:

Traceback (most recent call last):\n  File \"/usr/local/lib/python3.10/site-packages/requests/models.py\", line 971, in json\n    return complexjson.loads(self.text, **kwargs)\n  File \"/usr/local/lib/python3.10/json/__init__.py\", line 346, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/local/lib/python3.10/json/decoder.py\", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File \"/usr/local/lib/python3.10/json/decoder.py\", line 355, in raw_decode\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/servicex/servicex/resources/transformation/submit.py\", line 153, in post\n    if not self.docker_repo_adapter.check_image_exists(request_rec.image):\n  File \"/home/servicex/servicex/docker_repo_adapter.py\", line 56, in check_image_exists\n    f\"last updated {r.json()['last_updated']}\")\n  File \"/usr/local/lib/python3.10/site-packages/requests/models.py\", line 975, in json\n    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)\nrequests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n",

If you look at that bug you'll see it is in the middle of the check of the docker science image existence up on the hub.

To Reproduce
Steps to reproduce the behavior:

  1. Specify a docker science image with two namespaces: sslhep/sslhep/servicex_func_adl_xaod_transformer:v1.1.4
  2. Run the transform.

Expected behavior
A clear message declaring the image could not be found. The error should include the image that it was trying to look up so the user can try to track this back to something they did (or how SX was misconfigured).
@gordonwatts
Copy link
Collaborator Author

I suspect this is a good first issue.

@draghuram
Copy link
Contributor

Hi, I started exploring ServiceX only few days back and so far, I am able to deploy it locally and am trying to run a transformation on a ROOT file. This issue indeed looks like a good one for beginners but can you please give more info on how to reproduce it? Also, is there a document describing all the REST APIs? If so, I can test the fix by directly invoking the API (using "curl" for example).

I looked at docker_repo_adapter.py and here is the relevant code:

(repo, image, tag) = search_result.groups()

query = f'{self.registry_endpoint}/v2/repositories/{repo}/{image}/tags/{tag}'
r = requests.get(query)
if r.status_code == 404:
    return False

current_app.logger.info(f"Requested Image: {tagged_image} exists, "
                        f"last updated {r.json()['last_updated']}")

As can be seen, there is already a check for 404 error. In this particular case, I suspect that the return status is something else (may even be 500) and the response contains non-JSON (most probably some HTML error message). So r.json() failed with JSON decode exception.

Finally, I see the file docker_repo_adapter.py in this repo as well as in "ServiceX_App" repo. I am guessing change should be made in this repo?

draghuram added a commit to draghuram/ServiceX that referenced this issue Mar 4, 2023
@draghuram
Check and return errors while validating science image.
66d4d13 
Fixes ssl-hep#540.
@draghuram draghuram linked a pull request Mar 4, 2023 that will close this issue
Draft
@robrwg
Copy link
Member

robrwg commented Oct 11, 2023

@gordonwatts is this issue still relevant?

@gordonwatts
Copy link
Collaborator Author

I have no idea - I've not tried to do a request with a different docker image recently to repro this error. @BenGalewsky - is this even possible with the new way of specifying code generators (since this is a transformer image)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Check and return errors while validating science image. draghuram/ServiceX
3 participants
@gordonwatts @robrwg @draghuram