[24.0] Close install model session when request ends

[mvdbeek]

This is probably the more general fix for
#18627.

@kysrpex could you test this without the explicit session management you suggested in #18627 ?

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[bgruening]

from typing import Optional

Is missing in lib/galaxy/webapps/galaxy/buildapp.py but github does not let me add suggestions into the right spot.

[kysrpex]

Thanks @mvdbeek. I am testing this with the code snippet from #18627 and so far I am not getting any HTTP 500. I'll let the tool installation CI run overnight just to be sure though.

Tomorrow I share the results with you and if it worked we can merge. Alternatively, you may also check it yourself here (the link won't work until the the CI ends). Once the job is finished, search for "GET: error 500: b'Internal Server Error'" within the file.

[kysrpex]

GET: error 500: b'Internal Server Error'

It showed up many times on the CI logs :(

[mvdbeek]

What did you test where and which commit ?

[kysrpex]

What did you test where and which commit ?

I tested a126f10 (added from typing import Optional manually) on the useGalaxy.eu server. This is the tool installation CI log showing the HTTP 500 errors when running a126f10.

[mvdbeek]

Do you have the logs somewhere ? The tool install log doesn't help. I'm not seeing anything suspicious in sentry.

[mvdbeek]

I think I see the problem though, if we have FastAPI routes that don't depend on trans or app we're never applying the session scope logic.

[kysrpex]

When the issue pops up it does it in this Sentry issue. The events from the last 24 hours

correspond to the last tool install run. Most likely many PendingRollbackError are linked to this other error.

I cannot get you the gunicorn logs from yesterday because the logs have already rotated (too many entries per second). To get them I have to set up some machinery to dump them somewhere else and trigger the failure again (which I may do). I assume you'd want to see the full wall of text during the whole duration of the tool installation CI run right?

[mvdbeek]

It's ok, I think I see the problem, see #18629 (comment)

[kysrpex]

I think I see the problem though, if we have FastAPI routes that don't depend on trans or app we're never applying the session scope logic.

Yes, in the stacktrace from the Sentry issue I linked you can see the request goes through FastAPI (this line).

[kysrpex]

@mvdbeek Trying d20b333 out (link to CI log). As usual I let you know once it finishes.

[kysrpex]

It works!! 🚀 🚀

[bgruening]

Are the failing startup tests relevant?

[mvdbeek]

No, it's 3.12 which isn't compatible with the pydantic version we install, all 24.0 PRs have that problem.

[galaxyproject-sentryintegration]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly /api/entry_points View Issue

_{Did you find this useful? React with a 👍 or 👎}