fix: use root for python installs

[aaronsteers]

Attempts to resolve: https://airbytehq.sentry.io/issues/6129237224/?project=4505596642983936

What

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 12, 2024 5:02pm

[natikgadzhi]

Hold up. I understand this is the consequence of having non-root connectors, with @alafanechere?

I would love to make the path writable for user airbyte in the base image instead of falling back to using root.

[aaronsteers]

@natikgadzhi - My understanding is that we don't want the user account used at runtime to have permissions to modify anything in the container - and I think that includes installing packages and modifying source code. For this reason, I think it is correct for root to be required for installations, so a toggle to root and back seems correct when we need to modify the image by installing packages or injecting other code that should not be able to be modified at runtime. Definitely happy to continue this conversation in other threads/channels...

@alafanechere - I got pulled into other projects this afternoon but I'm wrapping up this evening with an added version bump and changelog so this is ready for you to merge tomorrow when you start your business hours. As noted in DM, I wasn't sure how to test prior to merging. I believe you can simply merge this and then retest during your business hours - creating a new PR for follow-up if necessary. (Biasing for faster time-to-resolution instead of fewer merges, since the tests are already broken.)

[alafanechere]

@aaronsteers So after some research my conclusion is:

• At test time we install dev dependencies on top of a python container built from our non root base image. The current user is airbyte.
• As we disable venv creation in our docker image poetry tries to install the dev dependencies at the system level. But as the current user is airbyte it has no permission to write to system packages.
• If we used pure pip to install dependency the workaround would be to use pip install --user and/or set PYTHONUSERBASE=/airbyte. This was my original attempt in this commit 515214c
• Poetry does not have a way to install packages in the user space when the venv creation is disabled. Install globally with --user python-poetry/poetry#1214
• Given that these operation on the container are performed a "test environment provisioning" time it is fine switching the use to root to run the dev deps install and switch back to root afterward.
• I'll revert my commit 515214c

[alafanechere]

Running tests on source-shopify and source-pokeapi to confirm the issue is resolved https://github.com/airbytehq/airbyte/actions/runs/12300651088

[aaronsteers]

Thanks for keeping this one moving forward, @alafanechere. I can't officially approve this PR because I was the original author - but please feel free to merge when ready. ✅ 👍

[alafanechere]

Shopify unit tests passed on this run (which correctly used airbyte-ci installed from sources)