Upgrade conda-store to 2024.11.2

[marcelovilla]

Reference Issues or PRs

Closes #2523
Closes #2543

What does this implement/fix?

Put a x in the boxes that apply

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds a feature)
• Breaking change (fix or feature that would cause existing features not to work as expected)
• Documentation Update
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no API changes)
• Build related changes
• Other (please describe):

Testing

• Did you test the pull request locally?
• Did you add new tests?

How to test this PR?

Deploy Nebari off of this branch and make sure to use the jupyterlab docker image built from nebari-dev/nebari-docker-images#187. Then, follow the conda-store related steps in our testing checklist:

nebari/.github/ISSUE_TEMPLATE/testing-checklist.md

Lines 23 to 25 in 9b1310b

	- [ ] [Log into conda-store and create](https://www.nebari.dev/docs/tutorials/creating-new-environments)
	- [ ] a conda environment in a shared namespace and,
	- [ ] a conda environment in your personal namespace

Any other comments?

[marcelovilla]

For context, conda-store's internal API was privatized on conda-incubator/conda-store#820. To configure conda-store in Nebari, we're still relying on an internal object:

nebari/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/conda-store/config/conda_store_config.py

Line 14 in a4e331a

from conda_store_server._internal import schema

Initially, I tried vendoring the schema to avoid depending on an internal private object that does not guarantee backwards compatibility, but I ran into the following issue when trying to log in to conda-store in a local Nebari deployment:

Traceback (most recent call last):
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
         ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 187, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda-store-server/conda_store_server/_internal/server/app.py", line 263, in conda_store_middleware
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/sessions.py", line 85, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda-store-server/conda_store_server/server/auth.py", line 547, in post_login_method
    authentication_token = await self.authenticate(request)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/etc/conda-store/conda_store_config.py", line 448, in authenticate
    return AuthenticationToken(
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/pydantic/main.py", line 339, in __init__
    values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/pydantic/main.py", line 1074, in validate_model
    v_, errors_ = field.validate(value, values, loc=field.alias, cls=cls_)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/conda-store-server/lib/python3.12/site-packages/pydantic/fields.py", line 857, in validate
    raise ConfigError(
pydantic.errors.ConfigError: field "role_bindings" not yet prepared so type is still a ForwardRef, you might need to call AuthenticationToken.update_forward_refs().

After trying to look into it and try to solve the issue, I realized that if conda-store is publicly exposing their RBACAuthorizationBackend class, its public methods should not rely on a private schema.

We're currently depending on importing the schema (amongst other things) because we need to pass it on:

nebari/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/conda-store/config/conda_store_config.py

Lines 360 to 364 in a4e331a

	entity = schema.AuthenticationToken(
	primary_namespace=username, role_bindings={}
	)
	self.log.info(f"entity: {entity}")
	entity_bindings = self.authorization.get_entity_bindings(entity)

[marcelovilla]

I'm marking this PR as a draft because we're hitting conda-incubator/conda-store#899 when upgrading conda-store to 2024.10.1. It seems a fix has already been merged conda-incubator/conda-store#932 so we'll wait for the next conda-store release and upgrade to that version instead.

[viniciusdc]

I can have a look at why the roles are missing in the tests, change the docker images during deployment based on nebari-dev/nebari-docker-images#187 image tag