Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Loosen deps ranges for compat with api, worker #432

Merged
merged 5 commits into from
Nov 22, 2024

Conversation

suejung-sentry
Copy link
Contributor

@suejung-sentry suejung-sentry commented Nov 21, 2024

Loosen deps version ranges to be compatible with that currently in api and worker. Sets deps versions here in shared as minimum(api, worker) version
api requirements.txt
worker requirements.txt

Follows up to https://github.com/codecov/shared/pull/423/files

Tested by spinning up api and worker using this version of shared with docker compose up & checking for any build errors. Also ran pytest within those containers for any build errors there

Copy link

codecov bot commented Nov 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.81%. Comparing base (12361de) to head (31380a2).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #432      +/-   ##
==========================================
- Coverage   90.56%   89.81%   -0.75%     
==========================================
  Files         390      324      -66     
  Lines       11768     9379    -2389     
  Branches     1974     1677     -297     
==========================================
- Hits        10658     8424    -2234     
+ Misses       1026      891     -135     
+ Partials       84       64      -20     
Flag Coverage Δ
shared-docker-uploader ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@suejung-sentry suejung-sentry changed the title fix: Pin sentry-sdk to previous version fix: Pin deps to previous version for compat with api Nov 21, 2024
@suejung-sentry
Copy link
Contributor Author

suejung-sentry commented Nov 22, 2024

NOTE here are the errors I'm seeing:

API

api-1  |   File "/app-dir/graphql_api/types/flake_aggregates/flake_aggregates.py", line 3, in <module>
api-1  |     import polars as pl
api-1  | ModuleNotFoundError: No module named 'polars'

Same No module named 'polars' error when spinning up the container and when running pytest

Worker

when spinning up the docker container

2024-11-21 17:56:48 Traceback (most recent call last):
2024-11-21 17:56:48   File "/app-dir/main.py", line 142, in <module>
2024-11-21 17:56:48     main()
2024-11-21 17:56:48   File "/app-dir/main.py", line 138, in main
2024-11-21 17:56:48     cli(obj={})
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
2024-11-21 17:56:48     return self.main(*args, **kwargs)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1078, in main
2024-11-21 17:56:48     rv = self.invoke(ctx)
2024-11-21 17:56:48          ^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
2024-11-21 17:56:48     return _process_result(sub_ctx.command.invoke(sub_ctx))
2024-11-21 17:56:48                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
2024-11-21 17:56:48     return ctx.invoke(self.callback, **ctx.params)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 783, in invoke
2024-11-21 17:56:48     return __callback(*args, **kwargs)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/app-dir/main.py", line 122, in worker
2024-11-21 17:56:48     return app.celery_app.worker_main(argv=args)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/app/base.py", line 389, in worker_main
2024-11-21 17:56:48     self.start(argv=argv)
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/app/base.py", line 369, in start
2024-11-21 17:56:48     celery.main(args=argv, standalone_mode=False)
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1078, in main
2024-11-21 17:56:48     rv = self.invoke(ctx)
2024-11-21 17:56:48          ^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
2024-11-21 17:56:48     return _process_result(sub_ctx.command.invoke(sub_ctx))
2024-11-21 17:56:48                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
2024-11-21 17:56:48     return ctx.invoke(self.callback, **ctx.params)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/core.py", line 783, in invoke
2024-11-21 17:56:48     return __callback(*args, **kwargs)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
2024-11-21 17:56:48     return f(get_current_context(), *args, **kwargs)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/bin/base.py", line 134, in caller
2024-11-21 17:56:48     return f(ctx, *args, **kwargs)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/bin/worker.py", line 348, in worker
2024-11-21 17:56:48     worker = app.Worker(
2024-11-21 17:56:48              ^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/worker/worker.py", line 93, in __init__
2024-11-21 17:56:48     self.app.loader.init_worker()
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/loaders/base.py", line 110, in init_worker
2024-11-21 17:56:48     self.import_default_modules()
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/loaders/base.py", line 105, in import_default_modules
2024-11-21 17:56:48     return [self.import_task_module(m) for m in self.default_modules]
2024-11-21 17:56:48             ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/loaders/base.py", line 85, in import_task_module
2024-11-21 17:56:48     return self.import_from_cwd(module)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/loaders/base.py", line 91, in import_from_cwd
2024-11-21 17:56:48     return import_from_cwd(
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
2024-11-21 17:56:48     return imp(module, package=package)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/site-packages/celery/loaders/base.py", line 88, in import_module
2024-11-21 17:56:48     return importlib.import_module(module, package=package)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
2024-11-21 17:56:48     return _bootstrap._gcd_import(name[level:], package, level)
2024-11-21 17:56:48            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-11-21 17:56:48   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
2024-11-21 17:56:48   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
2024-11-21 17:56:48   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
2024-11-21 17:56:48   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
2024-11-21 17:56:48   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
2024-11-21 17:56:48   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
2024-11-21 17:56:48   File "/app-dir/tasks/__init__.py", line 5, in <module>
2024-11-21 17:56:48     from tasks.backfill_commit_data_to_storage import backfill_commit_data_to_storage_task
2024-11-21 17:56:48   File "/app-dir/tasks/backfill_commit_data_to_storage.py", line 12, in <module>
2024-11-21 17:56:48     from services.report import ReportService
2024-11-21 17:56:48   File "/app-dir/services/report/__init__.py", line 16, in <module>
2024-11-21 17:56:48     from shared.metrics import metrics
2024-11-21 17:56:48 ImportError: cannot import name 'metrics' from 'shared.metrics' (/shared/shared/metrics/__init__.py)
2024-11-21 17:56:48 [Command exited with 1]

when running pytest

docker exec -it umbrella-hat-worker-1 bash
root@95053cda1576:/app-dir# pytest
Configuration considered invalid, using dict as it is
ImportError while loading conftest '/app-dir/conftest.py'.
conftest.py:6: in <module>
    import vcr
/usr/local/lib/python3.12/site-packages/vcr/__init__.py:2: in <module>
    from .config import VCR
/usr/local/lib/python3.12/site-packages/vcr/config.py:11: in <module>
    from .cassette import Cassette
/usr/local/lib/python3.12/site-packages/vcr/cassette.py:12: in <module>
    from .patch import CassettePatcherBuilder
/usr/local/lib/python3.12/site-packages/vcr/patch.py:58: in <module>
    import httplib2
/usr/local/lib/python3.12/site-packages/httplib2/__init__.py:52: in <module>
    from . import auth
/usr/local/lib/python3.12/site-packages/httplib2/auth.py:18: in <module>
    auth_param_name = token.copy().setName("auth-param-name").addParseAction(pp.downcaseTokens)
E   AttributeError: module 'pyparsing' has no attribute 'downcaseTokens'

@@ -1385,15 +1385,15 @@ wheels = [

[[package]]
name = "sentry-sdk"
version = "2.18.0"
version = "2.13.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the only package thats actually being downgraded here, and I would suggest that you not pin this, but rather define a lower bound, like for the others.

The other packages are not actually being downgraded in the lockfile, but rather their minimum version is being lowered, which has an effect on api/worker pulling in shared and having to resolve dependency versions. So in api/worker, we could end up with lower dependencies.

If you are hitting actual errors with deprecated/removed API, you can pin this to a lower version using abs<x.y.z syntax like for sqlalchemy in the pyproject.toml.
Not quite sure whether you want to do that here in this repo, or in api/worker rather.

Copy link
Contributor Author

@suejung-sentry suejung-sentry Nov 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good points - thanks!
Fixed the language used in the PR to be more correct lol, also Michelle helped make that a range instead of pin.
I'll bump/fix the breaking change in sentry-sdk 2.18.0 after we get to a clean slate again - codecov/engineering-team#2994

RE: the other errors, looks like they are okay in staging per here so I probably had something wrong in my setup. But think we're good to go now 🚀

@michelletran-codecov
Copy link
Contributor

I tested both out by migrating them and then running them on staging (uh, tested with "critical path" workflows):

codecov/codecov-api#998
codecov/worker#910

Both work fine. Once this change is merged, I would do the updates to both repos straight away to unblock people.

@suejung-sentry suejung-sentry changed the title fix: Pin deps to previous version for compat with api fix: Loosen deps ranges for compat with api, worker Nov 22, 2024
Copy link
Contributor

@michelletran-codecov michelletran-codecov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can upgrade sentry-sdk in the lockfile later. I relaxed the pin, so that we have the option to use a newer version for some repos.

@suejung-sentry suejung-sentry added this pull request to the merge queue Nov 22, 2024
Merged via the queue into main with commit f96b72f Nov 22, 2024
6 checks passed
@suejung-sentry suejung-sentry deleted the sshin/fix/sentry-sdk branch November 22, 2024 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants