Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue/drop pkg resources - first part #7990

Closed
wants to merge 89 commits into from
Closed
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
37ee4b1
first commit
hlloreda Jul 31, 2024
7d333fe
Merge branch 'refs/heads/master' into issue/drop_pkg_resources
hlloreda Jul 31, 2024
021e009
wip
hlloreda Jul 31, 2024
1779105
mypy
hlloreda Jul 31, 2024
25893db
Merge branch 'refs/heads/master' into issue/drop_pkg_resources
hlloreda Aug 1, 2024
871e18d
refactor
hlloreda Aug 2, 2024
3a5743a
lower name
hlloreda Aug 8, 2024
a6a6dde
test
hlloreda Aug 9, 2024
c49b37e
Merge branch 'refs/heads/master' into issue/drop_pkg_resources
hlloreda Aug 9, 2024
a32e383
refactor
hlloreda Aug 9, 2024
3ea487c
Merge branch 'master' into issue/drop_pkg_resources
hlloreda Aug 19, 2024
3377996
rollback changes
hlloreda Aug 20, 2024
880d9da
rollback changes
hlloreda Aug 20, 2024
c5187f8
refactor
hlloreda Aug 21, 2024
a239a06
refactor
hlloreda Aug 21, 2024
06ea0cf
refactor
hlloreda Aug 21, 2024
aeb68cb
fix broken test
hlloreda Aug 21, 2024
c5d3246
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Aug 29, 2024
87e381a
refactor
hlloreda Aug 29, 2024
ae4915b
refactor
hlloreda Aug 29, 2024
4b8e2ad
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Aug 29, 2024
0768b2b
refactor
hlloreda Aug 29, 2024
8d71c66
add test safe requirement
hlloreda Aug 29, 2024
f884732
Remove SafeRequirement
hlloreda Aug 29, 2024
e523023
Remove SafeRequirement
hlloreda Aug 29, 2024
b0f0d8b
renaming
hlloreda Aug 29, 2024
b797205
Remove SafeRequirement
hlloreda Aug 29, 2024
259626f
fix parsing
hlloreda Aug 29, 2024
26420b2
fix name parameter
hlloreda Aug 30, 2024
b443555
refactor
hlloreda Aug 30, 2024
9a44a24
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Aug 30, 2024
fca2bfc
format mypy
hlloreda Aug 30, 2024
db0591c
fix comparing versions
hlloreda Sep 2, 2024
10b41bd
refactor + fix broken calls
hlloreda Sep 2, 2024
0c8d292
rollback + fix broken tests
hlloreda Sep 2, 2024
7962876
refactor
hlloreda Sep 2, 2024
f902c99
fix broken tests
hlloreda Sep 2, 2024
00c5506
refactor
hlloreda Sep 6, 2024
d65b1c0
Merge branch 'refs/heads/master' into issue/drop_pkg_resources_2
hlloreda Sep 16, 2024
e6975c5
merge master + introduce new type
hlloreda Sep 16, 2024
c8ec877
fix broken tests
hlloreda Sep 16, 2024
43a64c0
fix broken tests
hlloreda Sep 16, 2024
917f9ef
add warnings + use new name property
hlloreda Sep 16, 2024
1c43dc7
fix + mypy
hlloreda Sep 16, 2024
1dce986
refactor
hlloreda Sep 16, 2024
d241433
retake with plugins
hlloreda Sep 16, 2024
3a64bed
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Sep 17, 2024
8a88093
rollback old changes
hlloreda Sep 17, 2024
a046cf8
add missing test
hlloreda Sep 17, 2024
1896d2e
missing <
hlloreda Sep 17, 2024
22e4d1d
wrong number
hlloreda Sep 17, 2024
0c3ab6e
resolve mypy problems
hlloreda Sep 17, 2024
4f13c5d
refactor
hlloreda Sep 17, 2024
8e15960
more cases
hlloreda Sep 17, 2024
d8d03ac
refactor
hlloreda Sep 18, 2024
cc46563
refactor wrong type
hlloreda Sep 18, 2024
5fd461a
mypy-sync
hlloreda Sep 18, 2024
9037100
refactor
hlloreda Sep 18, 2024
d34ba90
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Sep 18, 2024
84270ae
fix broken tests
hlloreda Sep 18, 2024
6134daa
refactor
hlloreda Sep 18, 2024
45ef63f
refactor
hlloreda Sep 18, 2024
ab6f098
refactor
hlloreda Sep 18, 2024
4fb8902
regex broke tests
hlloreda Sep 18, 2024
784e533
remove old regex
hlloreda Sep 18, 2024
c750b49
incomplete refactor
hlloreda Sep 18, 2024
0993635
refactor
hlloreda Sep 18, 2024
cd11875
refactor
hlloreda Sep 18, 2024
c40fe2b
fix
hlloreda Sep 19, 2024
63f0a5f
refactor
hlloreda Sep 19, 2024
8facdcf
docstring
hlloreda Sep 19, 2024
bc43a12
refactor
hlloreda Sep 19, 2024
ccab505
add changelog
hlloreda Sep 20, 2024
254c14e
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Sep 20, 2024
d86a63f
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Sep 23, 2024
ae80eb8
refactor
hlloreda Sep 23, 2024
2d7bda9
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Sep 24, 2024
ba749a6
fix broken tests
hlloreda Sep 24, 2024
6aae7ae
revert change
hlloreda Sep 25, 2024
ba01c4d
rollback
hlloreda Sep 30, 2024
b12c5d2
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Sep 30, 2024
24692f8
refactor
hlloreda Sep 30, 2024
dc95c99
refactor
hlloreda Sep 30, 2024
3e71ebf
refactor to make sure to not share this instance
hlloreda Sep 30, 2024
2d79e9e
Merge branch 'master' into issue/drop_pkg_resources_2
hlloreda Oct 1, 2024
d77a95d
refactor
hlloreda Oct 2, 2024
af785c4
remove constraint on setuptools
hlloreda Oct 2, 2024
8e4ae57
missing revert changes
hlloreda Oct 2, 2024
73d9cf3
missing revert changes
hlloreda Oct 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# serve to show the default.
import importlib.metadata
import shutil
import sys, os, pkg_resources, datetime
import sys, os, datetime
from importlib.metadata import PackageNotFoundError
from sphinx.errors import ConfigError

Expand Down
6 changes: 2 additions & 4 deletions src/inmanta/agent/executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,12 @@
from dataclasses import dataclass
from typing import Any, Dict, Optional, Sequence

import pkg_resources

import inmanta.types
import inmanta.util
from inmanta import const
from inmanta.agent import config as cfg
from inmanta.data.model import PipConfig, ResourceIdStr, ResourceType, ResourceVersionIdStr
from inmanta.env import PythonEnvironment
from inmanta.env import PythonEnvironment, SafeRequirement
from inmanta.loader import ModuleSource
from inmanta.resources import Id
from inmanta.types import JsonType
Expand Down Expand Up @@ -310,7 +308,7 @@ def create_and_install_environment(self, blueprint: EnvBlueprint) -> None:
self.init_env()
if len(req): # install_for_config expects at least 1 requirement or a path to install
self.install_for_config(
requirements=list(pkg_resources.parse_requirements(req)),
requirements=[SafeRequirement(requirement_string=e) for e in req],
config=blueprint.pip_config,
)

Expand Down
5 changes: 2 additions & 3 deletions src/inmanta/agent/in_process_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,6 @@
from concurrent.futures.thread import ThreadPoolExecutor
from typing import Any, Optional

import pkg_resources

import inmanta.agent.cache
import inmanta.protocol
import inmanta.util
Expand All @@ -35,6 +33,7 @@
from inmanta.agent.handler import HandlerAPI, SkipResource
from inmanta.const import ParameterSource
from inmanta.data.model import AttributeStateChange, ResourceIdStr, ResourceVersionIdStr
from inmanta.env import SafeRequirement
from inmanta.loader import CodeLoader
from inmanta.resources import Id, Resource
from inmanta.types import Apireturn
Expand Down Expand Up @@ -572,7 +571,7 @@ async def _install(self, blueprint: executor.ExecutorBlueprint) -> None:
await loop.run_in_executor(
self.process.thread_pool,
self._env.install_for_config,
list(pkg_resources.parse_requirements(blueprint.requirements)),
[SafeRequirement(requirement_string=e) for e in blueprint.requirements],
blueprint.pip_config,
)
await loop.run_in_executor(self.process.thread_pool, self._loader.deploy_version, blueprint.sources)
125 changes: 77 additions & 48 deletions src/inmanta/env.py

Large diffs are not rendered by default.

14 changes: 7 additions & 7 deletions src/inmanta/file_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,7 @@

import os

from pkg_resources import Requirement

from inmanta.env import SafeRequirement
from ruamel.yaml import YAML
from ruamel.yaml.comments import CommentedMap

Expand Down Expand Up @@ -56,11 +55,11 @@ class RequirementsTxtParser:
"""

@classmethod
def parse(cls, filename: str) -> list[Requirement]:
def parse(cls, filename: str) -> list[SafeRequirement]:
"""
Get all the requirements in `filename` as a list of `Requirement` instances.
Get all the requirements in `filename` as a list of `SafeRequirement` instances.
"""
return [Requirement.parse(r) for r in cls.parse_requirements_as_strs(filename)]
return [SafeRequirement(requirement_string=r) for r in cls.parse_requirements_as_strs(filename)]

@classmethod
def parse_requirements_as_strs(cls, filename: str) -> list[str]:
Expand All @@ -86,21 +85,22 @@ def get_content_with_dep_removed(cls, filename: str, remove_dep_on_pkg: str) ->
if not os.path.exists(filename):
raise Exception(f"File {filename} doesn't exist")

removed_dependency = SafeRequirement(requirement_string=remove_dep_on_pkg)
result = ""
line_continuation_buffer = ""
with open(filename, encoding="utf-8") as fd:
for line in fd.readlines():
if line_continuation_buffer:
line_continuation_buffer += line
if not line.endswith("\\"):
if Requirement.parse(line_continuation_buffer).key != remove_dep_on_pkg:
if SafeRequirement(requirement_string=line_continuation_buffer).name != removed_dependency.name:
result += line_continuation_buffer
line_continuation_buffer = ""
elif not line.strip() or line.strip().startswith("#"):
result += line
elif line.endswith("\\"):
line_continuation_buffer = line
elif Requirement.parse(line).key != remove_dep_on_pkg.lower():
elif SafeRequirement(requirement_string=line).name != removed_dependency.name:
result += line
else:
# Dependency matches `remove_dep_on_pkg` => Remove line from result
Expand Down
4 changes: 2 additions & 2 deletions src/inmanta/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@
from typing import Any, Callable, Optional, Union, cast

import click
import importlib_metadata
import texttable
from click_plugins import with_plugins
from pkg_resources import iter_entry_points

from inmanta import protocol
from inmanta.config import Config, cmdline_rest_transport
Expand Down Expand Up @@ -182,7 +182,7 @@ def get_table(header: list[str], rows: list[list[str]], data_type: Optional[list
return table.draw()


@with_plugins(iter_entry_points("inmanta.cli_plugins"))
@with_plugins(importlib_metadata.entry_points(group="inmanta.cli_plugins"))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mypy is raising a warning here:

src/inmanta/main.py:185: error: Argument 1 to "with_plugins" has incompatible type "EntryPoints"; expected "Generator[EntryPoint, None, None]"  [arg-type]

I don't know how should I fix this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
@with_plugins(importlib_metadata.entry_points(group="inmanta.cli_plugins"))
@with_plugins(iter(importlib_metadata.entry_points(group="inmanta.cli_plugins")))

like so?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't seem to be maintained I think it is good to figure out why it is here at all (I suspect LSM).

We may want to vendor it or find a fork if we need it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are now experiencing this error:

src/inmanta/main.py:185: error: Argument 1 to "iter" has incompatible type "EntryPoints"; expected "SupportsIter[Generator[EntryPoint, None, None]]"  [arg-type]
src/inmanta/main.py:185: note: Following member(s) of "EntryPoints" have conflicts:
src/inmanta/main.py:185: note:     Expected:
src/inmanta/main.py:185: note:         def __iter__(self) -> Generator[EntryPoint, None, None]
src/inmanta/main.py:185: note:     Got:
src/inmanta/main.py:185: note:         def __iter__(self) -> Iterator[Any]

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, click-plugin is just outdated, perhaps we can vendor it?

@click.group(help="Base command")
@click.option("--host", help="The server hostname to connect to")
@click.option("--port", help="The server port to connect to")
Expand Down
Loading