Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dependencies and Python version to 3.12 #1249

Closed
wants to merge 64 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
a549fc0
made updates to get tests passing on Python 3.11. Ran black on 3 files
jamesstottmoj Feb 14, 2024
bb03c5d
Merge branch 'main' into python-311-test
jamesstottmoj Feb 14, 2024
db39385
Fixed flaky test by checking attached role is in role name rather tha…
jamesstottmoj Feb 14, 2024
8ced4cd
Merge branch 'python-311-test' of github.com:ministryofjustice/analyt…
jamesstottmoj Feb 14, 2024
053e465
attempted to bump version to python 311 in dockerfile
jamesstottmoj Feb 15, 2024
4734bfd
bumped down requirements for kubernetes
jamesstottmoj Feb 15, 2024
05f84f3
added 3.11 image to Dockerfile
jamesstottmoj Feb 26, 2024
d1d87fa
Attempted to fix add user now we're using an alpine image
jamesstottmoj Feb 26, 2024
11136da
Further attempt to fix Dockerfile using apk command
jamesstottmoj Feb 26, 2024
0c6266d
replaced failing packages with what I think are their apk equivalents
jamesstottmoj Feb 26, 2024
6fc6de7
updated python3-dev to python3.11-dev
jamesstottmoj Feb 26, 2024
94cd951
reverted python-dev
jamesstottmoj Feb 26, 2024
4bf5e4d
added new image dependency
jamesstottmoj Feb 26, 2024
dcaa117
added another package in an attempt to fix stdio error
jamesstottmoj Feb 26, 2024
a72997a
removed warning
jamesstottmoj Feb 26, 2024
109a919
updated to test if file gets picked up
jamesstottmoj Feb 26, 2024
42220b3
added purposeful breaking change to see if it is picked up on test pi…
jamesstottmoj Feb 26, 2024
bef6250
removed breaking change as tests did not run
jamesstottmoj Feb 26, 2024
4dc5d05
added import on line 8 to check if it gets picked up
jamesstottmoj Feb 26, 2024
224e3fc
removed import
jamesstottmoj Feb 26, 2024
31d159f
adding new test to see if it's picked up
jamesstottmoj Feb 26, 2024
872983c
removing all tests from file as a test
jamesstottmoj Feb 26, 2024
a985206
adding tests back
jamesstottmoj Feb 26, 2024
510e087
Merge branch 'main' into python-311-test
jamesstottmoj Feb 26, 2024
33288d9
fixed merge conflicts
jamesstottmoj Feb 26, 2024
f6b2521
removed test
jamesstottmoj Feb 26, 2024
a319ad9
Merge branch 'main' into python-311-test
jamesstottmoj Feb 27, 2024
c27b801
Merge branch 'main' into python-311-test
jamesstottmoj Mar 12, 2024
86fd452
merge main into branch
jamesstottmoj Apr 4, 2024
880628f
bumped python version to 3.12. Bumped some dependencies to latest ver…
jamesstottmoj Apr 4, 2024
4b31610
attempt to fix tests
jamesstottmoj Apr 4, 2024
a3a6f5a
fixed tests that failed as a result of bumping to 3.12
jamesstottmoj Apr 4, 2024
d404b0b
updated to model-bakery
jamesstottmoj Apr 5, 2024
527d03e
moved model-bakery to requirements
jamesstottmoj Apr 5, 2024
7440b58
bumped model-bakery to latest version
jamesstottmoj Apr 5, 2024
d03f5e8
Merge branch 'main' into python-311-test
jamesstottmoj Apr 8, 2024
85c76f0
updated node package
jamesstottmoj Apr 8, 2024
f1f3afe
Merge branch 'python-311-test' of github.com:ministryofjustice/analyt…
jamesstottmoj Apr 8, 2024
14e09ba
Removed commented out tests. Updated running.md to specify python 3.12
jamesstottmoj Apr 9, 2024
0d28395
made updates to get tests passing on Python 3.11. Ran black on 3 files
jamesstottmoj Feb 14, 2024
7a44e27
attempted to bump version to python 311 in dockerfile
jamesstottmoj Feb 15, 2024
fd8c9f4
added 3.11 image to Dockerfile
jamesstottmoj Feb 26, 2024
ae5f964
Attempted to fix add user now we're using an alpine image
jamesstottmoj Feb 26, 2024
99a0d63
Further attempt to fix Dockerfile using apk command
jamesstottmoj Feb 26, 2024
a9bdac5
replaced failing packages with what I think are their apk equivalents
jamesstottmoj Feb 26, 2024
539bc44
updated python3-dev to python3.11-dev
jamesstottmoj Feb 26, 2024
668bd84
reverted python-dev
jamesstottmoj Feb 26, 2024
19ef268
added new image dependency
jamesstottmoj Feb 26, 2024
f4438fc
added another package in an attempt to fix stdio error
jamesstottmoj Feb 26, 2024
3b7ce7f
removed warning
jamesstottmoj Feb 26, 2024
7aaa6eb
updated to test if file gets picked up
jamesstottmoj Feb 26, 2024
7e81bbb
adding new test to see if it's picked up
jamesstottmoj Feb 26, 2024
b6189bd
fixed merge conflicts
jamesstottmoj Feb 26, 2024
e8c9f87
removed test
jamesstottmoj Feb 26, 2024
516edee
bumped python version to 3.12. Bumped some dependencies to latest ver…
jamesstottmoj Apr 4, 2024
5cb135f
attempt to fix tests
jamesstottmoj Apr 4, 2024
caa4a2c
fixed tests that failed as a result of bumping to 3.12
jamesstottmoj Apr 4, 2024
14f4ae9
updated to model-bakery
jamesstottmoj Apr 5, 2024
95fb48a
moved model-bakery to requirements
jamesstottmoj Apr 5, 2024
a69b978
updated node package
jamesstottmoj Apr 8, 2024
659ed79
Removed commented out tests. Updated running.md to specify python 3.12
jamesstottmoj Apr 9, 2024
0904be1
Merge branch 'python-311-test' of github.com:ministryofjustice/analyt…
jamesstottmoj Apr 9, 2024
34475c9
Resetting commit history - Python bumped to 3.12, dependencies bumped…
jamesstottmoj Apr 9, 2024
663f774
Merge branch 'python-311-test' of github.com:ministryofjustice/analyt…
jamesstottmoj Apr 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ __pycache__
/static
/tags
/venv
/venv_311
/.python-version
/coverage/
.idea/
Expand Down
34 changes: 18 additions & 16 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM 593291632749.dkr.ecr.eu-west-1.amazonaws.com/node:18.12.1-slim AS jsdep
FROM public.ecr.aws/docker/library/node:20.11.1 AS build-node
COPY package.json package-lock.json ./
COPY jest.config.js controlpanel/frontend/static /src/

Expand All @@ -9,7 +9,7 @@ RUN ./node_modules/.bin/sass --load-path=node_modules/ --style=compressed src/ap
WORKDIR /src
RUN /node_modules/.bin/jest

FROM 593291632749.dkr.ecr.eu-west-1.amazonaws.com/python:3.9-slim-buster AS base
FROM public.ecr.aws/docker/library/python:3.12-alpine3.18 AS base

ARG HELM_VERSION=3.14.1
ARG HELM_TARBALL=helm-v${HELM_VERSION}-linux-amd64.tar.gz
Expand All @@ -22,18 +22,20 @@ ENV DJANGO_SETTINGS_MODULE="controlpanel.settings" \
HELM_DATA_HOME=/tmp/helm/data

# create a user to run as
RUN addgroup -gid 1000 controlpanel && \
adduser -uid 1000 --gid 1000 controlpanel
RUN addgroup -g 1000 controlpanel \
&& adduser -G controlpanel -u 1000 controlpanel -D

RUN apt-get update \
&& apt-get install -y --no-install-recommends \
RUN apk update \
&& apk add --no-cache \
postgresql-client \
wget \
gcc \
libcurl4-gnutls-dev \
curl-dev \
python3-dev \
libgnutls28-dev \
libssl-dev \
gnutls-dev \
openssl-dev \
libffi-dev \
musl-dev \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /home/controlpanel
Expand Down Expand Up @@ -65,13 +67,13 @@ COPY docker docker
COPY tests tests

# install javascript dependencies
COPY --from=jsdep dist/app.css dist/app.js static/
COPY --from=jsdep node_modules/accessible-autocomplete/dist/ static/accessible-autocomplete
COPY --from=jsdep node_modules/govuk-frontend static/govuk-frontend
COPY --from=jsdep node_modules/@ministryofjustice/frontend/moj static/ministryofjustice-frontend
COPY --from=jsdep node_modules/html5shiv/dist static/html5-shiv
COPY --from=jsdep node_modules/jquery/dist static/jquery
COPY --from=jsdep node_modules/jquery-ui/dist/ static/jquery-ui
COPY --from=build-node dist/app.css dist/app.js static/
COPY --from=build-node node_modules/accessible-autocomplete/dist/ static/accessible-autocomplete
COPY --from=build-node node_modules/govuk-frontend static/govuk-frontend
COPY --from=build-node node_modules/@ministryofjustice/frontend/moj static/ministryofjustice-frontend
COPY --from=build-node node_modules/html5shiv/dist static/html5-shiv
COPY --from=build-node node_modules/jquery/dist static/jquery
COPY --from=build-node node_modules/jquery-ui/dist/ static/jquery-ui

# empty .env file to prevent warning messages
RUN touch .env
Expand Down
4 changes: 2 additions & 2 deletions doc/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ versions of the services.

## 1. Required Dependencies

The Control Panel app requires Python 3.8+ It has been confirmed to work
with Python 3.8.12.
The Control Panel app requires Python 3.12. It has been confirmed to work
with Python 3.12.2.

Install python dependencies with the following command:
```sh
Expand Down
50 changes: 25 additions & 25 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 9 additions & 9 deletions requirements.dev.txt
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
black==24.3.0
django-debug-toolbar==3.2.4
django-debug-toolbar==4.3.0
django-debug-toolbar-requests==1.0.5
django-elasticsearch-debug-toolbar==2.0.0
flake8==6.0.0
django-elasticsearch-debug-toolbar==3.0.2
flake8==7.0.0
ipdb==0.13.13
ipython==8.13.2
isort==5.12.0
pre-commit==2.20.0
pylint==2.17.4
pylint-django==2.4.4
pandas==1.5.2
ipython==8.21.0
isort==5.13.2
pandas==2.2.0
pre-commit==3.6.1
pylint==3.0.3
pylint-django==2.5.5
30 changes: 15 additions & 15 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,39 +1,39 @@
asgiref==3.7.2
auth0-python==4.5.0
asgiref==3.8.1
auth0-python==4.7.0
authlib==1.3.0
beautifulsoup4==4.12.2
beautifulsoup4==4.12.3
boto3==1.34.64
celery[sqs]==5.3.1
celery[sqs]==5.3.6
channels==4.0.0
channels-redis==4.2.0
daphne==4.0.0
Django==4.2.11
daphne==4.1.0
Django==5.0.4
django-crequest==2018.5.11
django-extensions==3.2.3
django-filter==24.1
django-prometheus==2.3.1
django-redis==5.4.0
django-simple-history==3.3.0
django-structlog==2.2.0
django-simple-history==3.4.0
django-structlog==7.1.0
djangorestframework==3.14.0
djproxy==2.3.6
elasticsearch-dsl==7.4.1
gunicorn==20.1.0
gunicorn==21.2.0
Jinja2==3.1.3
kubernetes==25.3.0
MarkupSafe==2.1.3
model-mommy==2.0.0
moto==1.3.14
MarkupSafe==2.1.5
model-bakery==1.17.0
moto==5.0.1
mozilla-django-oidc==4.0.0
psycopg2-binary==2.9.9
PyNaCl==1.5.0
pytest==7.3.1
pytest-django==4.5.2
pytest==8.0.0
pytest-django==4.8.0
python-dotenv==1.0.1
python-jose==3.3.0
pyyaml==6.0.1
rules==3.3
sentry-sdk==1.40.5
slackclient==2.9.4
urllib3==1.26.18
urllib3==2.0.7
uvicorn[standard]==0.28.0
10 changes: 5 additions & 5 deletions tests/api/cluster/test_access_to_s3buckets.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

# Third-party
import pytest
from model_mommy import mommy
from model_bakery import baker

# First-party/Local
from controlpanel.api.cluster import App, RoleGroup, User
Expand Down Expand Up @@ -32,7 +32,7 @@ def grant_policy_bucket_access():
) as grant_policy_bucket_access_action:
yield grant_policy_bucket_access_action

@pytest.yield_fixture
@pytest.fixture
def grant_policy_folder_access():
with patch(
"controlpanel.api.cluster.AWSPolicy.grant_folder_access"
Expand All @@ -47,14 +47,14 @@ def enable_db_for_all_tests(db):

@pytest.fixture
def bucket():
return mommy.prepare("api.S3Bucket")
return baker.prepare("api.S3Bucket")


@pytest.fixture
def entities(bucket, users):
return {
"app": App(mommy.prepare("api.App")),
"group": RoleGroup(mommy.prepare("api.IAMManagedPolicy")),
"app": App(baker.prepare("api.App")),
"group": RoleGroup(baker.prepare("api.IAMManagedPolicy")),
"user": User(users["normal_user"]),
}

Expand Down
4 changes: 2 additions & 2 deletions tests/api/cluster/test_bucket.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# Third-party
import pytest
from django.conf import settings
from model_mommy import mommy
from model_bakery import baker

# First-party/Local
from controlpanel.api import cluster
Expand All @@ -18,7 +18,7 @@ def enable_db_for_all_tests(db):

@pytest.fixture
def bucket():
return mommy.prepare("api.S3Bucket", name="test-bucket")
return baker.prepare("api.S3Bucket", name="test-bucket")


def test_arn(bucket):
Expand Down
4 changes: 2 additions & 2 deletions tests/api/cluster/test_group.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

# Third-party
import pytest
from model_mommy import mommy
from model_bakery import baker

# First-party/Local
from controlpanel.api import cluster
Expand All @@ -16,7 +16,7 @@ def enable_db_for_all_tests(db):

@pytest.fixture
def iam_managed_policy():
return mommy.make("api.IAMManagedPolicy", name="test")
return baker.make("api.IAMManagedPolicy", name="test")


def test_arn(settings, iam_managed_policy):
Expand Down
23 changes: 12 additions & 11 deletions tests/api/cluster/test_user.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Standard library
from unittest.mock import MagicMock, call, patch
from unittest.mock import call, patch

# Third-party
import pytest
Expand All @@ -26,18 +26,19 @@ def test_create(helm, settings, users):
call(
f"bootstrap-user-{user.slug}",
"mojanalytics/bootstrap-user",
f"--namespace=user-{user.slug}",
f"--set=Username={user.slug},",
f"Efsvolume={settings.EFS_VOLUME}",
f"--namespace=cpanel",
f"--set=Username={user.slug}",
),
call(
f"config-user-{user.slug}",
"mojanalytics/config-user",
f"provision-user-{user.slug}",
"mojanalytics/provision-user",
f"--namespace=user-{user.slug}",
f"--set=Username={user.slug}",
(f"--set=Username={user.slug},Efsvolume={settings.EFS_VOLUME},"
"OidcDomain=oidc.idp.example.com,Email=,Fullname="),
),
]
helm.upgrade_release.has_calls(expected_calls)

helm.upgrade_release.assert_has_calls(expected_calls)


def test_reset_home(helm, users):
Expand All @@ -52,6 +53,7 @@ def test_reset_home(helm, users):
f"--set=Username={user.slug}",
),
]

helm.upgrade_release.assert_has_calls(expected_calls)


Expand All @@ -74,10 +76,9 @@ def test_delete(aws_delete_role, helm, users):
cluster.User(user).delete()
aws_delete_role.assert_called_with(user.iam_role_name)
expected_calls = [
call(f"user-{user.slug}", "chart-release"),
call("cpanel", "chart-release"),
call('user-bob', 'chart-release', dry_run=False),
]
helm.delete.has_calls(expected_calls)
helm.delete.assert_has_calls(expected_calls)


def test_delete_eks_with_no_releases(aws_delete_role, helm, users):
Expand Down
14 changes: 7 additions & 7 deletions tests/api/filters/test_app_filter.py
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
# Third-party
from model_mommy import mommy
from model_bakery import baker
from rest_framework.reverse import reverse
from rest_framework.test import APITestCase


class AppFilterTest(APITestCase):
def setUp(self):
self.superuser = mommy.make("api.User", is_superuser=True)
self.app_admin = mommy.make("api.User", is_superuser=False)
self.app_1 = mommy.make("api.App", name="App 1")
self.app_2 = mommy.make("api.App", name="App 2")
mommy.make("api.UserApp", user=self.app_admin, app=self.app_1, is_admin=True)
mommy.make("api.UserApp", user=self.app_admin, app=self.app_2, is_admin=True)
self.superuser = baker.make("api.User", is_superuser=True)
self.app_admin = baker.make("api.User", is_superuser=False)
self.app_1 = baker.make("api.App", name="App 1")
self.app_2 = baker.make("api.App", name="App 2")
baker.make("api.UserApp", user=self.app_admin, app=self.app_1, is_admin=True)
baker.make("api.UserApp", user=self.app_admin, app=self.app_2, is_admin=True)

def test_everyone_see_everything(self):
for user in [self.superuser, self.app_admin]:
Expand Down
Loading
Loading