Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBTP-457 Add DBT PaaS CodeBuild configuration #464

Merged
merged 66 commits into from
Oct 9, 2023
Merged
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
933b9fd
Rename celery stuff in Procfile
WillGibson Sep 22, 2023
f0ab58e
Merge branch 'main' into DBTP-457-add-dbt-paas-codebuild-configuratio…
WillGibson Sep 22, 2023
6fb4fd2
Merge branch 'main' into DBTP-457-add-dbt-paas-codebuild-configuratio…
WillGibson Sep 22, 2023
98df597
Add CodeBuild badge to README
WillGibson Sep 22, 2023
0238806
Add initial-codebuild-configuration
WillGibson Sep 22, 2023
0980bb8
Set PAKETO_BUILDER_VERSION and LIFECYCLE_VERSION
WillGibson Sep 22, 2023
441cb2b
Try just building a single celery image
WillGibson Sep 22, 2023
a2b6e27
Sort pii-ner-exclude.txt
WillGibson Sep 22, 2023
dc7dc54
Try compliling assets during build
WillGibson Sep 22, 2023
4c83f0c
Exclude some false Named Entity Recognition positives
WillGibson Sep 22, 2023
7ab0ad2
cd src to collectstatic
WillGibson Sep 22, 2023
3a9d23b
Add config.settings/build.py
WillGibson Sep 22, 2023
0364817
Skip celery for now
WillGibson Sep 22, 2023
7b871dc
Separate out config/settings/base_after_django_environ.py
WillGibson Sep 22, 2023
82f0189
Add secret key, lose authbroker stuff
WillGibson Sep 22, 2023
5107e11
Add ELASTICSEARCH_DSL
WillGibson Sep 22, 2023
2f4bdd0
Add ClamAV stuff
WillGibson Sep 22, 2023
623b4f3
Add APP_ENV
WillGibson Sep 22, 2023
37a2026
Add DATABASES
WillGibson Sep 22, 2023
146f13b
Delete comented out stuff
WillGibson Sep 22, 2023
4ebad3c
Add a little debug output
WillGibson Sep 22, 2023
ed5f86d
Correct import in src/config/settings/env.py
WillGibson Sep 25, 2023
f9f78dd
Don't initialise_db_dict() for build
WillGibson Sep 25, 2023
de35afb
Let black do the reformatting
WillGibson Sep 25, 2023
277525f
Fix ruff checks
WillGibson Sep 25, 2023
24cac9a
WAGTAILSEARCH_BACKENDS["default"]["URLS"]
WillGibson Sep 25, 2023
44228cb
Import .base_after_django_environ in local.py
WillGibson Sep 25, 2023
b93daae
Fix WAGTAILSEARCH_BACKENDS["default"]["URLS"]
WillGibson Sep 25, 2023
6e5a968
Move WAGTAILSEARCH_BACKENDS
WillGibson Sep 25, 2023
c9060c1
Re-enable celery in codebuild/process.yml
WillGibson Sep 25, 2023
3fc1501
Merge branch 'main' into DBTP-457-add-dbt-paas-codebuild-configuratio…
WillGibson Sep 25, 2023
39c8348
Build out the three :-( images
WillGibson Sep 25, 2023
1594c7d
Keep build.py separate from original settings files
WillGibson Sep 26, 2023
adc12ed
Drop collect static back into Procfile
WillGibson Sep 26, 2023
319feb4
Add comment to explain build.py's existence
WillGibson Sep 26, 2023
66e4654
Update AWS CodeBuild badge
WillGibson Sep 26, 2023
4ce9b33
Restore settings-temp
WillGibson Sep 26, 2023
5fbc656
Try test settings and .env file
WillGibson Sep 26, 2023
61e8be9
Back out a directory before we try and move it back
WillGibson Sep 26, 2023
633dfa3
Make user_post.sh output a bit more informative
WillGibson Sep 26, 2023
49fca88
Delete redundant settings files
WillGibson Sep 26, 2023
ab31b8b
Exit early if something goes wrong
WillGibson Sep 26, 2023
ed06ef0
Added celery and beat builds back in
antroy-madetech Sep 26, 2023
caecd7f
Trigger build
antroy-madetech Sep 26, 2023
2c088c7
Removed elastic search app and config
antroy-madetech Sep 26, 2023
ea0fa88
Append ssl parameter to the CELERY_BROKER_URL
antroy-madetech Sep 27, 2023
8fdbb93
Only use AWS_ACCESS_KEY_ID and SECRET_ACCESS_KEY when in GovPaaS
acodeninja Sep 27, 2023
a9a0437
Dump opensearch query logs
acodeninja Sep 29, 2023
0baa29d
Better debug logging
acodeninja Sep 29, 2023
d9f6027
Logger instance not writing to stdout
acodeninja Sep 29, 2023
855bc7b
WSGIRequest is not JSON serializable
acodeninja Sep 29, 2023
af2f25f
Remove query from debug
acodeninja Sep 29, 2023
da2d4d1
More logging
antroy-madetech Sep 29, 2023
3f39e4f
Revert to existing requirements
acodeninja Oct 2, 2023
c13f8ad
Run `make fix` to correct the formatting
CamLamb Oct 2, 2023
8cfdf87
Remove `ssl_cert_reqs=required` from local and testing scenarios
CamLamb Oct 2, 2023
facfab2
Remove django_extensions from testing INSTALLED_APPS
CamLamb Oct 2, 2023
88576da
Test if order matters between Procfile and process.yml
acodeninja Oct 2, 2023
56d6952
Merge branch 'main' into DBTP-457-add-dbt-paas-codebuild-configuratio…
antroy-madetech Oct 6, 2023
649260e
Remove codebuild check for now
antroy-madetech Oct 6, 2023
17eca34
Remove debugging
antroy-madetech Oct 6, 2023
8bb33ed
Merge branch 'main' into DBTP-457-add-dbt-paas-codebuild-configuratio…
antroy-madetech Oct 6, 2023
8f10726
Removed accidentally added whitespace
antroy-madetech Oct 6, 2023
8133b56
Refactor settings files
CamLamb Oct 9, 2023
4bd5b9c
Fix typo
CamLamb Oct 9, 2023
91abd1f
Add `# noqa F405` to prod settings
CamLamb Oct 9, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@
A [Wagtail](https://www.wagtail.io)-based intranet for the Department for Business & Trade.

Project documentation is available [here](https://uktrade.github.io/digital-workspace-v2/).

13 changes: 13 additions & 0 deletions buildpack.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"buildpacks": [
{
"paketo-buildpacks": "python"
},
{
"paketo-buildpacks": "nodejs"
},
{
"fagiani" : "run"
}
]
}
12 changes: 12 additions & 0 deletions codebuild/process.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
application:
name: intranet
process:
# The way DBT PaaS works currently we need one entry
# here for each of the entries in the Procfile.
# The only actual different in the three images created
# is the process they start up with.
# There is a plan ensure it only needs to build one
# image in the near future.
- web
- beat # celery beat
- worker # celery worker
20 changes: 20 additions & 0 deletions codebuild/web/buildspec.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
version: 0.2
env:
parameter-store:
SLACK_WORKSPACE_ID: "/codebuild/slack_workspace_id"
SLACK_CHANNEL_ID: "/codebuild/slack_channel_id"
SLACK_TOKEN: "/codebuild/slack_api_token"
variables:
PAKETO_BUILDER_VERSION: 0.2.443-full
LIFECYCLE_VERSION: 0.16.5

phases:
# install:

pre_build:
commands:
- codebuild-breakpoint

build:
commands:
- /work/build.sh
Empty file added src/__init__.py
Empty file.
Empty file.
12 changes: 7 additions & 5 deletions src/config/settings/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,16 +47,14 @@
AWS_REGION = app_bucket_creds["aws_region"]
AWS_S3_REGION_NAME = app_bucket_creds["aws_region"]
AWS_STORAGE_BUCKET_NAME = app_bucket_creds["bucket_name"]
AWS_ACCESS_KEY_ID = env("AWS_ACCESS_KEY_ID")
CamLamb marked this conversation as resolved.
Show resolved Hide resolved
AWS_SECRET_ACCESS_KEY = env("AWS_SECRET_ACCESS_KEY")
AWS_S3_HOST = "s3-eu-west-2.amazonaws.com"
else:
AWS_STORAGE_BUCKET_NAME = env("AWS_STORAGE_BUCKET_NAME")
AWS_REGION = env("AWS_REGION")
AWS_S3_REGION_NAME = env("AWS_REGION", default="eu-west-2")

# You don't seem to be able to sign S3 URLs with VCAP S3 creds
AWS_ACCESS_KEY_ID = env("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = env("AWS_SECRET_ACCESS_KEY")
AWS_S3_HOST = "s3-eu-west-2.amazonaws.com"

# Asset path used in parser
NEW_ASSET_PATH = env("NEW_ASSET_PATH")

Expand Down Expand Up @@ -432,6 +430,10 @@
credentials["host"],
credentials["port"],
)
elif is_copilot():
CELERY_BROKER_URL = (
env("CELERY_BROKER_URL", default=None) + "?ssl_cert_reqs=required"
)
else:
CELERY_BROKER_URL = env("CELERY_BROKER_URL", default=None)

Expand Down
16 changes: 8 additions & 8 deletions src/config/settings/env.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,16 @@
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")

INSTALLED_APPS += [ # noqa F405
"elasticapm.contrib.django",
# "elasticapm.contrib.django",
]

ELASTIC_APM = {
"SERVICE_NAME": "Digital Workspace",
"SECRET_TOKEN": env("ELASTIC_APM_SECRET_TOKEN"), # noqa F405
"SERVER_URL": env("ELASTIC_APM_SERVER_URL"), # noqa F405
"ENVIRONMENT": env("APP_ENV"), # noqa F405
"SERVER_TIMEOUT": env("ELASTIC_APM_SERVER_TIMEOUT", default="20s"), # noqa F405
}
# ELASTIC_APM = {
# "SERVICE_NAME": "Digital Workspace",
# "SECRET_TOKEN": env("ELASTIC_APM_SECRET_TOKEN"), # noqa F405
# "SERVER_URL": env("ELASTIC_APM_SERVER_URL"), # noqa F405
# "ENVIRONMENT": env("APP_ENV"), # noqa F405
# "SERVER_TIMEOUT": env("ELASTIC_APM_SERVER_TIMEOUT", default="20s"), # noqa F405
# }

CamLamb marked this conversation as resolved.
Show resolved Hide resolved
LOGGING = {
"version": 1,
Expand Down
5 changes: 0 additions & 5 deletions src/config/settings/test.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

from .base import * # noqa


APP_ENV = "test"
DEBUG = True
TEMPLATE_DEBUG = True
Expand All @@ -18,10 +17,6 @@
"AUTO_UPDATE": False,
}

INSTALLED_APPS += [ # noqa F405
"django_extensions",
]

marcelkornblum marked this conversation as resolved.
Show resolved Hide resolved
LOGGING["handlers"] |= { # noqa F405
"file": {
"level": "DEBUG",
Expand Down
2 changes: 1 addition & 1 deletion src/extended_search/apps.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@ def ready(self):

search_settings.initialise_field_dict()
search_settings.initialise_env_dict()
if settings.APP_ENV != "test":
if settings.APP_ENV not in ["test", "build"]:
CamLamb marked this conversation as resolved.
Show resolved Hide resolved
search_settings.initialise_db_dict()
1 change: 1 addition & 0 deletions src/search/templatetags/search.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ def search_category(context, *, category, limit=None, show_heading=False):
# pages will have the pinned results removed after pagination and cause
# the pages to have odd lengths.
search_results = list(pinned_results) + list(search_vector.search(query))

CamLamb marked this conversation as resolved.
Show resolved Hide resolved
count = len(search_results)

if limit:
Expand Down
24 changes: 24 additions & 0 deletions user-post.sh
CamLamb marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/usr/bin/env bash

# Exit early if something goes wrong
set -e

echo "Running post build script"

echo "Running pip install"
pip install -r requirements.txt

echo "Running npm ci"
npm ci

echo "Renaming .env.ci to .env"
mv ".env.ci" ".env"

cd src

echo "Running collectstatic"
python manage.py collectstatic --settings=config.settings.test --noinput

echo "Renaming .env to .env.ci"
cd ../
mv ".env" ".env.ci"