Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Terraform provisioning for Jupiter into Azure Cloud #2733

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Terraform provisioning for Jupiter into Azure Cloud #2733

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
820961d
Move POC files over, with TODOs
murny Jan 15, 2022
9a03189
updates to scripts/env vars
murny Jan 26, 2022
012ace2
merge master into branch
murny Jan 26, 2022
294c8c8
Remove puma worker hack
murny Jan 26, 2022
740eaeb
Fix healthcheck on app pod by allowing private ips, up memory size of…
murny Jan 27, 2022
112ebbc
Get solr and rest of jupiter running
murny Jan 30, 2022
5d34020
Push up sample tfvars and rollback healthcheck work
murny Jan 30, 2022
158317e
Try fixing healthcheck
murny Jan 30, 2022
c970af3
Fix rubocop issues
murny Jan 30, 2022
be6d10d
Back out healthcheck work, need to revisit this
murny Jan 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,12 @@ coverage/
.ruby-version
vendor/bundle

/storage/*

/public/packs
/public/packs-test
/node_modules
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably shouldn't check all this stuff into our docker images 😆. This saves a good 100 MB or so from my images I was building


.dockerignore
docker-compose.yml
docker-compose.lightweight.yml
Expand Down
4 changes: 0 additions & 4 deletions .env_deployment_sample
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
#Rails application env variables
RAILS_ENV=uat
DATABASE_URL=postgresql://jupiter:mysecretpassword@postgres:5432/
FCREPO_URL=http://fcrepo:8080/fcrepo/rest
Copy link
Contributor Author

@murny murny Jan 30, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fedora was removed long time ago...updated all references I could find here

SOLR_URL=http://solr:8983/solr/jupiter-uat
REDIS_URL=redis://redis/1
SECRET_KEY_BASE=33d83bc35b707a1f70ac9475cdaabd4224fca0d3edc07e0ce6db13515d1c1e3aaf4c3b8bf580d9ea6198ec90bae2005ca807fa46a9aee5906bd64be99cc2e065
Expand All @@ -10,16 +9,13 @@ SAML_CERTIFICATE=
ROLLBAR_TOKEN=
GOOGLE_ANALYTICS_TOKEN=
RAILS_LOG_TO_STDOUT=true
# Comma delimited string of rack attack safelisted IPs
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rack attack gem was removed long time ago

RACK_ATTACK_SAFELISTED_IPS=
TLD_LENGTH=3
GOOGLE_CLIENT_ID=
GOOGLE_CLIENT_SECRET=
GOOGLE_DEVELOPER_KEY=
ERA_HOST=era.uat.library.ualberta.ca
DIGITIZATION_HOST=digitalcollections.uat.library.ualberta.ca


# Postgres environment variables
PGDATA=/var/lib/postgresql/data/pgdata
POSTGRES_PASSWORD=mysecretpassword
Expand Down
4 changes: 4 additions & 0 deletions Gemfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,10 @@ gem 'google-api-client',
gem 'builder_deferred_tagging', github: 'ualbertalib/builder_deferred_tagging', tag: 'v0.01'
gem 'oaisys', github: 'ualbertalib/oaisys', tag: 'v1.0.3'

group :uat do
gem 'azure-storage-blob', require: false
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Required for Activestorage to use Azure blob storage as a bucket

end

# Seeds
group :development, :test, :uat do
gem 'faker', require: false
Expand Down
11 changes: 11 additions & 0 deletions Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,14 @@ GEM
public_suffix (>= 2.0.2, < 5.0)
amazing_print (1.4.0)
ast (2.4.2)
azure-storage-blob (2.0.3)
azure-storage-common (~> 2.0)
nokogiri (~> 1, >= 1.10.8)
azure-storage-common (2.0.4)
faraday (~> 1.0)
faraday_middleware (~> 1.0, >= 1.0.0.rc1)
net-http-persistent (~> 4.0)
nokogiri (~> 1, >= 1.10.8)
bcrypt (3.1.16)
better_errors (2.9.1)
coderay (>= 1.0.0)
Expand Down Expand Up @@ -233,6 +241,8 @@ GEM
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday_middleware (1.2.0)
faraday (~> 1.0)
ffi (1.15.5)
flipper (0.23.1)
flipper-active_record (0.23.0)
Expand Down Expand Up @@ -620,6 +630,7 @@ DEPENDENCIES
active_link_to
acts_as_rdfable!
addressable (~> 2.8.0)
azure-storage-blob
bcrypt (>= 3.1.13)
better_errors (>= 2.3.0)
binding_of_caller
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

**Requirements:**

- **Ruby** 2.5+
- **Ruby** 2.6+
- **PostgreSQL**
- **Redis**
- **Solr**
Expand Down
15 changes: 3 additions & 12 deletions config/puma.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
# the maximum value specified for Puma. Default is set to 5 threads for minimum
# and maximum; this matches the default thread size of Active Record.
#
max_threads_count = ENV.fetch('PUMA_MAX_THREADS', 5)
min_threads_count = ENV.fetch('PUMA_MIN_THREADS') { max_threads_count }
max_threads_count = ENV.fetch('RAILS_MAX_THREADS', 5)
min_threads_count = ENV.fetch('RAILS_MIN_THREADS') { max_threads_count }
threads min_threads_count, max_threads_count

# Specifies the `worker_timeout` threshold that Puma will use to wait before
Expand Down Expand Up @@ -39,14 +39,5 @@
#
# preload_app!

if ENV['RAILS_ENV'] == 'uat'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to fork puma workers

workers ENV.fetch('WEB_CONCURRENCY', 3)
preload_app!
on_worker_boot do
ActiveSupport.on_load(:active_record) do
ActiveRecord::Base.establish_connection
end
end
end
# Allow puma to be restarted by `rails restart` command.
# Allow puma to be restarted by `bin/rails restart` command.
plugin :tmp_restart
23 changes: 0 additions & 23 deletions config/secrets.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ shared:
allow_crawlers: <%= ENV['RAILS_ALLOW_CRAWLERS'] || false %>
preservation_queue_name: <%= ENV['PMPY_QUEUE_NAME'] || 'prod:pmpy_queue' %>
fits_path: <%= ENV['FITS_PATH'] || 'fits.sh' %>
rack_attack_safelisted_ips: <%= ENV['RACK_ATTACK_SAFELISTED_IPS'] || '""' %>
system_user_api_key: <%= ENV['SYSTEM_USER_API_KEY'] %>

tld_length: <%= ENV['TLD_LENGTH'] || 1 %>
Expand All @@ -63,11 +62,6 @@ development:
saml_idp_cert: ''
saml_idp_sso_target_url: ''

fcrepo_url: <%= ENV['FCREPO_URL'] || 'http://localhost:8080/fcrepo/rest' %>
fcrepo_user: fedoraAdmin
fcrepo_password: fedoraAdmin
fcrepo_base_path: /dev

preservation_queue_name: <%= ENV['PMPY_QUEUE_NAME'] || 'dev:pmpy_queue'%>
system_user_api_key: <%= ENV['SYSTEM_USER_API_KEY'] || '3eeb395e-63b7-11ea-bc55-0242ac130003' %>

Expand All @@ -88,11 +82,6 @@ test:
saml_idp_cert: ''
saml_idp_sso_target_url: ''

fcrepo_url: <%= ENV['FCREPO_URL'] || 'http://localhost:8080/fcrepo/rest' %>
fcrepo_user: fedoraAdmin
fcrepo_password: fedoraAdmin
fcrepo_base_path: /test

preservation_queue_name: <%= ENV['PMPY_QUEUE_NAME'] || 'test:pmpy_queue' %>
system_user_api_key: <%= ENV['SYSTEM_USER_API_KEY'] || '3eeb395e-63b7-11ea-bc55-0242ac130003' %>

Expand All @@ -117,10 +106,6 @@ uat:
saml_idp_sso_target_url: 'https://login-uat.ualberta.ca/saml2/idp/SSOService.php'

database_url: <%= ENV['DATABASE_URL'] %>
fcrepo_user: <%= ENV['FCREPO_USER'] %>
fcrepo_password: <%= ENV['FCREPO_PASSWORD'] %>
fcrepo_url: <%= ENV['FCREPO_URL'] %>
fcrepo_base_path: /uat
solr_url: <%= ENV['SOLR_URL'] %>

staging:
Expand All @@ -135,10 +120,6 @@ staging:
saml_idp_sso_target_url: 'https://login-uat.ualberta.ca/saml2/idp/SSOService.php'

database_url: <%= ENV['DATABASE_URL'] %>
fcrepo_user: <%= ENV['FCREPO_USER'] %>
fcrepo_password: <%= ENV['FCREPO_PASSWORD'] %>
fcrepo_url: <%= ENV['FCREPO_URL'] %>
fcrepo_base_path: /prod
solr_url: <%= ENV['SOLR_URL'] %>

production:
Expand All @@ -154,10 +135,6 @@ production:
saml_idp_sso_target_url: 'https://login.ualberta.ca/saml2/idp/SSOService.php'

database_url: <%= ENV['DATABASE_URL'] %>
fcrepo_user: <%= ENV['FCREPO_USER'] %>
fcrepo_password: <%= ENV['FCREPO_PASSWORD'] %>
fcrepo_url: <%= ENV['FCREPO_URL'] %>
fcrepo_base_path: /prod
solr_url: <%= ENV['SOLR_URL'] %>

skylight_authentication: <%= ENV['SKYLIGHT_AUTHENTICATION'] %>
10 changes: 5 additions & 5 deletions config/storage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,11 @@ local:
# bucket: your_own_bucket

# Use rails credentials:edit to set the Azure Storage secret (as azure_storage:storage_access_key)
# microsoft:
# service: AzureStorage
# storage_account_name: your_account_name
# storage_access_key: <%= Rails.application.credentials.dig(:azure_storage, :storage_access_key) %>
# container: your_container_name
microsoft:
service: AzureStorage
storage_account_name: <%= ENV["AZURE_STORAGE_ACCOUNT_NAME"] %>
storage_access_key: <%= ENV["AZURE_STORAGE_ACCESS_KEY"] %>
container: <%= ENV["AZURE_STORAGE_CONTAINER"] %>

# mirror:
# service: Mirror
Expand Down
47 changes: 47 additions & 0 deletions terraform/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log
# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate
*.tfstate.*

# Crash log files
crash.log
crash.*.log

# Exclude all .tfvars files, which are likely to contain sentitive data, such as
# password, private keys, and other secrets. These should not be part of version
# control as they are data points which are potentially sensitive and subject
# to change depending on the environment.
#
*.tfvars

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
#
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

kubeconfig
94 changes: 94 additions & 0 deletions terraform/.terraform.lock.hcl
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading