diff --git a/README.md b/README.md index 6d1673c6..1a2de50f 100644 --- a/README.md +++ b/README.md @@ -1,37 +1,37 @@ [![Maintained by Gruntwork.io](https://img.shields.io/badge/maintained%20by-gruntwork.io-%235849a6.svg)](https://gruntwork.io/?ref=repo_aws_vault) # Vault AWS Module -This repo contains a set of modules in the [modules folder](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules) for deploying a [Vault](https://www.vaultproject.io/) cluster on +This repo contains a set of modules in the [modules folder](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules) for deploying a [Vault](https://www.vaultproject.io/) cluster on [AWS](https://aws.amazon.com/) using [Terraform](https://www.terraform.io/). Vault is an open source tool for managing -secrets. By default, this Module uses [Consul](https://www.consul.io) as a [storage +secrets. By default, this Module uses [Consul](https://www.consul.io) as a [storage backend](https://www.vaultproject.io/docs/configuration/storage/index.html). You can optionally add an [S3](https://aws.amazon.com/s3/) backend for durability. ![Vault architecture](https://github.com/hashicorp/terraform-aws-vault/blob/master/_docs/architecture.png?raw=true) This Module includes: -* [install-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault): This module can be used to install Vault. It can be used in a - [Packer](https://www.packer.io/) template to create a Vault +* [install-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault): This module can be used to install Vault. It can be used in a + [Packer](https://www.packer.io/) template to create a Vault [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html). -* [run-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/run-vault): This module can be used to configure and run Vault. It can be used in a - [User Data](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts) +* [run-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/run-vault): This module can be used to configure and run Vault. It can be used in a + [User Data](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts) script to fire up Vault while the server is booting. -* [vault-cluster](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster): Terraform code to deploy a cluster of Vault servers using an [Auto Scaling +* [vault-cluster](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster): Terraform code to deploy a cluster of Vault servers using an [Auto Scaling Group](https://aws.amazon.com/autoscaling/). - -* [vault-elb](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-elb): Configures an [Elastic Load Balancer + +* [vault-elb](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-elb): Configures an [Elastic Load Balancer (ELB)](https://aws.amazon.com/elasticloadbalancing/classicloadbalancer/) in front of Vault if you need to access it from the public Internet. - -* [private-tls-cert](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/private-tls-cert): Generate a private TLS certificate for use with a private Vault + +* [private-tls-cert](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/private-tls-cert): Generate a private TLS certificate for use with a private Vault cluster. - -* [update-certificate-store](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/update-certificate-store): Add a trusted, CA public key to an OS's + +* [update-certificate-store](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/update-certificate-store): Add a trusted, CA public key to an OS's certificate store. This allows you to establish TLS connections to services that use this TLS certs signed by this CA without getting x509 certificate errors. - + ## How do you use this Module? @@ -41,45 +41,47 @@ This repo has the following structure: * [modules](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules): This folder contains several standalone, reusable, production-grade modules that you can use to deploy Vault. * [examples](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples): This folder shows examples of different ways to combine the modules in the `modules` folder to deploy Vault. * [test](https://github.com/hashicorp/terraform-aws-vault/tree/master/test): Automated tests for the modules and examples. -* [root folder](https://github.com/hashicorp/terraform-aws-vault/tree/master): The root folder is *an example* of how to use the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster) +* [root folder](https://github.com/hashicorp/terraform-aws-vault/tree/master): The root folder is *an example* of how to use the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster) module to deploy a [Vault](https://www.vaultproject.io/) cluster in [AWS](https://aws.amazon.com/). The Terraform Registry requires the root of every repo to contain Terraform code, so we've put one of the examples there. This example is great for learning and experimenting, but for production use, please use the underlying modules in the [modules folder](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules) directly. - -To deploy Vault to production with this repo, you will need to deploy two separate clusters: one to run -[Consul](https://www.consul.io/) servers (which Vault uses as a [storage -backend](https://www.vaultproject.io/docs/configuration/storage/index.html)) and one to run Vault servers. -To deploy the Consul server cluster, use the [Consul AWS Module](https://github.com/hashicorp/terraform-aws-consul). +To deploy Vault to production with this repo, you will need to deploy two separate clusters: one to run +[Consul](https://www.consul.io/) servers (which Vault uses as a [storage +backend](https://www.vaultproject.io/docs/configuration/storage/index.html)) and one to run Vault servers. + +To deploy the Consul server cluster, use the [Consul AWS Module](https://github.com/hashicorp/terraform-aws-consul). To deploy the Vault cluster: 1. Create an AMI that has Vault installed (using the [install-vault module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault)) and the Consul - agent installed (using the [install-consul - module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-consul)). Here is an - [example Packer template](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami). - + agent installed (using the [install-consul + module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-consul)). Here is an + [example Packer template](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami). + If you are just experimenting with this Module, you may find it more convenient to use one of our official public AMIs: - [Latest Ubuntu 16 AMIs](https://github.com/hashicorp/terraform-aws-vault/tree/master/_docs/ubuntu16-ami-list.md). - [Latest Amazon Linux 2 AMIs](https://github.com/hashicorp/terraform-aws-vault/tree/master/_docs/amazon-linux-ami-list.md). - - **WARNING! Do NOT use these AMIs in your production setup. In production, you should build your own AMIs in your + + **WARNING! Do NOT use these AMIs in your production setup. In production, you should build your own AMIs in your own AWS account.** -1. Deploy that AMI across an Auto Scaling Group in a private subnet using the Terraform [vault-cluster - module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). +1. Deploy that AMI across an Auto Scaling Group in a private subnet using the Terraform [vault-cluster + module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). 1. Execute the [run-consul script](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/run-consul) - with the `--client` flag during boot on each Instance to have the Consul agent connect to the Consul server cluster. + with the `--client` flag during boot on each Instance to have the Consul agent connect to the Consul server cluster. -1. Execute the [run-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/run-vault) script during boot on each Instance to create the Vault cluster. +1. Execute the [run-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/run-vault) script during boot on each Instance to create the Vault cluster. -1. If you only need to access Vault from inside your AWS account (recommended), run the [install-dnsmasq - module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq) on each server, and - that server will be able to reach Vault using the Consul Server cluster as the DNS resolver (e.g. using an address - like `vault.service.consul`). See the [vault-cluster-private example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) for working +1. If you only need to access Vault from inside your AWS account (recommended), run the [install-dnsmasq + module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq) on each server or + [setup-systemd-resolved](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved) + (in the case of Ubuntu 18.04) and + that server will be able to reach Vault using the Consul Server cluster as the DNS resolver (e.g. using an address + like `vault.service.consul`). See the [vault-cluster-private example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) for working sample code. -1. If you need to access Vault from the public Internet, deploy the [vault-elb module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-elb) in a public - subnet and have all requests to Vault go through the ELB. See the [main.tf in the root folder of this repo +1. If you need to access Vault from the public Internet, deploy the [vault-elb module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-elb) in a public + subnet and have all requests to Vault go through the ELB. See the [main.tf in the root folder of this repo example](https://github.com/hashicorp/terraform-aws-vault/blob/master/main.tf) for working sample code. 1. Head over to the [How do you use the Vault cluster?](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster#how-do-you-use-the-vault-cluster) guide @@ -90,22 +92,22 @@ To deploy the Vault cluster: ## What's a Module? -A Module is a canonical, reusable, best-practices definition for how to run a single piece of infrastructure, such -as a database or server cluster. Each Module is created primarily using [Terraform](https://www.terraform.io/), -includes automated tests, examples, and documentation, and is maintained both by the open source community and -companies that provide commercial support. +A Module is a canonical, reusable, best-practices definition for how to run a single piece of infrastructure, such +as a database or server cluster. Each Module is created primarily using [Terraform](https://www.terraform.io/), +includes automated tests, examples, and documentation, and is maintained both by the open source community and +companies that provide commercial support. -Instead of having to figure out the details of how to run a piece of infrastructure from scratch, you can reuse -existing code that has been proven in production. And instead of maintaining all that infrastructure code yourself, +Instead of having to figure out the details of how to run a piece of infrastructure from scratch, you can reuse +existing code that has been proven in production. And instead of maintaining all that infrastructure code yourself, you can leverage the work of the Module community and maintainers, and pick up infrastructure improvements through a version number bump. - - - + + + ## Who maintains this Module? -This Module is maintained by [Gruntwork](http://www.gruntwork.io/). If you're looking for help or commercial -support, send an email to [modules@gruntwork.io](mailto:modules@gruntwork.io?Subject=Vault%20Module). +This Module is maintained by [Gruntwork](http://www.gruntwork.io/). If you're looking for help or commercial +support, send an email to [modules@gruntwork.io](mailto:modules@gruntwork.io?Subject=Vault%20Module). Gruntwork can help with: * Setup, customization, and support for this Module. @@ -124,18 +126,18 @@ Contributions are very welcome! Check out the [Contribution Guidelines](https:// ## How is this Module versioned? -This Module follows the principles of [Semantic Versioning](http://semver.org/). You can find each new release, -along with the changelog, in the [Releases Page](../../releases). +This Module follows the principles of [Semantic Versioning](http://semver.org/). You can find each new release, +along with the changelog, in the [Releases Page](../../releases). -During initial development, the major version will be 0 (e.g., `0.x.y`), which indicates the code does not yet have a -stable API. Once we hit `1.0.0`, we will make every effort to maintain a backwards compatible API and use the MAJOR, -MINOR, and PATCH versions on each release to indicate any incompatibilities. +During initial development, the major version will be 0 (e.g., `0.x.y`), which indicates the code does not yet have a +stable API. Once we hit `1.0.0`, we will make every effort to maintain a backwards compatible API and use the MAJOR, +MINOR, and PATCH versions on each release to indicate any incompatibilities. ## License -This code is released under the Apache 2.0 License. Please see [LICENSE](https://github.com/hashicorp/terraform-aws-vault/tree/master/LICENSE) and [NOTICE](https://github.com/hashicorp/terraform-aws-vault/tree/master/NOTICE) for more +This code is released under the Apache 2.0 License. Please see [LICENSE](https://github.com/hashicorp/terraform-aws-vault/tree/master/LICENSE) and [NOTICE](https://github.com/hashicorp/terraform-aws-vault/tree/master/NOTICE) for more details. Copyright © 2017 Gruntwork, Inc. diff --git a/examples/vault-agent/README.md b/examples/vault-agent/README.md index 24f8f7a5..0a80c92a 100644 --- a/examples/vault-agent/README.md +++ b/examples/vault-agent/README.md @@ -21,10 +21,12 @@ of a custom VPC. You will need to create an [Amazon Machine Image (AMI)][ami] that has both Vault and Consul installed, which you can do using the [vault-consul-ami example][vault_consul_ami]). All the EC2 Instances in this example (including the EC2 Instance that authenticates to Vault) install -[Dnsmasq][dnsmasq] (via the [install-dnsmasq module][dnsmasq_module]) so that all DNS queries -for `*.consul` will be directed to the Consul Server cluster. Because Consul has knowledge of -all the Vault nodes (and in some cases, of other services as well), this setup allows the EC2 -Instance to use Consul's DNS server for service discovery, and thereby to discover the IP addresses +either [Dnsmasq][dnsmasq] (via the [install-dnsmasq module][dnsmasq_module]) +or [setup-systemd-resolved][setup_systemd_resolved] (in the case of Ubuntu 18.04) +so that all DNS queries for `*.consul` will be directed to the +Consul Server cluster. Because Consul has knowledge of all the Vault nodes (and in +some cases, of other services as well), this setup allows the EC2 Instance to use +Consul's DNS server for service discovery, and thereby to discover the IP addresses of the Vault nodes. @@ -48,6 +50,7 @@ of the Vault nodes. [auto_auth]: https://www.vaultproject.io/docs/agent/autoauth/index.html [dnsmasq_module]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq [dnsmasq]: http://www.thekelleys.org.uk/dnsmasq/doc.html +[setup_systemd_resolved]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved [examples_helper]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-examples-helper/vault-examples-helper.sh [iam_auth]: https://www.vaultproject.io/docs/auth/aws.html#iam-auth-method [iam_example]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-iam-auth diff --git a/examples/vault-auto-unseal/README.md b/examples/vault-auto-unseal/README.md index 19bd69de..829e15e3 100644 --- a/examples/vault-auto-unseal/README.md +++ b/examples/vault-auto-unseal/README.md @@ -13,6 +13,7 @@ with the Vault VPC. The Vault cluster uses [Consul][consul] as the storage backe so this example also deploys a separate Consul server cluster using the [consul-cluster module][consul_cluster] from the Consul AWS Module. Each of the servers in this example has [Dnsmasq][dnsmasq] installed (via the [install-dnsmasq module][dnsmasq_module]) +or [setup-systemd-resolved][setup_systemd_resolved] (in the case of Ubuntu 18.04) which allows them to use the Consul server cluster for service discovery and thereby access Vault via DNS using the domain name `vault.service.consul`. @@ -111,6 +112,7 @@ Enterprise License to the cluster with `vault write /sys/license "text=$LICENSE_ [consul]: https://www.consul.io/ [dnsmasq_module]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq [dnsmasq]: http://www.thekelleys.org.uk/dnsmasq/doc.html +[setup_systemd_resolved]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved [examples_helper]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-examples-helper/vault-examples-helper.sh [key_creation]: https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html [kms]: https://aws.amazon.com/kms/ diff --git a/examples/vault-cluster-private/README.md b/examples/vault-cluster-private/README.md index 38814566..79984ee4 100644 --- a/examples/vault-cluster-private/README.md +++ b/examples/vault-cluster-private/README.md @@ -1,9 +1,9 @@ -# Private Vault Cluster Example +# Private Vault Cluster Example -This folder shows an example of Terraform code to deploy a [Vault](https://www.vaultproject.io/) cluster in -[AWS](https://aws.amazon.com/) using the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). The Vault cluster uses -[Consul](https://www.consul.io/) as a storage backend, so this example also deploys a separate Consul server cluster -using the [consul-cluster module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/consul-cluster) +This folder shows an example of Terraform code to deploy a [Vault](https://www.vaultproject.io/) cluster in +[AWS](https://aws.amazon.com/) using the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). The Vault cluster uses +[Consul](https://www.consul.io/) as a storage backend, so this example also deploys a separate Consul server cluster +using the [consul-cluster module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/consul-cluster) from the Consul AWS Module. This example creates a private Vault cluster, which is private in the sense that the EC2 Instances are not fronted by a @@ -12,20 +12,22 @@ nodes are deployed to public subnets (i.e. subnets that have a route to the publ still be accessible from the public Internet. Each of the servers in this example has [Dnsmasq](http://www.thekelleys.org.uk/dnsmasq/doc.html) installed (via the -[install-dnsmasq module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq)) -which allows it to use the Consul server cluster for service discovery and thereby access Vault via DNS using the +[install-dnsmasq module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq)) or +[setup-systemd-resolved](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved) +(in the case Ubuntu of 18.04) +which allows it to use the Consul server cluster for service discovery and thereby access Vault via DNS using the domain name `vault.service.consul`. For an example of a Vault cluster that is publicly accessible, see [the root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example). ![Vault architecture](https://github.com/hashicorp/terraform-aws-vault/blob/master/_docs/architecture.png?raw=true) -You will need to create an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) +You will need to create an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) that has Vault and Consul installed, which you can do using the [vault-consul-ami example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami)). For more info on how the Vault cluster works, check out the [vault-cluster](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster) documentation. -**Note**: To keep this example as simple to deploy and test as possible, it deploys the Vault cluster into your default -VPC and default subnets, all of which are publicly accessible. This is OK for learning and experimenting, but for +**Note**: To keep this example as simple to deploy and test as possible, it deploys the Vault cluster into your default +VPC and default subnets, all of which are publicly accessible. This is OK for learning and experimenting, but for production usage, we strongly recommend deploying the Vault cluster into the private subnets of a custom VPC. @@ -50,5 +52,5 @@ To deploy a Vault Cluster: print out the IP addresses of the Vault servers and some example commands you can run to interact with the cluster: `../vault-examples-helper/vault-examples-helper.sh`. -To see how to connect to the Vault cluster, initialize it, and start reading and writing secrets, head over to the +To see how to connect to the Vault cluster, initialize it, and start reading and writing secrets, head over to the [How do you use the Vault cluster?](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster#how-do-you-use-the-vault-cluster) docs. diff --git a/examples/vault-consul-ami/README.md b/examples/vault-consul-ami/README.md index e495326c..97b6eeaf 100644 --- a/examples/vault-consul-ami/README.md +++ b/examples/vault-consul-ami/README.md @@ -1,21 +1,23 @@ # Vault and Consul AMI -This folder shows an example of how to use the [install-vault module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault) from this Module and +This folder shows an example of how to use the [install-vault module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault) from this Module and the [install-consul](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-consul) -and [install-dnsmasq](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq) modules -from the Consul AWS Module with [Packer](https://www.packer.io/) to create [Amazon Machine Images +and [install-dnsmasq](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq) or the +[setup-systemd-resolved](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved) +modules from the Consul AWS Module with [Packer](https://www.packer.io/) to create [Amazon Machine Images (AMIs)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) that have Vault and Consul installed on top of: - + +1. Ubuntu 18.04 1. Ubuntu 16.04 1. Amazon Linux 2 You can use this AMI to deploy a [Vault cluster](https://www.vaultproject.io/) by using the [vault-cluster -module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). This Vault cluster will use Consul as its storage backend, so you can also use the -same AMI to deploy a separate [Consul server cluster](https://www.consul.io/) by using the [consul-cluster -module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/consul-cluster). +module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). This Vault cluster will use Consul as its storage backend, so you can also use the +same AMI to deploy a separate [Consul server cluster](https://www.consul.io/) by using the [consul-cluster +module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/consul-cluster). -Check out the [vault-cluster-private](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) and -[the root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) examples for working sample code. For more info on Vault +Check out the [vault-cluster-private](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) and +[the root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) examples for working sample code. For more info on Vault installation and configuration, check out the [install-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault) documentation. @@ -63,8 +65,8 @@ ubuntu AMI. ## Creating your own Packer template for production usage -When creating your own Packer template for production usage, you can copy the example in this folder more or less -exactly, except for one change: we recommend replacing the `file` provisioner with a call to `git clone` in the `shell` +When creating your own Packer template for production usage, you can copy the example in this folder more or less +exactly, except for one change: we recommend replacing the `file` provisioner with a call to `git clone` in the `shell` provisioner. Instead of: ```json @@ -100,6 +102,6 @@ Your code should look more like this: You should replace `` in the code above with the version of this module that you want to use (see the [Releases Page](https://github.com/hashicorp/terraform-aws-vault/releases) for all available versions). That's because for production usage, you should always -use a fixed, known version of this Module, downloaded from the official Git repo. On the other hand, when you're -just experimenting with the Module, it's OK to use a local checkout of the Module, uploaded from your own +use a fixed, known version of this Module, downloaded from the official Git repo. On the other hand, when you're +just experimenting with the Module, it's OK to use a local checkout of the Module, uploaded from your own computer. diff --git a/examples/vault-consul-ami/vault-consul.json b/examples/vault-consul-ami/vault-consul.json index 9813f6d9..206cee45 100644 --- a/examples/vault-consul-ami/vault-consul.json +++ b/examples/vault-consul-ami/vault-consul.json @@ -3,8 +3,8 @@ "variables": { "aws_region": "us-east-1", "vault_version": "1.1.0", - "consul_module_version": "v0.4.2", - "consul_version": "1.3.1", + "consul_module_version": "v0.7.3", + "consul_version": "1.5.3", "consul_download_url": "{{env `CONSUL_DOWNLOAD_URL`}}", "vault_download_url": "{{env `VAULT_DOWNLOAD_URL`}}", "install_auth_signing_script": "true", @@ -13,7 +13,26 @@ "tls_private_key_path": null }, "builders": [{ - "ami_name": "vault-consul-ubuntu-{{isotime | clean_ami_name}}-{{uuid}}", + "ami_name": "vault-consul-ubuntu18-{{isotime | clean_ami_name}}-{{uuid}}", + "ami_description": "An Ubuntu 18.04 AMI that has Vault and Consul installed.", + "instance_type": "t2.micro", + "name": "ubuntu18-ami", + "region": "{{user `aws_region`}}", + "type": "amazon-ebs", + "source_ami_filter": { + "filters": { + "virtualization-type": "hvm", + "architecture": "x86_64", + "name": "*ubuntu-bionic-18.04-amd64-server-*", + "block-device-mapping.volume-type": "gp2", + "root-device-type": "ebs" + }, + "owners": ["099720109477"], + "most_recent": true + }, + "ssh_username": "ubuntu" + },{ + "ami_name": "vault-consul-ubuntu16-{{isotime | clean_ami_name}}-{{uuid}}", "ami_description": "An Ubuntu 16.04 AMI that has Vault and Consul installed.", "instance_type": "t2.micro", "name": "ubuntu16-ami", @@ -111,7 +130,7 @@ "fi" ], "inline_shebang": "/bin/bash -e", - "only": ["ubuntu16-ami"] + "only": ["ubuntu16-ami","ubuntu18-ami"] },{ "type": "shell", "inline": [ @@ -130,8 +149,21 @@ " /tmp/terraform-aws-consul/modules/install-consul/install-consul --download-url {{user `consul_download_url`}};", "else", " /tmp/terraform-aws-consul/modules/install-consul/install-consul --version {{user `consul_version`}};", - "fi", + "fi" + ], + "pause_before": "30s" + },{ + "type": "shell", + "only": ["ubuntu16-ami", "amazon-linux-2-ami"], + "inline": [ "/tmp/terraform-aws-consul/modules/install-dnsmasq/install-dnsmasq" ] + },{ + "type": "shell", + "only": ["ubuntu18-ami"], + "inline": [ + "/tmp/terraform-aws-consul/modules/setup-systemd-resolved/setup-systemd-resolved" + ], + "pause_before": "30s" }] } diff --git a/examples/vault-ec2-auth/README.md b/examples/vault-ec2-auth/README.md index e829aa9e..a7811fc4 100644 --- a/examples/vault-ec2-auth/README.md +++ b/examples/vault-ec2-auth/README.md @@ -25,7 +25,8 @@ of a custom VPC. You will need to create an [Amazon Machine Image (AMI)][ami] that has both Vault and Consul installed, which you can do using the [vault-consul-ami example][vault_consul_ami]). All the EC2 Instances in this example (including the EC2 Instance that authenticates to Vault) install -[Dnsmasq][dnsmasq] (via the [install-dnsmasq module][dnsmasq_module]) so that all DNS queries +[Dnsmasq][dnsmasq] (via the [install-dnsmasq module][dnsmasq_module]) or +[setup-systemd-resolved][setup_systemd_resolved] (in the case of Ubuntu 18.04) so that all DNS queries for `*.consul` will be directed to the Consul Server cluster. Because Consul has knowledge of all the Vault nodes (and in some cases, of other services as well), this setup allows the EC2 Instance to use Consul's DNS server for service discovery, and thereby to discover the IP addresses @@ -183,6 +184,7 @@ To see the full script for authenticating check the [client user data script][us [create_role]: https://www.vaultproject.io/api/auth/aws/index.html#create-role [dnsmasq_module]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq [dnsmasq]: http://www.thekelleys.org.uk/dnsmasq/doc.html +[setup_systemd_resolved]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved [ec2_auth]: https://www.vaultproject.io/docs/auth/aws.html#ec2-auth-method [examples_helper]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-examples-helper/vault-examples-helper.sh [iam_example]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-iam-auth diff --git a/examples/vault-iam-auth/README.md b/examples/vault-iam-auth/README.md index fc2c930b..7557e5ab 100644 --- a/examples/vault-iam-auth/README.md +++ b/examples/vault-iam-auth/README.md @@ -25,7 +25,8 @@ of a custom VPC. You will need to create an [Amazon Machine Image (AMI)][ami] that has both Vault and Consul installed, which you can do using the [vault-consul-ami example][vault_consul_ami]). All the EC2 Instances in this example (including the EC2 Instance that authenticates to Vault) install -[Dnsmasq][dnsmasq] (via the [install-dnsmasq module][dnsmasq_module]) so that all DNS queries +[Dnsmasq][dnsmasq] (via the [install-dnsmasq module][dnsmasq_module]) or +[setup-systemd-resolved][setup_systemd_resolved] (in the case of Ubuntu 18.04) so that all DNS queries for `*.consul` will be directed to the Consul Server cluster. Because Consul has knowledge of all the Vault nodes (and in some cases, of other services as well), this setup allows the EC2 Instance to use Consul's DNS server for service discovery, and thereby to discover the IP addresses @@ -194,6 +195,7 @@ vault login -method=aws header_value=vault.service.consul role=vault-role-name [create_role]: https://www.vaultproject.io/api/auth/aws/index.html#create-role [dnsmasq_module]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq [dnsmasq]: http://www.thekelleys.org.uk/dnsmasq/doc.html +[setup_systemd_resolved]: https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved [ec2_example]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-ec2-auth [examples_helper]: https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-examples-helper/vault-examples-helper.sh [iam_auth]: https://www.vaultproject.io/docs/auth/aws.html#iam-auth-method diff --git a/examples/vault-s3-backend/README.md b/examples/vault-s3-backend/README.md index 3e6b76f6..30d70648 100644 --- a/examples/vault-s3-backend/README.md +++ b/examples/vault-s3-backend/README.md @@ -1,27 +1,27 @@ -# Vault Cluster with S3 backend example +# Vault Cluster with S3 backend example -This folder shows an example of Terraform code to deploy a [Vault](https://www.vaultproject.io/) cluster in +This folder shows an example of Terraform code to deploy a [Vault](https://www.vaultproject.io/) cluster in [AWS](https://aws.amazon.com/) using the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). The Vault cluster uses [Consul](https://www.consul.io/) as a high-availability storage backend and [S3](https://aws.amazon.com/s3/) for durable storage, so this example also deploys a separate Consul server cluster using the [consul-cluster module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/consul-cluster) from the Consul AWS Module. This example creates a Vault cluster spread across the subnets in the default VPC of the AWS account. Each of the -servers in this example has [Dnsmasq](http://www.thekelleys.org.uk/dnsmasq/doc.html) installed (via the -[install-dnsmasq module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq)) -which allows it to use the Consul server cluster for service discovery and thereby access Vault via DNS using the -domain name `vault.service.consul`. For an example of a Vault cluster +servers in this example has [Dnsmasq](http://www.thekelleys.org.uk/dnsmasq/doc.html) installed (via the +[install-dnsmasq module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq)) +or [setup-systemd-resolved](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved) +(in the case Ubuntu of 18.04) which allow it to use the Consul server cluster for service discovery and thereby access Vault via DNS using the domain name `vault.service.consul`. For an example of a Vault cluster that is publicly accessible, see [the root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example). ![Vault architecture](https://github.com/hashicorp/terraform-aws-vault/blob/master/_docs/architecture-with-s3.png?raw=true) -You will need to create an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) +You will need to create an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) that has Vault and Consul installed, which you can do using the [vault-consul-ami example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami)). For more info on how the Vault cluster works, check out the [vault-cluster](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster) documentation. -**Note**: To keep this example as simple to deploy and test as possible, it deploys the Vault cluster into your default -VPC and default subnets, some of which might be publicly accessible. This is OK for learning and experimenting, but for +**Note**: To keep this example as simple to deploy and test as possible, it deploys the Vault cluster into your default +VPC and default subnets, some of which might be publicly accessible. This is OK for learning and experimenting, but for production usage, we strongly recommend deploying the Vault cluster into the private subnets of a custom VPC. @@ -46,5 +46,5 @@ To deploy a Vault Cluster: print out the IP addresses of the Vault servers and some example commands you can run to interact with the cluster: `../vault-examples-helper/vault-examples-helper.sh`. -To see how to connect to the Vault cluster, initialize it, and start reading and writing secrets, head over to the +To see how to connect to the Vault cluster, initialize it, and start reading and writing secrets, head over to the [How do you use the Vault cluster?](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster#how-do-you-use-the-vault-cluster) docs. diff --git a/modules/install-vault/README.md b/modules/install-vault/README.md index 4c2020ab..6bb7538a 100644 --- a/modules/install-vault/README.md +++ b/modules/install-vault/README.md @@ -1,13 +1,14 @@ # Vault Install Script This folder contains a script for installing Vault and its dependencies. You can use this script, along with the -[run-vault script](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/run-vault) it installs, to create a Vault [Amazon Machine Image -(AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) that can be deployed in +[run-vault script](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/run-vault) it installs, to create a Vault [Amazon Machine Image +(AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) that can be deployed in [AWS](https://aws.amazon.com/) across an Auto Scaling Group using the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). This script has been tested on the following operating systems: * Ubuntu 16.04 +* Ubuntu 18.04 * Amazon Linux 2 There is a good chance it will work on other flavors of Debian, CentOS, and RHEL as well. @@ -16,7 +17,7 @@ There is a good chance it will work on other flavors of Debian, CentOS, and RHEL ## Quick start -To install Vault, use `git` to clone this repository at a specific tag (see the [releases page](../../../../releases) +To install Vault, use `git` to clone this repository at a specific tag (see the [releases page](../../../../releases) for all available tags) and run the `install-vault` script: ``` @@ -28,10 +29,10 @@ The `install-vault` script will install Vault, its dependencies, and the [run-va You can then run the `run-vault` script when the server is booting to start Vault. We recommend running the `install-vault` script as part of a [Packer](https://www.packer.io/) template to create a -Vault [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) (see the -[vault-consul-ami example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami) for sample code). You can then deploy the AMI across an Auto -Scaling Group using the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster) (see the -[root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) and [vault-cluster-private](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) +Vault [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) (see the +[vault-consul-ami example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami) for sample code). You can then deploy the AMI across an Auto +Scaling Group using the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster) (see the +[root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) and [vault-cluster-private](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) examples for fully-working sample code). @@ -101,7 +102,7 @@ After the `install-vault` script finishes running, you may wish to do the follow `/opt/vault/config`). 1. If `/usr/local/bin` isn't already part of `PATH`, you should add it so you can run the `vault` command without specifying the full path. - + ## Dependencies @@ -114,7 +115,7 @@ using `journalctl`. ## Why use Git to install this code? -We needed an easy way to install these scripts that satisfied a number of requirements, including working on a variety +We needed an easy way to install these scripts that satisfied a number of requirements, including working on a variety of operating systems and supported versioning. Our current solution is to use `git`, but this may change in the future. -See [Package Managers](https://github.com/hashicorp/terraform-aws-consul/blob/master/_docs/package-managers.md) for +See [Package Managers](https://github.com/hashicorp/terraform-aws-consul/blob/master/_docs/package-managers.md) for a full discussion of the requirements, trade-offs, and why we picked `git`. diff --git a/modules/install-vault/install-vault b/modules/install-vault/install-vault index 7111a3e8..e1564049 100755 --- a/modules/install-vault/install-vault +++ b/modules/install-vault/install-vault @@ -3,7 +3,8 @@ # operating systems: # # 1. Ubuntu 16.04 -# 2. Amazon Linux 2 +# 2. Ubuntu 18.04 +# 3. Amazon Linux 2 set -e @@ -21,7 +22,7 @@ function print_usage { echo echo "Usage: install-vault [OPTIONS]" echo - echo "This script can be used to install Vault and its dependencies. This script has been tested with Ubuntu 16.04 and Amazon Linux 2." + echo "This script can be used to install Vault and its dependencies. This script has been tested with Ubuntu 16.04, Ubuntu 18.04 and Amazon Linux 2." echo echo "Options:" echo @@ -118,7 +119,7 @@ function install_dependencies { if $(has_apt_get); then sudo apt-get update -y - sudo apt-get install -y awscli curl unzip jq + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y awscli curl unzip jq libcap2-bin elif $(has_yum); then sudo yum update -y sudo yum install -y awscli curl unzip jq @@ -208,6 +209,7 @@ function install_binary { # For more info, see: https://www.vaultproject.io/docs/configuration/#disable_mlock function configure_mlock { echo "Giving Vault permission to use the mlock syscall" + # Requires installing libcap2-bin on Ubuntu 18 sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault)) } diff --git a/modules/run-vault/README.md b/modules/run-vault/README.md index f3e9af0d..356f18a9 100644 --- a/modules/run-vault/README.md +++ b/modules/run-vault/README.md @@ -4,6 +4,7 @@ This folder contains a script for configuring and running Vault on an [AWS](http script has been tested on the following operating systems: * Ubuntu 16.04 +* Ubuntu 18.04 * Amazon Linux 2 There is a good chance it will work on other flavors of Debian, CentOS, and RHEL as well. @@ -41,7 +42,7 @@ will automatically restart Vault if it crashes or the EC2 instance reboots. Note that `systemd` logs to its own journal by default. To view the Vault logs, run `journalctl -u vault.service`. To change the log output location, you can specify the `StandardOutput` and `StandardError` options by using the `--systemd-stdout` and `--systemd-stderr` options. See the [`systemd.exec` man pages](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardOutput=) for available -options, but note that the `file:path` option requires [systemd version >= 236](https://stackoverflow.com/a/48052152), which is not provided +options, but note that the `file:path` option requires [systemd version >= 236](https://stackoverflow.com/a/48052152), which is not provided in the base Ubuntu 16.04 and Amazon Linux 2 images. See the [root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) and @@ -206,5 +207,3 @@ HTTP. However, you can configure those agents to talk to the Consul servers usin encryption docs](https://www.consul.io/docs/agent/encryption.html) and the Consul AWS Module [How do you handle encryption docs](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/run-consul#how-do-you-handle-encryption) for more info. - - diff --git a/modules/update-certificate-store/README.md b/modules/update-certificate-store/README.md index 52a32c0c..f2ee1836 100644 --- a/modules/update-certificate-store/README.md +++ b/modules/update-certificate-store/README.md @@ -1,15 +1,16 @@ # Update Certificate Store -This folder contains a script for adding a trusted, Certificate Authority (CA) certificate to an OS's certificate -store. This allows you to establish TLS connections to services that use TLS certs signed by that CA without getting +This folder contains a script for adding a trusted, Certificate Authority (CA) certificate to an OS's certificate +store. This allows you to establish TLS connections to services that use TLS certs signed by that CA without getting x509 certificate errors. This script has been tested on the following operating systems: * Ubuntu 16.04 +* Ubuntu 18.04 * Amazon Linux 2 There is a good chance it will work on other flavors of Debian, CentOS, and RHEL as well. -If you're unfamiliar with how TLS certificates work, check out the [Background +If you're unfamiliar with how TLS certificates work, check out the [Background section](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/private-tls-cert#background). @@ -18,7 +19,7 @@ section](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/pr ## Motivation -Let's say you deployed a server (e.g. a Vault server) with a self-signed TLS certificate. If you try to make a request +Let's say you deployed a server (e.g. a Vault server) with a self-signed TLS certificate. If you try to make a request to that server using some sort of TLS client (e.g. a Vault client), you will get an error: ``` @@ -47,7 +48,7 @@ to trust this certificate. ## Quick start -To use the `update-certificate-script`, use `git` to clone this repository at a specific tag (see the +To use the `update-certificate-script`, use `git` to clone this repository at a specific tag (see the [releases page](../../../../releases) for all available tags) and run the `update-certificate-script` script: ``` @@ -79,8 +80,8 @@ See the [vault-consul-ami example](https://github.com/hashicorp/terraform-aws-va The `run-vault` script accepts the following arguments: * `--cert-file-path` (required): The path to the CA certificate public key to add to the OS certificate store. -* `--dest-file-name` (optional): This script will copy `--cert-file-path` to a file with this name in a shared - certificate folder on the OS. The default file name is `custom.crt`, but you can use this parameter to customize +* `--dest-file-name` (optional): This script will copy `--cert-file-path` to a file with this name in a shared + certificate folder on the OS. The default file name is `custom.crt`, but you can use this parameter to customize it. The extension MUST be `.crt` or the OS will ignore the file. Example: @@ -88,4 +89,3 @@ Example: ``` terraform-aws-vault/modules/update-certificate-script/update-certificate-script --cert-file-path /opt/vault/tls/ca.cert.pem ``` - diff --git a/modules/update-certificate-store/update-certificate-store b/modules/update-certificate-store/update-certificate-store index a803b824..e07d9a1d 100755 --- a/modules/update-certificate-store/update-certificate-store +++ b/modules/update-certificate-store/update-certificate-store @@ -4,7 +4,8 @@ # This script has been tested with the following operating systems: # # 1. Ubuntu 16.04 -# 2. Amazon Linux 2 +# 2. Ubuntu 18.04 +# 3. Amazon Linux 2 set -e diff --git a/modules/vault-cluster/README.md b/modules/vault-cluster/README.md index f435c932..ff832488 100644 --- a/modules/vault-cluster/README.md +++ b/modules/vault-cluster/README.md @@ -1,8 +1,8 @@ # Vault Cluster -This folder contains a [Terraform](https://www.terraform.io/) module that can be used to deploy a -[Vault](https://www.vaultproject.io/) cluster in [AWS](https://aws.amazon.com/) on top of an Auto Scaling Group. This -module is designed to deploy an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) +This folder contains a [Terraform](https://www.terraform.io/) module that can be used to deploy a +[Vault](https://www.vaultproject.io/) cluster in [AWS](https://aws.amazon.com/) on top of an Auto Scaling Group. This +module is designed to deploy an [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) that had Vault installed via the [install-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault) module in this Module. @@ -20,8 +20,8 @@ module "vault_cluster" { # Specify the ID of the Vault AMI. You should build this using the scripts in the install-vault module. ami_id = "ami-abcd1234" - - # Configure and start Vault during boot. + + # Configure and start Vault during boot. user_data = <<-EOF #!/bin/bash /opt/vault/bin/run-vault --tls-cert-file /opt/vault/tls/vault.crt.pem --tls-key-file /opt/vault/tls/vault.key.pem @@ -43,32 +43,32 @@ module "vault_cluster" { propagate_at_launch = true } ] - + # ... See vars.tf for the other parameters you must define for the vault-cluster module } ``` Note the following parameters: -* `source`: Use this parameter to specify the URL of the vault-cluster module. The double slash (`//`) is intentional - and required. Terraform uses it to specify subfolders within a Git repo (see [module - sources](https://www.terraform.io/docs/modules/sources.html)). The `ref` parameter specifies a specific Git tag in - this repo. That way, instead of using the latest version of this module from the `master` branch, which +* `source`: Use this parameter to specify the URL of the vault-cluster module. The double slash (`//`) is intentional + and required. Terraform uses it to specify subfolders within a Git repo (see [module + sources](https://www.terraform.io/docs/modules/sources.html)). The `ref` parameter specifies a specific Git tag in + this repo. That way, instead of using the latest version of this module from the `master` branch, which will change every time you run Terraform, you're using a fixed version of the repo. -* `ami_id`: Use this parameter to specify the ID of a Vault [Amazon Machine Image +* `ami_id`: Use this parameter to specify the ID of a Vault [Amazon Machine Image (AMI)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) to deploy on each server in the cluster. You should install Vault in this AMI using the scripts in the [install-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault) module. - -* `user_data`: Use this parameter to specify a [User + +* `user_data`: Use this parameter to specify a [User Data](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts) script that each - server will run during boot. This is where you can use the [run-vault script](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/run-vault) to configure and - run Vault. The `run-vault` script is one of the scripts installed by the [install-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault) - module. + server will run during boot. This is where you can use the [run-vault script](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/run-vault) to configure and + run Vault. The `run-vault` script is one of the scripts installed by the [install-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault) + module. You can find the other parameters in [vars.tf](vars.tf). -Check out the [root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) and +Check out the [root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) and [vault-cluster-private](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) examples for working sample code. @@ -78,9 +78,9 @@ Check out the [root example](https://github.com/hashicorp/terraform-aws-vault/tr ## How do you use the Vault cluster? To use the Vault cluster, you will typically need to SSH to each of the Vault servers. If you deployed the -[vault-cluster-private](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) or [the root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) -examples, the [vault-examples-helper.sh script](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-examples-helper/vault-examples-helper.sh) will do the -tag lookup for you automatically (note, you must have the [AWS CLI](https://aws.amazon.com/cli/) and +[vault-cluster-private](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) or [the root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) +examples, the [vault-examples-helper.sh script](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-examples-helper/vault-examples-helper.sh) will do the +tag lookup for you automatically (note, you must have the [AWS CLI](https://aws.amazon.com/cli/) and [jq](https://stedolan.github.io/jq/) installed locally): ``` @@ -95,8 +95,8 @@ Your Vault servers are running at the following IP addresses: ### Initializing the Vault cluster -The very first time you deploy a new Vault cluster, you need to [initialize the -Vault](https://www.vaultproject.io/intro/getting-started/deploy.html#initializing-the-vault). The easiest way to do +The very first time you deploy a new Vault cluster, you need to [initialize the +Vault](https://www.vaultproject.io/intro/getting-started/deploy.html#initializing-the-vault). The easiest way to do this is to SSH to one of the servers that has Vault installed and run: ``` @@ -112,11 +112,11 @@ Initial Root Token: eaf5cc32-b48f-7785-5c94-90b5ce300e9b Vault initialized with 5 keys and a key threshold of 3! ``` -Vault will print out the [unseal keys](https://www.vaultproject.io/docs/concepts/seal.html) and a [root -token](https://www.vaultproject.io/docs/concepts/tokens.html#root-tokens). This is the **only time ever** that all of -this data is known by Vault, so you **MUST** save it in a secure place immediately! Also, this is the only time that +Vault will print out the [unseal keys](https://www.vaultproject.io/docs/concepts/seal.html) and a [root +token](https://www.vaultproject.io/docs/concepts/tokens.html#root-tokens). This is the **only time ever** that all of +this data is known by Vault, so you **MUST** save it in a secure place immediately! Also, this is the only time that the unseal keys should ever be so close together. You should distribute each one to a different, trusted administrator -for safe keeping in completely separate secret stores and NEVER store them all in the same place. +for safe keeping in completely separate secret stores and NEVER store them all in the same place. In fact, a better option is to initialize Vault with [PGP, GPG, or Keybase](https://www.vaultproject.io/docs/concepts/pgp-gpg-keybase.html) so that each unseal key is encrypted with a @@ -137,7 +137,7 @@ See [Using PGP, GPG, and Keybase](https://www.vaultproject.io/docs/concepts/pgp- ### Unsealing the Vault cluster -Now that you have the unseal keys, you can [unseal Vault](https://www.vaultproject.io/docs/concepts/seal.html) by +Now that you have the unseal keys, you can [unseal Vault](https://www.vaultproject.io/docs/concepts/seal.html) by having 3 out of the 5 administrators (or whatever your key shard threshold is) do the following: 1. SSH to a Vault server. @@ -170,7 +170,7 @@ There are three ways to connect to Vault: #### Access Vault from a Vault server -When you SSH to a Vault server, the Vault client is already configured to talk to the Vault server on localhost, so +When you SSH to a Vault server, the Vault client is already configured to talk to the Vault server on localhost, so you can directly run Vault commands: ``` @@ -185,15 +185,15 @@ value bar #### Access Vault from other servers in the same AWS account -To access Vault from a different server in the same account, you need to specify the URL of the Vault cluster. You -could manually look up the Vault cluster's IP address, but since this module uses Consul not only as a [storage -backend](https://www.vaultproject.io/docs/configuration/storage/consul.html) but also as a way to register [DNS -entries](https://www.consul.io/docs/guides/forwarding.html), you can access Vault +To access Vault from a different server in the same account, you need to specify the URL of the Vault cluster. You +could manually look up the Vault cluster's IP address, but since this module uses Consul not only as a [storage +backend](https://www.vaultproject.io/docs/configuration/storage/consul.html) but also as a way to register [DNS +entries](https://www.consul.io/docs/guides/forwarding.html), you can access Vault using a nice domain name instead, such as `vault.service.consul`. -To set this up, use the [install-dnsmasq -module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq) on each server that -needs to access Vault. This allows you to access Vault from your EC2 Instances as follows: +To set this up, use the [install-dnsmasq +module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq) on each server that +needs to access Vault or [setup-systemd-resolved](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved) if using Ubuntu 18.04. This allows you to access Vault from your EC2 Instances as follows: ``` vault -address=https://vault.service.consul:8200 read secret/foo @@ -221,10 +221,10 @@ refresh_interval 768h0m0s value bar ``` -Note that if you're using a self-signed TLS cert (e.g. generated from the [private-tls-cert -module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/private-tls-cert)), you'll need to have the public key of the CA that signed that cert or you'll get +Note that if you're using a self-signed TLS cert (e.g. generated from the [private-tls-cert +module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/private-tls-cert)), you'll need to have the public key of the CA that signed that cert or you'll get an "x509: certificate signed by unknown authority" error. You could pass the certificate manually: - + ``` vault read -ca-cert=/opt/vault/tls/ca.crt.pem secret/foo @@ -234,7 +234,7 @@ refresh_interval 768h0m0s value bar ``` -However, to avoid having to add the `-ca-cert` argument to every single call, you can use the [update-certificate-store +However, to avoid having to add the `-ca-cert` argument to every single call, you can use the [update-certificate-store module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/update-certificate-store) to configure the server to trust the CA. Check out the [vault-cluster-private example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) for working sample code. @@ -242,12 +242,12 @@ Check out the [vault-cluster-private example](https://github.com/hashicorp/terra #### Access Vault from the public Internet -We **strongly** recommend only running Vault in private subnets. That means it is not directly accessible from the -public Internet, which reduces your surface area to attackers. If you need users to be able to access Vault from -outside of AWS, we recommend using VPN to connect to AWS. - -If VPN is not an option, and Vault must be accessible from the public Internet, you can use the [vault-elb -module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-elb) to deploy an [Elastic Load Balancer +We **strongly** recommend only running Vault in private subnets. That means it is not directly accessible from the +public Internet, which reduces your surface area to attackers. If you need users to be able to access Vault from +outside of AWS, we recommend using VPN to connect to AWS. + +If VPN is not an option, and Vault must be accessible from the public Internet, you can use the [vault-elb +module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-elb) to deploy an [Elastic Load Balancer (ELB)](https://aws.amazon.com/elasticloadbalancing/classicloadbalancer/) in your public subnets, and have all your users access Vault via this ELB: @@ -255,7 +255,7 @@ access Vault via this ELB: vault -address=https:// read secret/foo ``` -Where `ELB_DNS_NAME` is the DNS name for your ELB, such as `vault.example.com`. You can configure the Vault address as +Where `ELB_DNS_NAME` is the DNS name for your ELB, such as `vault.example.com`. You can configure the Vault address as an environment variable: ``` @@ -290,7 +290,7 @@ This architecture consists of the following resources: ### Auto Scaling Group This module runs Vault on top of an [Auto Scaling Group (ASG)](https://aws.amazon.com/autoscaling/). Typically, you -should run the ASG with 3 or 5 EC2 Instances spread across multiple [Availability +should run the ASG with 3 or 5 EC2 Instances spread across multiple [Availability Zones](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). Each of the EC2 Instances should be running an AMI that has had Vault installed via the [install-vault](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/install-vault) module. You pass in the ID of the AMI to run using the `ami_id` input parameter. @@ -299,21 +299,21 @@ module. You pass in the ID of the AMI to run using the `ami_id` input parameter. ### Security Group Each EC2 Instance in the ASG has a Security Group that allows: - + * All outbound requests * Inbound requests on Vault's API port (default: port 8200) * Inbound requests on Vault's cluster port for server-to-server communication (default: port 8201) * Inbound SSH requests (default: port 22) -The Security Group ID is exported as an output variable if you need to add additional rules. +The Security Group ID is exported as an output variable if you need to add additional rules. -Check out the [Security section](#security) for more details. +Check out the [Security section](#security) for more details. ### IAM Role and Permissions -Each EC2 Instance in the ASG has an [IAM Role](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) attached. -The IAM Role ARN is exported as an output variable so you can add custom permissions. +Each EC2 Instance in the ASG has an [IAM Role](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) attached. +The IAM Role ARN is exported as an output variable so you can add custom permissions. ### S3 bucket (Optional) @@ -327,8 +327,8 @@ a separate Consul server cluster to be deployed as a high availability backend. ## How do you roll out updates? -Please note that Vault does not support true zero-downtime upgrades, but with proper upgrade procedure the downtime -should be very short (a few hundred milliseconds to a second depending on how the speed of access to the storage +Please note that Vault does not support true zero-downtime upgrades, but with proper upgrade procedure the downtime +should be very short (a few hundred milliseconds to a second depending on how the speed of access to the storage backend). See the [Vault upgrade guide instructions](https://www.vaultproject.io/docs/guides/upgrading/index.html) for details. @@ -375,10 +375,10 @@ upgraded: ## What happens if a node crashes? There are two ways a Vault node may go down: - + 1. The Vault process may crash. In that case, `systemd` should restart it automatically. At this point, you will need to have each Vault admin SSH to the Instance to unseal it again. -1. The EC2 Instance running Vault dies. In that case, the Auto Scaling Group should launch a replacement automatically. +1. The EC2 Instance running Vault dies. In that case, the Auto Scaling Group should launch a replacement automatically. Once again, the Vault admins will have to SSH to the replacement Instance and unseal it. Given the need for manual intervention, you will want to have alarms set up that go off any time a Vault node gets @@ -408,38 +408,38 @@ Vault uses TLS to encrypt its network traffic. For instructions on configuring T Vault servers keep everything in memory and does not write any data to the local hard disk. To persist data, Vault encrypts it, and sends it off to its storage backends, so no matter how the backend stores that data, it is already -encrypted. By default, this Module uses Consul as a storage backend, so if you want an additional layer of -protection, you can check out the [official Consul encryption docs](https://www.consul.io/docs/agent/encryption.html) -and the Consul AWS Module [How do you handle encryption +encrypted. By default, this Module uses Consul as a storage backend, so if you want an additional layer of +protection, you can check out the [official Consul encryption docs](https://www.consul.io/docs/agent/encryption.html) +and the Consul AWS Module [How do you handle encryption docs](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/run-consul#how-do-you-handle-encryption) for more info. -Note that if you want to enable encryption for the root EBS Volume for your Vault Instances (despite the fact that -Vault itself doesn't write anything to this volume), you need to enable that in your AMI. If you're creating the AMI -using Packer (e.g. as shown in the [vault-consul-ami example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami)), you need to set the [encrypt_boot +Note that if you want to enable encryption for the root EBS Volume for your Vault Instances (despite the fact that +Vault itself doesn't write anything to this volume), you need to enable that in your AMI. If you're creating the AMI +using Packer (e.g. as shown in the [vault-consul-ami example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-consul-ami)), you need to set the [encrypt_boot parameter](https://www.packer.io/docs/builders/amazon-ebs.html#encrypt_boot) to `true`. ### Dedicated instances -If you wish to use dedicated instances, you can set the `tenancy` parameter to `"dedicated"` in this module. +If you wish to use dedicated instances, you can set the `tenancy` parameter to `"dedicated"` in this module. ### Security groups This module attaches a security group to each EC2 Instance that allows inbound requests as follows: -* **Vault**: For the Vault API port (default: 8200), you can use the `allowed_inbound_cidr_blocks` parameter to control +* **Vault**: For the Vault API port (default: 8200), you can use the `allowed_inbound_cidr_blocks` parameter to control the list of [CIDR blocks](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) that will be allowed access and the `allowed_inbound_security_group_ids` parameter to control the security groups that will be allowed access. * **SSH**: For the SSH port (default: 22), you can use the `allowed_ssh_cidr_blocks` parameter to control the list of [CIDR blocks](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) that will be allowed access. You can use the `allowed_ssh_security_group_ids` parameter to control the list of source Security Groups that will be allowed access. - + Note that all the ports mentioned above are configurable via the `xxx_port` variables (e.g. `api_port`). See [vars.tf](vars.tf) for the full list. - - + + ### SSH access @@ -472,18 +472,18 @@ same cluster because: set in memory. That means for every 1 byte of data in Vault, you'd also have 1 byte of data in Consul, doubling your memory consumption on each server. -Check out the [Consul AWS Module](https://github.com/hashicorp/terraform-aws-consul) for how to deploy a Consul -server cluster in AWS. See the [root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) and +Check out the [Consul AWS Module](https://github.com/hashicorp/terraform-aws-consul) for how to deploy a Consul +server cluster in AWS. See the [root example](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/root-example) and [vault-cluster-private](https://github.com/hashicorp/terraform-aws-vault/tree/master/examples/vault-cluster-private) examples for sample code that shows how to run both a Vault server cluster and Consul server cluster. ### Monitoring, alerting, log aggregation -This module does not include anything for monitoring, alerting, or log aggregation. All ASGs and EC2 Instances come -with limited [CloudWatch](https://aws.amazon.com/cloudwatch/) metrics built-in, but beyond that, you will have to -provide your own solutions. We especially recommend looking into Vault's [Audit -backends](https://www.vaultproject.io/docs/audit/index.html) for how you can capture detailed logging and audit +This module does not include anything for monitoring, alerting, or log aggregation. All ASGs and EC2 Instances come +with limited [CloudWatch](https://aws.amazon.com/cloudwatch/) metrics built-in, but beyond that, you will have to +provide your own solutions. We especially recommend looking into Vault's [Audit +backends](https://www.vaultproject.io/docs/audit/index.html) for how you can capture detailed logging and audit information. Given that any time Vault crashes, reboots, or restarts, you have to have the Vault admins manually unseal it (see @@ -493,7 +493,6 @@ notify these admins whenever they need to take action! ### VPCs, subnets, route tables -This module assumes you've already created your network topology (VPC, subnets, route tables, etc). You will need to -pass in the the relevant info about your network topology (e.g. `vpc_id`, `subnet_ids`) as input variables to this +This module assumes you've already created your network topology (VPC, subnets, route tables, etc). You will need to +pass in the the relevant info about your network topology (e.g. `vpc_id`, `subnet_ids`) as input variables to this module. - diff --git a/modules/vault-elb/README.md b/modules/vault-elb/README.md index 70d2f73e..09407aef 100644 --- a/modules/vault-elb/README.md +++ b/modules/vault-elb/README.md @@ -5,7 +5,8 @@ Balancer (ELB)](https://aws.amazon.com/elasticloadbalancing/classicloadbalancer/ from the [vault-cluster module](https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/vault-cluster). This is useful if you need to access Vault from the public Internet. Note that for most users, we recommend NOT making Vault accessible from the public Internet and using DNS to access your Vault cluster instead (see the [install-dnsmasq -module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq) for details). +module](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/install-dnsmasq) or [setup-systemd-resolved](https://github.com/hashicorp/terraform-aws-consul/tree/master/modules/setup-systemd-resolved) +in the case of Ubuntu 18.04 for details). diff --git a/test/vault_main_test.go b/test/vault_main_test.go index e694e8c2..905a37d2 100644 --- a/test/vault_main_test.go +++ b/test/vault_main_test.go @@ -25,10 +25,12 @@ type amiData struct { } var amisData = []amiData{ - {"vaultEnterpriseUbuntu", "ubuntu16-ami", "ubuntu", true}, - {"vaultEnterpriseAmazonLinux", "amazon-linux-2-ami", "ec2-user", true}, - {"vaultUbuntu", "ubuntu16-ami", "ubuntu", false}, - {"vaultAmazonLinux", "amazon-linux-2-ami", "ec2-user", false}, + {"vaultEnterpriseOnUbuntu18", "ubuntu18-ami", "ubuntu", true}, + {"vaultEnterpriseOnUbuntu16", "ubuntu16-ami", "ubuntu", true}, + {"vaultEnterpriseOnAmazonLinux", "amazon-linux-2-ami", "ec2-user", true}, + {"vaultOpenSourceOnUbuntu18", "ubuntu18-ami", "ubuntu", false}, + {"vaultOpenSourceOnUbuntu16", "ubuntu16-ami", "ubuntu", false}, + {"vaultOpenSourceOnAmazonLinux", "amazon-linux-2-ami", "ec2-user", false}, } var testCases = []testCase{