add Recursive DNS Forwarder module and example vpc environment code f…

…or all three RDNS resolution options (inert by default), tweak documentation

README.md

@@ -4,7 +4,7 @@ This infrastructure-as-code (IaC) repository is intended to help you efficiently
 
 There is no one-size-fits-all blueprint for an entire VPC; while they all generally have the same building blocks, the details can vary widely depending on your individual needs.  To that end, this repository provides:
 
-  1. several reusable Terraform modules (under `modules/`) to construct the various individual components that make up an Enterprise VPC, abstracting away internal details where possible
+  1. a collection of reusable Terraform modules (under `modules/`) to construct the various individual components that make up an Enterprise VPC, abstracting away internal details where possible
 
   2. a set of example IaC environments for shared networking resources (`global/` and `vpc/`) which combine those modules and a few primitives together into a fully-functional Enterprise VPC
 
@@ -27,36 +27,40 @@ You will need:
 
   * an AWS account
 
-  * an official name (e.g. 'aws-foobar-vpc') and IPv4 allocation (e.g. 10.x.y.0/24) for your Enterprise VPC
+  * an official name (e.g. "aws-foobar-vpc") and IPv4 allocation (e.g. 10.x.y.0/24) for your Enterprise VPC
 
   * an S3 bucket **with versioning enabled** for storing Terraform state, and a DynamoDB table for state locking (see also https://www.terraform.io/docs/backends/types/s3.html).  To create these resources:
 
-    1. Choose a [valid S3 bucket name](http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules).
+    1. Set up the AWS Command Line Interface on your workstation (see "Workstation Setup" further down).
+
+    2. Choose a [valid S3 bucket name](http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules).
 
        * S3 bucket names are _globally_ unique, so you must choose one that is not already in use by another AWS account. One possible strategy is to use the pattern
 
              bucket = "terraform.uiuc-tech-services-sandbox.aws.illinois.edu"
 
          replacing 'uiuc-tech-services-sandbox' with the friendly name of your AWS account.
 
-    2. Use AWS CLI to create the chosen bucket (replacing 'FIXME') and enable versioning:
+    3. Use AWS CLI to create the chosen bucket (replacing 'FIXME') and enable versioning:
 
            aws s3api create-bucket --create-bucket-configuration LocationConstraint=us-east-2 \
              --bucket FIXME
            aws s3api put-bucket-versioning --versioning-configuration Status=Enabled \
              --bucket FIXME
 
-    3. Use AWS CLI to create a DynamoDB table for state locking called 'terraform' (this name does _not_ need to be globally unique):
+    4. Use AWS CLI to create a DynamoDB table for state locking, called "terraform" (this name does _not_ need to be globally unique):
 
            aws dynamodb create-table --region us-east-2 --table-name terraform \
              --attribute-definitions AttributeName=LockID,AttributeType=S \
              --key-schema AttributeName=LockID,KeyType=HASH \
              --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1
 
-  * your own copy of the sample environment code, in your own source control repository, **customized** to reflect your AWS account and the specific subnets and other components you want your VPC to comprise.  (Note that you do _not_ need your own copy of the modules.)
+  * your own copy of the sample environment code, in your own source control repository, **customized** to reflect your AWS account and the specific subnets and other components you want your VPC to comprise.
 
     Download the [latest release of this repository](https://github.com/cites-illinois/aws-enterprise-vpc/releases/latest) to use as a starting point.
 
+    Note that you do _not_ need your own copy of the module code; the [module source paths](https://www.terraform.io/docs/modules/sources.html) specified in the example environments point directly to this online repository.
+
 **At minimum, you must edit the values marked with '#FIXME' comments in the following files**:
    * in `global/terraform.tfvars`:
      - account_id
@@ -96,7 +100,7 @@ To set up a new workstation:
          Default region name [None]: us-east-2
          Default output format [None]: json
 
-     If you do, set the `AWS_PROFILE` environment variable so that Terragrunt and Terraform (as well as the AWS CLI itself) will know which set of credentials to use:
+     If you do, set the `AWS_PROFILE` environment variable so that Terraform (as well as the AWS CLI itself) will know which set of credentials to use:
 
          export AWS_PROFILE=uiuc-tech-services-sandbox
 
@@ -115,7 +119,7 @@ To set up a new workstation:
        terraform apply
        cd ..
 
-   * As an optional feature, this environment automatically deploys the [AWS Solution for monitoring VPN Connections](https://aws.amazon.com/answers/networking/vpn-monitor/) and a Simple Notification Service topic which will be used later (by modules/vpn-connection) to create alarm notifications based on this monitoring.
+   * As an optional feature, the example global environment automatically deploys the [AWS Solution for monitoring VPN Connections](https://aws.amazon.com/answers/networking/vpn-monitor/) and creates a [Simple Notification Service](https://aws.amazon.com/sns/) topic which will be used later (by modules/vpn-connection) to create alarm notifications based on this monitoring.
 
      If you wish to receive these alarm notifications by email, use the AWS CLI to subscribe one or more email addresses to the SNS topic (indicated by the Terraform output "vpn_monitor_arn"):
 
@@ -139,7 +143,7 @@ To set up a new workstation:
 
    * Do you need a Core Services VPC peering, VPN connections, or both?
 
-   * Attach the `details.txt` file generated in the previous step.  This contains your AWS account number, your VPC's name, ID, and CIDR block, and additional configuration details (in XML format) for the on-campus side of each VPN connection.
+   * Attach the `details.txt` file generated in the previous step.  This contains your AWS account number, your VPC's name, ID, and CIDR block, and some additional configuration details (in XML format) for the on-campus side of each VPN connection.
 
 5. If you requested a Core Services VPC peering connection, Technology Services will initiate one and provide you with its ID.  Edit `vpc/terraform.tfvars` to add the new peering connection ID (enclosed in quotes), e.g.
 
@@ -152,6 +156,18 @@ To set up a new workstation:
        terraform apply
        cd ..
 
+6. By default, recursive DNS queries from instances within your VPC will be handled by [AmazonProvidedDNS](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_DHCP_Options.html#AmazonDNS).  If you wish to use one of the other options documented in [Amazon Web Services Recursive DNS Guide for Illinois](https://answers.uillinois.edu/illinois/page.php?id=74081),
+
+   * Edit `vpc/terraform.tfvars` to specify the IPv4 addresses of the Core Services Resolvers in the Core Services VPC with which your VPC has a peering connection, e.g.
+
+         core_services_resolvers = ["10.224.1.50", "10.224.1.100"]
+
+   * Edit `vpc/rdns.tf` and uncomment _only one section_, depending on which option you need.
+
+     _Note_: be sure to read and understand [`modules/rdns-forwarder/README.md`](modules/rdns-forwarder/README.md) before deploying Option 3.
+
+   * Deploy the `vpc` environment again (as above).
+
 
 ### Example Service
 
@@ -190,7 +206,7 @@ After your VPC is deployed, the next logical step is to write additional infrast
 
 If you need to create a second VPC in the same AWS account, just copy the `vpc/` environment directory (**excluding** the `vpc/.terraform/` subdirectory, if any) to e.g. `vpc2/` and modify the necessary values in the new files.
 
-Important: **don't forget to change `key`** in the backend configuration stanza of `vpc2/main.tf`
+**IMPORTANT**: **don't forget to change `key`** in the backend configuration stanza of `vpc2/main.tf`
 
     .
     ├── global/
@@ -214,6 +230,21 @@ Note that each AWS account will need to use a different S3 bucket for Terraform
 
 
 
+## Versioning
+-------------
+
+MAJOR.MINOR.PATCH versions of this repository are immutable releases tracked with git tags, e.g. `vX.Y.Z`.
+
+MAJOR.MINOR versions of this repository are tracked as git branches, e.g. `vX.Y`.  These are mutable, but only for non-breaking changes (once `vX.Y.0` has been released).
+
+All [module source paths](https://www.terraform.io/docs/modules/sources.html) used within the code specify a `vX.Y` branch.
+
+What this means (using hypothetical version numbers) is that if you base your own live IaC on the example environment code from release `v1.2.3`, and then re-run it in the future after a `terraform get -update`,
+* You will automatically receive any module changes released as `v1.2.4` (which should be safe), because they appear on the `v1.2` branch.
+* You will _not_ automatically receive any module changes released as `v1.3.*` or `v2.0.*` (which might be incompatible with your usage and/or involve refactoring that could cause Terraform to unexpectedly destroy and recreate existing resources).
+
+
+
 ## Known Issues
 ---------------
 
@@ -224,6 +255,3 @@ Note that each AWS account will need to use a different S3 bucket for Terraform
   The workaround is to manually remove the offending data source's existing state:
 
       terraform state rm module.public1-a-net.module.subnet.data.aws_vpc_peering_connection.pcx
-
-Wishlist:
-- include optional RDNS Forwarders (and DHCP options)

modules/rdns-forwarder/README.md

@@ -0,0 +1,121 @@
+# AWS Recursive DNS Forwarder
+
+This directory provides both a Terraform module and an ansible-pull playbook to launch and configure an EC2 instance which will serve as a [**Recursive DNS Forwarder**](https://answers.uillinois.edu/illinois/page.php?id=74081) for your [Enterprise VPC](https://answers.uillinois.edu/illinois/page.php?id=71015).
+
+RDNS Forwarders accept and answer recursive DNS queries _only_ from clients within your VPC.
+
+  * If the query is for a University domain, your RDNS Forwarder forwards it to the **Core Services Resolvers** located in your peer Core Services VPC.  These resolvers are able to resolve DNS records in zones which are restricted to University clients only.
+
+  * If the query is for any other domain, your RDNS Forwarder instead forwards it to [AmazonProvidedDNS](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_DHCP_Options.html#AmazonDNS).  AmazonProvidedDNS offers some special features whose behavior is specific to your VPC (and therefore cannot be implemented by the Core Services Resolvers).
+
+
+## Automated Updates
+
+RDNS Forwarders are designed to run completely unattended.  They use cron and [ansible-pull](http://docs.ansible.com/ansible/playbooks_intro.html#ansible-pull) to perform two distinct types of automated self-updates:
+
+  * Once per hour, the *zone configuration* (i.e. which individual zones' queries should be forwarded to the Core Services Resolvers as opposed to AmazonProvidedDNS) is updated to reflect the latest list of zones maintained by the [IP Address Management service](http://techservices.illinois.edu/services/ip-address-management), and `named` is instructed to reload the new configuration if it has changed.
+
+  * Once per month, a *full update* is performed based on the ansible code published in this git repository.  This includes a `yum -y update` to get the latest versions of all installed system packages; note that [Amazon Linux is maintained as a rolling release](https://aws.amazon.com/amazon-linux-ami/faqs/#updates_frequency), so the available package updates are not limited to a specific AMI version.
+
+    **The full update often involves a reboot, during which time the RDNS Forwarder will briefly stop answering queries.**
+
+To avoid impacting other resources in your VPC, please observe the following recommendations:
+
+  1. Deploy at least two RDNS Forwarders and configure them to perform their automated updates at different times.
+
+  2. Periodically test (from within your VPC) that each of your RDNS Forwarders can successfully answer queries for at least one University domain and at least one non-University domain.
+
+  3. If you ever need to destroy and recreate an RDNS Forwarder (e.g. to upgrade to a larger size instance, or to a new MAJOR.MINOR version branch of this repository),
+     * Take down only one RDNS Forwarder at a time.
+     * Test the other one first to make sure it is working as expected.
+     * Be sure the other one is not scheduled to perform its automated full update during your maintenance window.
+
+
+## Troubleshooting
+
+If an RDNS Forwarder stops working, destroy and recreate it from scratch.
+
+If a _newly created_ RDNS Forwarder (using the latest release of this repository) doesn't work, contact Technology Services for help.  Note that a newly created RDNS Forwarder may take up to 5 minutes to configure itself and begin answering queries.
+
+System logs are published in [CloudWatch Logs](http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/) to help with post-mortem analysis of problems.
+
+
+## How to Deploy
+
+The AWS Enterprise VPC Example environment code includes a working example of how to deploy RDNS Forwarders.  This section explains the module usage in greater detail.
+
+0. Make sure that:
+   * your Enterprise VPC has a VPC peering connection to a Core Services VPC
+   * you know the IPv4 addresses of the Core Services Resolvers within that particular Core Services VPC 
+
+1. Within your Enterprise VPC Shared Networking infrastructure-as-code (IaC), use this module to deploy at least two RDNS Forwarders (for redundancy).  We suggest placing them in different public-facing Subnets in different Availability Zones, with staggered update times.  For example:
+
+     ```hcl
+     module "rdns-a" {
+         source = "git::https://github.com/cites-illinois/aws-enterprise-vpc.git//modules/rdns-forwarder?ref=vX.Y" #FIXME
+         tags = {
+             Name = "rdns-a"
+         }
+         instance_type = "t2.micro"
+         core_services_resolvers = [ "10.224.1.50", "10.224.1.100" ] #FIXME
+         subnet_id = "${module.public1-a-net.id}"
+         private_ip = "192.168.0.5" #FIXME
+         zone_update_minute = "5"
+         full_update_day_of_month = "1"
+     }
+
+     module "rdns-b" {
+         source = "git::https://github.com/cites-illinois/aws-enterprise-vpc.git//modules/rdns-forwarder?ref=vX.Y" #FIXME
+         tags = {
+             Name = "rdns-b"
+         }
+         instance_type = "t2.micro"
+         core_services_resolvers = [ "10.224.1.50", "10.224.1.100" ] #FIXME
+         subnet_id = "${module.public1-b-net.id}"
+         private_ip = "192.168.1.5" #FIXME
+         zone_update_minute = "35"
+         full_update_day_of_month = "15"
+     }
+     ```
+
+   Notes:
+
+     * Do not set `full_update_day_of_month` higher than 28!
+
+     * You can also specify `full_update_hour` and `full_update_minute` if you want; the defaults correspond to 08:17 UTC.
+
+     * Using a public-facing subnet is simplest, but a campus-facing or private-facing subnet will also work as long as it has outbound Internet connectivity (via a NAT Gateway).  If you do use a campus-facing or private-facing subnet, you must also specify `associate_public_ip_address = false` in the module parameters.
+
+2. Deploy a custom [VPC DHCP Options Set](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_DHCP_Options.html) which instructs other instances in your VPC to send their DNS queries to the private IP addresses of your RDNS Forwarders, and associate that DHCP Options Set with the VPC.
+
+     ```hcl
+     resource "aws_vpc_dhcp_options" "dhcp_options" {
+         tags {
+             Name = "${var.vpc_short_name}-dhcp"
+         }
+         domain_name_servers = ["${module.rdns-a.private_ip}", "${module.rdns-b.private_ip}"]
+         domain_name = "${var.region}.compute.internal"
+     }
+
+     resource "aws_vpc_dhcp_options_association" "dhcp_assoc" {
+         vpc_id = "${aws_vpc.vpc.id}"
+         dhcp_options_id = "${aws_vpc_dhcp_options.dhcp_options.id}"
+     }
+     ```
+
+   Note:
+
+     * `domain_name` is not required, but makes your custom DHCP Options Set behave more like the default one.
+
+     * If your VPC already contains active clients, it's a good idea to manually test your new RDNS Forwarder instances _before_ enabling the custom DHCP Options Set.
+
+     * If you deploy RDNS Forwarders in your VPC and later decide to retire them, you will need to re-associate your VPC with the default DHCP Options Set (which directs clients to AmazonProvidedDNS).  After that, leave your RDNS Forwarder instances in place for a little while longer so they can continue to answer queries from running clients which have not yet picked up the new DHCP options.
+
+
+## Known Issues
+
+Wishlist:
+- external notifications (SNS/email) in case of trouble
+  - ansible failures
+  - dig @localhost tests
+- how to detect if RDNS Forwarder is oversubscribed (i.e. instance_type is too small)

modules/rdns-forwarder/local.yml

@@ -0,0 +1,9 @@
+# ansible-pull playbook to configure a Recursive DNS Forwarder
+#
+# Copyright (c) 2017 Board of Trustees University of Illinois
+---
+- name: Localhost
+  hosts: localhost
+  become: true
+  roles:
+    - rdns-forwarder