Initial work to do reliable backup of SSM parameters

.gitignore

@@ -0,0 +1,35 @@
+*.retry
+*~
+*venv
+aws_credentials*
+*password*
+\#*
+*environment
+*postfix
+.#*
+*.dat
+*.pub
+gpgtemp
+keys/
+build
+*.log
+dump.sql*
+*.pri
+mockstmp
+.anslk_*
+.prepare*
+test-config*
+__pycache__
+.mypy_cache
+*.gpg
+bin/backup_encrypt
+*.pyc
+dist
+MANIFEST
+deploy_key
+deploy_key.bck
+encrypted_build_files
+encrypted_build_files.tjz
+*.egg-info
+*.makestamp
+.hypothesis

.pre-commit-config.yaml

@@ -0,0 +1,30 @@
+exclude: '^$'
+fail_fast: false
+repos:
+-   repo: https://github.com/ambv/black
+    rev: stable
+    hooks:
+    - id: black
+      language_version: python3.7
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v1.2.3
+    hooks:
+    - id: check-added-large-files
+    - id: check-json
+    - id: detect-private-key
+    - id: end-of-file-fixer
+    - id: flake8
+      exclude: '^features/.*_steps/*'
+      args:
+        - --ignore=W503,E402,E501
+        - --max-line-length=131
+    - id: forbid-new-submodules
+    - id: check-yaml
+- repo: https://github.com/pre-commit/mirrors-mypy
+  rev: 'master'  # Use the sha / tag you want to point at
+  hooks:
+  - id: mypy
+# -   repo: https://github.com/Lucas-C/pre-commit-hooks-go
+#     sha: v1.0.0
+#     hooks:
+#     -   id: checkmake

Makefile

@@ -0,0 +1,40 @@
+AWS_ACCOUNT_NAME ?= michael
+AWS_DEFAULT_REGION ?= eu-west-1
+PYTHON ?= python3
+BEHAVE ?= behave
+
+LIBFILES := $(shell find backup_cloud_ssm -name '*.py')
+
+all: lint test
+
+# pytest-mocked is much faster than non-mocked which is slower even than
+# the functional tests so run it first, then behave then finally the
+# full pytest tests so that failures are detected early where possible.
+test: develop pytest-mocked behave pytest 
+
+behave:
+	behave --tags ~@future
+
+pytest-mocked:
+	MOCK_AWS=true pytest
+
+pytest:
+	pytest
+
+wip: develop
+	$(BEHAVE) --wip
+
+lint:
+	pre-commit install --install-hooks
+	pre-commit run -a
+
+
+# develop is needed to install scripts that are called during testing 
+develop: .develop.makestamp
+
+.develop.makestamp: setup.py backup_cloud_ssm/aws_ssm_cli.py $(LIBFILES)
+	$(PYTHON) setup.py install --force
+	$(PYTHON) setup.py develop
+	touch $@
+
+.PHONY: all test behave pytest-mocked pytest wip lint develop

README.md

@@ -0,0 +1,81 @@
+Backup SSM parameter store to a file.  Optional (but default)
+encryption to be added.
+
+## Using CLI tools
+
+The CLI tool provides a simple interface to dump or restore the full
+set of SSM parameters.
+
+1) set up the appropriate environment including AWS variables
+
+    export AWS_REGION=us-west-2
+    export AWS_ACCESS_KEY_ID=AKIABCDEFGHIJKLMNOPQ
+    export AWS_SECRET_ACCESS_KEY=1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZABCD
+
+2) to backup run
+
+   aws-ssm-backup > `<filename>`
+
+3) to restore run
+
+   aws-ssm-backup --restore > `<filename>`
+
+Special notes:
+
+1) the tool does not overwrite - if you want to replace an existing
+parameter, simply manually delete it and run again.
+
+2) ssm seems to be eventually consistent - you will not want to update
+SSM shortly before doing a backup.  You may want to wait a second or
+so after restoring.
+
+## Using python interface
+
+The backup and restore functions are provided as a libray.  For backup
+
+    import backup_aws_ssm
+    backup_aws_ssm.backup_to_file("myfile")
+
+for restore:
+
+    import backup_aws_ssm
+    backup_aws_ssm.restore_from_file("myfile")
+
+set the appropriate AWS variables to configure the aws region where
+this will work.
+
+## Using python ssm library
+
+Included in the package is a library which provides a dict object
+which accesses SSM parameter store.  This will likely, later, be split out into a separate packge.  In the meantime it can be used in Alpha testing mode.
+
+      from backup_cloud_ssm.aws_ssm_dict import aws_ssm_dict
+      ssm_dict = aws_ssm_dict()
+      ssm_dict["parameter"] = "value"
+      print(ssm_dict["parameter"])
+
+SSM parameter store treats storing no description and storing the
+empty description ("") as the same thing and will not return any
+description.  For simplicity we have now chosen to represent this as
+the empty string.  This decision may change in future and feedback is
+appreciated.
+
+When parameters are deleted the parameter description sometimes seems
+to persist for some time, possibly only when it was '0'.  Do not rely
+on the description to be empty or see testing/test_parameter_storage
+for how to handle this.
+
+
+## Development
+
+We aim to use Behavior Driven Development to encourage reasonable feature descriptions and a level of tests appropriate for the business functionality included here.  Test Driven Development and to some extent Test Driven Design are encouraged in order to improve testability and eas of modification of the code.
+
+Some of the tests are designed to run against either the Moto library or a real AWS instance.  By defining the shell variable MOCK_AWS as "true" all of the tests which can be run in mocked form will be.  
+
+    export MOCK_AWS=true
+
+This considerably speeds up testing but slightly increases risk since Moto's model of SSM is missing a number of features.  
+
+## Defined functionality
+
+See the features directory for the supported features of the software.  This is considered part of the documentation. 

backup_cloud_ssm/__init__.py

@@ -0,0 +1,4 @@
+from backup_cloud_ssm.backup_aws_ssm import (  # noqa: F401
+    backup_to_file,
+    restore_from_file,
+)

backup_cloud_ssm/aws_ssm_cli.py

@@ -0,0 +1,17 @@
+import argparse
+import sys
+import backup_cloud_ssm
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Backup AWS SSM Parameter Store")
+    parser.add_argument("--restore", help="restore from stdin", action="store_true")
+    args = parser.parse_args()
+    if args.restore:
+        backup_cloud_ssm.restore_from_file(sys.stdin)
+    else:
+        backup_cloud_ssm.backup_to_file(sys.stdout)
+
+
+if __name__ == "__main__":
+    main()