Releases: cloudfoundry/bosh-backup-and-restore
1.3.1
Features
- BBR multi-deployment backup log file names now contain timestamps
- BBR generates one log file per deployment
- Users will be able to retain logs even after running BBR multiple times
- BBR writes the log files in the directory passed in the
--backup-artifactflag
Improvements
- Bumped Golang dependency to the latest patch
1.3.0
New:
Introduces parallel multi-deployment backup capabilities
bbrwill apply the relevant action to all the BOSH deployments that can be accessed using the provided BOSH credentials- The following actions are supported:
$> bbr deployment --all-deployments pre-backup-check
$> bbr deployment --all-deployments backup
$> bbr deployment --all-deployments backup-cleanup
Improvement:
Improves support for tunnelling (BOSH_ALL_PROXY -- https://bosh.io/docs/cli-tunnel/) for the interaction between bbr and a BOSH director
- Adds tunnelling support for the
bbr directorcommands
1.2.8
Improvement:
bbr backup with BOSH_ALL_PROXY was still occasionally failing with connection reset by peer errors. We changed approach in how we handle SSH connection which should make bbr backup more stable when used with a SOCKS5 proxy. Pivotal Tracker Story and GitHub Issue #21
Improvement:
BBR commands now print the job name in drain logs:
OLD
[bbr] XXXX INFO - Copying backup -- 2.6M uncompressed -- from backup-prepare/XXXX...
NEW
[bbr] XXXX INFO - Copying backup -- 2.6M uncompressed -- for job bbr-usage-servicedb on backup-prepare/XXXX...
This should help users understand where backup artifacts are coming from. Pivotal Tracker Story
Fixed:
Backup metadata were wrongly reporting the job index for backup artifacts. Consider the following example, a multi-master CFCR cluster. The backup artifact is always coming from master/0 while master/1 and master/2 provide no backup artifact. However, previously it was not always reported (in the backup metdatata) that the artifact was coming from master/0.
This makes no noticeable difference but it fixes the accuracy of the metadata file. Pivotal Tracker Story
1.2.7
1.2.6
13cdbfe
MirahImage
Mirah Gary
1.2.4
- Ability to run against deployments with Windows VMs
- Built with Go 1.10.3
Stories
- https://www.pivotaltracker.com/story/show/154209539
- https://www.pivotaltracker.com/story/show/158169350
- https://www.pivotaltracker.com/story/show/158423237
Commits
Add .envrc for system/windows tests
[#158423237]
Rename IsLinux -> IsWindows
This represents the check we are doing correctly
[#158423237]
Print a warning msg when detecting a windows vm
[#158423237]
Merge os checker interface into remote runner interface
No need for a intermediate struct between bosh.Client and
ssh.RemoteRunner
[#158423237]
Call OS Checker in FindInstances before FindJobs
[#158423237]
Add bosh client unit test to check OS of instances
[#158423237]
Re-include jumpbox setup for system-deployment restore tests
They use a different workspace directory.
[#158423237]
Extract system test helper to write BOSH_CA_CERT to temp file
Need to set file permissions to be world readable so that the cert can
be read by the user created for bosh ssh sessions in the tests.
[#158423237]
Add github-key in system-windows task
This is required for running go get
[#158423237]
Fix syntax error in system-deployment task
[#158423237]
Add system-windows test
- Remove repeat jumpbox setup from system-deployment restore tests
- Use BOSH_CA_CERT instead of BOSH_CERT_PATH in system tests
- Add vm_extension to jumpbox-windows-ci deployment to give access to
bosh director - Do not require BOSH_GW_* when BOSH_ALL_PROXY is set
[#158423237]
Rename system tests to system-deployment tests
[#158423237]
Add separate system/windows suite for windows pre-backup-check
[#158423237]
Remove windows pre-backup-check from system/deployment suite
[#158423237]
Fix azs in fixtures/redis-windows-ci.yml manifest
[#158423237]
Update windows system tests deployment names
- jumpbox-windows => jumpbox-windows-ci
- windows-vm => redis-windows-ci
[#158423237]
Merge branch 'wip-windows-system-test'
Add manifest for jumpbox deployment for windows test
[#158423237]
Correct syntax when setting BOSH_ENVIRONMENT
[#154209539]
Rename BOSH_HOST => BOSH_ENVIRONMENT
[#154209539]
WIP system test for windows VM
[#158423237]
Write BOSH_CA_CERT to a file and set BOSH_CERT_PATH in system task
[#158423237]
Rename BOSH_GW_* env vars
- No need to pass as flags to bosh cli, just assert they are set
[#158423237]
Refactor BOSH_URL => BOSH_ENVIRONMENT in sys tests
[#158423237]
Add github sshkey before dep ensure
[#158423237]
Update deployment system to work with BOSH_ALL_PROXY
[#158423237]
Use BOSH_CA_CERT/PATH in upload-systest-releases
[#158423237]
Get $BOSH_CA_CERT from a path in upload-systest-release task
[#158423237]
Require $BOSH_CA_CERT in upload-systest-release task
[#158423237]
Change upload-systest-release to use bosh v2 cli
- Only upload to one bosh director. To upload to two bosh directors the
task can be repeated in the pipeline. - Support $BOSH_ALL_PROXY for bosh directors that are not publicly
accessible.
[#158423237]
Correct stemcell name in windows-vm deployment
[#158423237]
Add Windows VM manifest
[#158423237]
Fix shellcheck warning in ci scripts
- Use separate lines for each bosh command flag
- Do not use set -x
[#158169350]
Print go version when building the binary
[#158169350]
Trim readme
[#158169350]
1.2.3
Fix issue where BBR fails if a locking dependency is specified but missing.
Stories
- https://www.pivotaltracker.com/story/show/156570297
- https://www.pivotaltracker.com/story/show/157043996
- https://www.pivotaltracker.com/story/show/157485442
- https://www.pivotaltracker.com/story/show/157694528
- https://www.pivotaltracker.com/story/show/157779432
- https://www.pivotaltracker.com/story/show/157994374
Commits
Do not fail if a specified locking dependency is missing
[#157994374]
TEAM_GPG_KEY is no longer needed
The git-crypt feature of git-resource is now used to unlock the meta
repo.
[#157485442]
Revert "Establish socks proxy if BOSH_ALL_PROXY set"
This reverts commit b01c84c.
We tried to revert both commits (b01c84c and d106c15) in the previous
commit, but only reverted d106c15.
[#157779432]
Revert support for BOSH_ALL_PROXY
This reverts commits:
Establish socks proxy if BOSH_ALL_PROXY set
[#157779432]
Bump bosh-cli and bosh-utils
We only uses the "uuid" and "logger" package in bosh-utils,
so we added constraint to only vendor those two packages.
[#157779432]
Pull ssh docker image once before ssh suite runs
[#157694528]
Bump the Eventually timeout to 15 minutes in systests
[#156570297]
Wait longer for backup to start in Ctrl-C test
[#157043996]
Add missing log tags in standalone.DeployedInstance
[#156570297]
1.2.2
Fix ssh error introduced by calling scripts in parallel.
Stories
Commits
Fix vm_type in many-bbr-jobs manifest
[#156816267]
Limit parallelism to avoid SSH issues
Trying to perform too many SSH connections at the same time does not
work because of 2 reasons:
- By default, SSH will only authenticate 10 connections at a time, and
drop any other connection (see the MaxStartups config parameter); - Even after increasing MaxStartups, SSH will still struggle to handle
too many simultaneous connection attempts, so it will drop some.
This commit changes the ParallelExecutor to only run a maximum number of
Executables at any time. The number is stored in the maxInFlight field
and currently hardcoded to 10 in the constructor, to match the default
value of MaxStartups.
[#156816267]
Add the many-bbr-jobs fixtures deployment
[#156816267]
1.2.1 - add optional backup artifact-path
82edd09
MirahImage
Mirah Gary
Stories
- https://www.pivotaltracker.com/story/show/154930266
- https://www.pivotaltracker.com/story/show/155550920
- https://www.pivotaltracker.com/story/show/155550955
- https://www.pivotaltracker.com/story/show/155735357
- https://www.pivotaltracker.com/story/show/155894534
- https://www.pivotaltracker.com/story/show/155896024
- https://www.pivotaltracker.com/story/show/156039429
- https://www.pivotaltracker.com/story/show/156301798
Commits
Change wording of cli help message
[#156301798]
Cleanup/rearrange BackupDirectorManager tests
[#156301798]
Improve error handling in BackupDirectorManager.Create()
[#156301798]
Add system tests for director backup with optional artifact-path
[#156301798]
Fix system tests for deployment backup with artifact-path
[#156301798]
Create backup artifact with a specified artifact path
[#156301798]
Use parallel executor in artifact copier during restore
[#155735357]
Refactor artifact copier tests to use fake executor
[#155735357]
Add unit test coverage for BackupDownloadExecutable and BackupUploadExecutable
[#155735357]
Extract BackupDownloadExecutable and BackupUploadExecutable from ArtifactCopier
[#155735357]
backfill tests for job executables
[#155735357]
Add tests for Executor
[#155735357]
Mock the Executor in the Pre/Post/Backup/Restore/Lock/Unlock unit tests
[#155735357]
Clean up Deployment tests
[#155735357]
Add tests for Deployment.BackupableInstances() and RestorableInstances()
[#155735357]
Clean up the ArtifactCopier tests
[#155735357]
Extract ArtifactCopier from Deployment
[#155735357]
Clean up CopyRemoteBackupToLocal
[#155735357]
Upload backup artifacts in parallel
[#155735357]
Add test for concurrent writes to backup metadata
This tests the BackupDirectory.AddChecksum mutex
[#155894534]
Replace Parallel Executor with Serial Executor in restorer
This is to resolve the race condition in post-restore-unlock
[#156039429]
Added integration tests for logging instance name and ID when draining artifacts
[#155896024]
Added artifact information to logs when starting and finishing validity checks
[#155896024]
Merge pull request #13 from takeyourhatoff/patch-1
Fix install instructions
Update README.md
Fix install instructions
Remove sleeps from integration tests
[#155550955]
invert dependency between orchestrator and executor
[#155550955]
removed the artifactexecutor and use the generic executor instead
[#155550955]
Replace JobExecutionStrategy with the more generic Executor
[#155550955]
jobexecutor -> executor
[#155550955]
Remove unused method
Refactor deployment.CopyRemoteBackupToLocal
[#155550955]
Extract artifact execution strategy and use ParallelExecutionStrategy in the drain step
[#154930266]
Add InstanceID method to Artifact interface
We need the function for logging in CopyRemoteToLocal function
[#154930266]
Merge branch 'master' into wip-155550955-2
First pass introducing artifactexecutor
[#155550955]
Remove unnecessary its in orchestrator/deployment_tests.go
[#155550955]
Match on ContainsSubstring instead of one error
We are now accumulating errors in CopyRemoteBackupsToLocal, matching on error string makes more sense
[#155550955]
more wip
First pass on parallelizing backup artifact draining
[#155550955]
Fix director/deployment bbr backup termination assertion integration tests
Asserting on the non-existance of the artifact tar instead of the entire backup directory
Adding sleep before terminating to ensure bbr backup is terminated during during the backup script
to make the test more stable
Fix wording
[#155550955]
Use ParellelJobExecutor in restore
[#155550920]
Make the director restore cleanup sys test less dependant on previous runs
Update director backup interrupted test to be consistent with deployment
Rename deploy-systest-release to upload-systest-release
1.2.0 - call lock / unlock scripts in parallel to minimize API downtime
5c3bb46
edwardecook
Ed Cook
Stories
- https://www.pivotaltracker.com/story/show/153632732
- https://www.pivotaltracker.com/story/show/154169419
- https://www.pivotaltracker.com/story/show/154760449
- https://www.pivotaltracker.com/story/show/155356016
Commits
Log errors in running scripts
[#154169419]
Switch log tag in backup_directory to bbr
[#155356016]
Add labels to stdout/stderr logging
[#155356016]
Remove redundant WaitGroup in the ParallelJobExecutor
[#153632732]
Perform backup locking and unlocking in parallel
[#153632732]
Improve logging
[#153632732]
Add ParallelJobExecutor and its tests
[#153632732]
Add tests for SerialJobExecutor
[#154760449]