Design of development, integration and production data storage hierarchy in Couchbase and Capella

[gopa-noaa]

We may want to revisit the current Couchbase Bucket=>Scope=>Collection hierarchy:

Current hierarchy:
vxdata=>development=>[METAR, COMMON, ...]
=>integration=>[METAR, COMMON, ...]
=>production=>[METAR, COMMON, ...]
But given the following documentation on XDCR:
https://docs.couchbase.com/server/current/manage/manage-xdcr/replicate-using-scopes-and-collections.html#replicate-data-between-collections-explicitly-with-the-ui
This below is important:
*** XDCR only permits one replication to be defined, for a given target-bucket.

So either we come up with another mechanism to transition data between development, integration, production other than XDCR
OR
We move development, integration, production one level up to bucket level.

[gopa-noaa]

How to set up the XDCR rule:
{_default.METAR:development.METAR}

[gopa-noaa]

To avoid this you would need to tell cbbackupmgr to ignore the scope IDs and restore the scope in your backup to the scope in your cluster using the --map-data flag e.g. --map-data bucket.testScope=bucket.testScope.

[gopa-noaa]

Steps:

1. Do one time backup of vxdatatest bucket. (cbbackupmgr backup)
2. Drop vxdatatest bucket
3. Re-create vxdatatest bucket with Magma
4. Restore from backup to new vxdatatest bucket (cbbackupmgr restore)
5. If all goes works, then do above steps for vxdata bucket
6. Setup permissions, XDCR etc on new vxdata bucket

[gopa-noaa]

Create the backup repo:
cbbackupmgr config -a /couchbase/backup/standalone -r vxdatatest --include-data vxdatatest

[gopa-noaa]

/opt/couchbase/bin/cbbackupmgr backup --archive /couchbase/backup/standalone -repo vxdatatest --cluster couchbase://adb-cb1.gsd.esrl.noaa.gov --username Administrator --password ####

[gopa-noaa]

/opt/couchbase/bin/cbbackupmgr restore --archive /couchbase/backup/standalone -repo vxdatatest --cluster couchbase://adb-cb1.gsd.esrl.noaa.gov --username Administrator --password #### --map-data vxdatatest=vxdatatest

[randytpierce]

So the current plan is to

1. Stop xdcr and the ingest processing.
2. Drop the vxdata bucket.
3. Re-create the vxdata bucket with magma storage.
4. Restore the bucket from the backups.

[ian-noaa]

Is the discussion here about recreating the bucket with magma storage for issue #409?

The issue description makes it sound like this issue is a continuation of #379.

[gopa-noaa]

@ian-noaa I agree, looks like we can close this one and probably continue with #379 , since I have been adding notes to #379 from our discussions.