CLDSRV-559 AWS KMS backend + account level encryption key #5682

nicolas2bert · 2024-09-26T07:28:37Z

Introduce a new KMS backend that supports the AWS KMS protocol.
Implement a more efficient approach to managing default encryption keys. Instead of creating a master encryption key per bucket, a single default master encryption key is associated with the account. This simplifies key management. The default key id will be stored in the account metadata.

Add configuration mechanism for the new AWS KMS connector. Depends on changes in Arsenal to have support of this new connector.

available Up to now, the datakey was always generated using a locally generated random number. This commit allow to use the "generateDataKey" operation of a KMS when it is implemented. It fallback to random number generation if not available. The benefit of generating the datakey in the KMS is a better entropy source resulting in a "better" datakey.

bert-e · 2024-09-26T07:28:40Z

Hello nicolas2bert,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2024-09-26T07:28:51Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-559 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

7.70.52
8.6.29
8.7.50
8.8.33

Please check the Fix Version/s of CLDSRV-559, or the target
branch of this pull request.

nicolas2bert · 2024-09-26T07:30:48Z

ping

bert-e · 2024-09-26T07:30:57Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

nicolas2bert · 2024-10-01T03:53:45Z

tests/functional/aws-node-sdk/test/object/encryptionHeaders.js

+                .forEach(existing => {
+                    const hasKey = target.masterKeyId ? 'a' : 'no';
+                    const { algo } = target;
+                    it('should override bucket encryption settings with '


Note to reviewer: Here I have not changed anything in the test, just improved the description a bit.

jonathan-gramain · 2024-10-03T09:43:14Z

lib/Config.js

+            ak: process.env.AWS_ACCESS_KEY_ID,
+            sk: process.env.AWS_SECRET_ACCESS_KEY


Suggested change

ak: process.env.AWS_ACCESS_KEY_ID,

sk: process.env.AWS_SECRET_ACCESS_KEY

accessKey: process.env.AWS_ACCESS_KEY_ID,

secretKey: process.env.AWS_SECRET_ACCESS_KEY

jonathan-gramain · 2024-10-03T09:45:49Z

lib/kms/wrapper.js

+                        cipherBundle.masterKeyId,
+                        log, (err, plainTextDataKey, cipheredDataKey) => {
+                            if (err) {
+                                log.debug('error from kms',


You may want to log this with error level? Your call, and the original code was also logging in debug level. Also maybe add some context like "error generating a new data key from KMS"

jonathan-gramain · 2024-10-03T09:49:27Z

lib/kms/wrapper.js

+                        })
+                } else {
+                    log.debug('creating a new data key');
+                    const dataKey = Common.createDataKey();


Suggested change

const dataKey = Common.createDataKey();

const plainTextDataKey = Common.createDataKey();

jonathan-gramain · 2024-10-03T09:58:12Z

lib/kms/wrapper.js

+        if (client.supportsDefaultKeyPerAccount && config.defaultEncryptionKeyPerAccount) {
+            return vault.getOrCreateEncryptionKeyId(bucket.getOwner(), log, (err, data) => {
+                if (err) {
+                    log.debug('error retreiving or creating the default encryption key at the account level from vault',


Suggested change

log.debug('error retreiving or creating the default encryption key at the account level from vault',

log.debug('error retrieving or creating the default encryption key at the account level from vault',

jonathan-gramain · 2024-10-03T09:59:22Z

lib/kms/wrapper.js

+                }
+
+                const { encryptionKeyId, action } = data;
+                log.trace('default encryption key retreived or created at the account level from vault',


Suggested change

log.trace('default encryption key retreived or created at the account level from vault',

log.trace('default encryption key retrieved or created at the account level from vault',

jonathan-gramain · 2024-10-03T10:09:53Z

tests/unit/api/bucketDelete.js

+            sinon.restore();
+        });
+
+        it('should delete the bucket-level encryption key is AES256 algorithm', done => {


Suggested change

it('should delete the bucket-level encryption key is AES256 algorithm', done => {

it('should delete the bucket-level encryption key if AES256 algorithm', done => {

BourgoisMickael · 2024-10-03T20:40:04Z

lib/kms/wrapper.js

+                /* There are 2 ways of generating a datakey :
+                  - using the generateDataKey of the KMS backend if it exists
+                    (currently only implemented for the AWS KMS backend). This is
+                    the prefered solution since a dedicated KMS should offer a better


Suggested change

the prefered solution since a dedicated KMS should offer a better

the preferred solution since a dedicated KMS should offer a better

bert-e · 2024-10-03T20:40:18Z

Incorrect fix version

The Fix Version/s in issue CLDSRV-559 contains:

7.70.52
8.6.29
8.7.50
8.8.33

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

7.70.52
8.6.29
8.7.50
8.8.34

Please check the Fix Version/s of CLDSRV-559, or the target
branch of this pull request.

bert-e · 2024-10-09T08:09:39Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

nicolas2bert · 2024-10-09T08:14:19Z

/create_integration_branches

bert-e · 2024-10-09T08:14:29Z

Conflict

A conflict has been raised during the creation of
integration branch w/8.6/improvement/CLDSRV-559/kms with contents from improvement/CLDSRV-559/kms
and development/8.6.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.6/improvement/CLDSRV-559/kms origin/development/8.6
 $ git merge origin/improvement/CLDSRV-559/kms
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.6/improvement/CLDSRV-559/kms

The following options are set: create_integration_branches

bert-e · 2024-10-09T09:53:20Z

Conflict

A conflict has been raised during the creation of
integration branch w/8.7/improvement/CLDSRV-559/kms with contents from w/8.6/improvement/CLDSRV-559/kms
and development/8.7.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.7/improvement/CLDSRV-559/kms origin/development/8.7
 $ git merge origin/w/8.6/improvement/CLDSRV-559/kms
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.7/improvement/CLDSRV-559/kms

The following options are set: create_integration_branches

bert-e · 2024-10-09T10:12:34Z

Conflict

A conflict has been raised during the creation of
integration branch w/8.8/improvement/CLDSRV-559/kms with contents from w/8.7/improvement/CLDSRV-559/kms
and development/8.8.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/8.8/improvement/CLDSRV-559/kms origin/development/8.8
 $ git merge origin/w/8.7/improvement/CLDSRV-559/kms
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.8/improvement/CLDSRV-559/kms

The following options are set: create_integration_branches

bert-e · 2024-10-09T14:09:02Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

nicolas2bert · 2024-10-09T16:31:02Z

@bert-e approve

bert-e · 2024-10-09T16:31:21Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/7.70
✔️ development/8.6
✔️ development/8.7
✔️ development/8.8

The following branches have NOT changed:

development/7.10
development/7.4

Please check the status of the associated issue CLDSRV-559.

Goodbye nicolas2bert.

The following options are set: approve, create_integration_branches

outscale-fba added 4 commits August 30, 2024 09:30

FT: Provide AWS KMS connector for bucket ciphering

9bf534d

Add configuration mechanism for the new AWS KMS connector. Depends on changes in Arsenal to have support of this new connector.

FT: AWS KMS, fix lint errors + env var renaming after 1st code review

db96279

AWS KMS: TLS configuration

853102f

nicolas2bert force-pushed the improvement/CLDSRV-559/kms branch from 3f9a7fb to 14ab601 Compare September 30, 2024 09:37

nicolas2bert commented Oct 1, 2024

View reviewed changes

nicolas2bert force-pushed the improvement/CLDSRV-559/kms branch from 14ab601 to 7cda219 Compare October 1, 2024 17:23

jonathan-gramain approved these changes Oct 3, 2024

View reviewed changes

BourgoisMickael reviewed Oct 3, 2024

View reviewed changes

BourgoisMickael approved these changes Oct 3, 2024

View reviewed changes

nicolas2bert force-pushed the improvement/CLDSRV-559/kms branch from 0e9495d to 7396bf2 Compare October 9, 2024 07:45

nicolas2bert added 2 commits October 9, 2024 09:50

CLDSRV-559 AWS KMS backend

a63387e

CLDSRV-559 bump package version

b7169de

nicolas2bert force-pushed the improvement/CLDSRV-559/kms branch from 7396bf2 to b7169de Compare October 9, 2024 07:51

bert-e merged commit b7169de into development/7.70 Oct 9, 2024
11 checks passed

bert-e deleted the improvement/CLDSRV-559/kms branch October 9, 2024 16:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLDSRV-559 AWS KMS backend + account level encryption key #5682

CLDSRV-559 AWS KMS backend + account level encryption key #5682

nicolas2bert commented Sep 26, 2024

bert-e commented Sep 26, 2024

bert-e commented Sep 26, 2024

nicolas2bert commented Sep 26, 2024

bert-e commented Sep 26, 2024

nicolas2bert Oct 1, 2024

jonathan-gramain Oct 3, 2024

jonathan-gramain Oct 3, 2024

jonathan-gramain Oct 3, 2024

jonathan-gramain Oct 3, 2024

jonathan-gramain Oct 3, 2024

jonathan-gramain Oct 3, 2024

BourgoisMickael Oct 3, 2024

bert-e commented Oct 3, 2024

bert-e commented Oct 9, 2024

nicolas2bert commented Oct 9, 2024

bert-e commented Oct 9, 2024

bert-e commented Oct 9, 2024

bert-e commented Oct 9, 2024

bert-e commented Oct 9, 2024

nicolas2bert commented Oct 9, 2024

bert-e commented Oct 9, 2024

		ak: process.env.AWS_ACCESS_KEY_ID,
		sk: process.env.AWS_SECRET_ACCESS_KEY

	const dataKey = Common.createDataKey();
	const plainTextDataKey = Common.createDataKey();

	log.debug('error retreiving or creating the default encryption key at the account level from vault',
	log.debug('error retrieving or creating the default encryption key at the account level from vault',

	log.trace('default encryption key retreived or created at the account level from vault',
	log.trace('default encryption key retrieved or created at the account level from vault',

	it('should delete the bucket-level encryption key is AES256 algorithm', done => {
	it('should delete the bucket-level encryption key if AES256 algorithm', done => {

	the prefered solution since a dedicated KMS should offer a better
	the preferred solution since a dedicated KMS should offer a better

CLDSRV-559 AWS KMS backend + account level encryption key #5682

CLDSRV-559 AWS KMS backend + account level encryption key #5682

Conversation

nicolas2bert commented Sep 26, 2024

bert-e commented Sep 26, 2024

Hello nicolas2bert,

bert-e commented Sep 26, 2024

Incorrect fix version

nicolas2bert commented Sep 26, 2024

bert-e commented Sep 26, 2024

Request integration branches

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bert-e commented Oct 3, 2024

Incorrect fix version

bert-e commented Oct 9, 2024

Request integration branches

nicolas2bert commented Oct 9, 2024

bert-e commented Oct 9, 2024

Conflict

bert-e commented Oct 9, 2024

Conflict

bert-e commented Oct 9, 2024

Conflict

bert-e commented Oct 9, 2024

Waiting for approval

nicolas2bert commented Oct 9, 2024

bert-e commented Oct 9, 2024