Support multiple AWS accounts

[chriskilding]

The plugin should be able to retrieve credentials from multiple AWS accounts, and present them as one combined list of credentials.

For example by using IAM cross-account roles.

Use case: Separate AWS accounts for deployment environments

• I have a Jenkins in my environment-independent tools account.
• I have dev secrets in my dev account.
• I have production secrets in my production account.
• I want Jenkins to access secrets in the dev and production accounts.

Moved from https://issues.jenkins.io/browse/JENKINS-63182

[loreleimccollum-work]

We could really use this feature as well, we have secrets in another account that are shared with us, and we need to pull them into Jenkins

[citizenken]

Why does the plugin not just take a secret ARN? Looking through the code, the plugin is doing some querying with filters for secrets. If it had the ARN, couldn't it just access that secret directly, even if it was in a different account?

[chriskilding]

I did experiment with using secret ARNs as credential IDs, but found that credential listing or binding was unreliable. Sometimes secret retrieval failed, and since Moto won't let us simulate cross-account operations yet, I couldn't write tests for this feature to find out why for sure. Perhaps the credentials API did not like some of the characters used in the secret ARNs.

[citizenken]

Makes sense. It seems like a limitation of the AWS API as well, that list secrets is scoped to only the requester's account. It looks like currently, the flow of the plugin is to fetch secrets at startup, then create Jenkins credentials/interpolate JSCAC yaml. Is that accurate? If that's the case, then I can understand why using the value in ${} from jcasc (our primary use case right now) to query AWS doesn't work.

[chriskilding]

You can retrieve secrets from other accounts if you do an sts:assumeRole first, and then reference the resource by full ARN (not just its name) when fetching it. So it's technically possible, but comes with some practical issues:

• how to do that in a reliable way (each cross-account operation is another HTTP call which could fail - how do you partition them so that a failure in one doesn't knock over the others?)
• how to namespace the retrieved credentials so that if two of them have the same name, we don't get a name clash. (This is more common than you'd think; products in my company typically have one AWS account for the staging environment and one for the production environment. Terraform is used to provision the accounts so they look as similar as possible. This means that almost all the secrets would have a name clash in Jenkins if there's no namespacing mechanism!)

[chriskilding]

The way the plugin works at the moment is loosely like this:

1. At startup a Future is created which knows how to retrieve the credentials using ListSecrets.
2. This Future is then wrapped in a memoizer function, which caches the results for 5 minutes if the list operation was successful, and but doesn't cache if not.
3. When the CredentialsProvider#listCredentials() function is called (the main entry point to the plugin), the Future is invoked and the result is handed back.
4. Presuming it was successful, subsequent calls to CredentialsProvider#listCredentials() within the cache period use the memoized result. This avoids hammering the AWS API unnecessarily.
5. The cache resets, and when CredentialsProvider#listCredentials() is next called, steps 3-4 repeat.

Later on, when a credential is bound in a Jenkins job, the secret value is retrieved online with GetSecretValue. The credential consumer may elect to cache the value - within a job, a given credential will only be bound once. But the credential provider never caches the secret value itself.

[chriskilding]

Just posting the first part of the solution in here. Comments and suggestions welcome!

My idea is to introduce a concept of namespaces to the credentials API plugin, to represent credentials that come from different places. In our case that will be different AWS accounts, but it's more generic than that.

This can be used, simply enough, by supplying an optional namespace argument to a credentials API lookup or binding in the Jenkinsfile.

A credentials binding in your Jenkinsfile would look like this:

pipeline {
    agent any
    stages {
        stage('Deploy to staging') {
            environment {
                API_KEY = credentials('api-key', namespace: '1111111111')
            }
            steps {
                sh 'curl -X POST -u "foo:$API_KEY" https://example.com'
            }
        }
        stage('Deploy to production') {
            environment {
                API_KEY = credentials('api-key', namespace: '2222222222')
            }
            steps {
                sh 'curl -X POST -u "foo:$API_KEY" https://example.com'
            }
        }
    }
}

A withCredentials binding would look like this:

node {
    withCredentials(bindings: [string(credentialsId: 'api-key', variable: 'API_KEY', namespace: '1111111111')]) {
        echo 'Hello world'
    }
}

Any credential that does not have an explicit namespace is implicitly in the default namespace. These credentials are bound by invoking withCredentials or credentials in the Jenkinsfile with no namespace argument - just as we do today. This also provides backwards compatibility; credentials from any currently existing provider would be deemed to be in the default namespace.

The next part of the problem will be to work out where in the Jenkins config (think in terms of the casc.yaml file) namespaces would be specified and how they would be connected to credentials providers that want to support them.

[chriskilding]

For part 2 we can revisit the config schema that was used for the clients beta feature:

 unclassified:
   awsCredentialsProvider:
     clients:
       - credentialsProvider:
           assumeRole:
             roleArn: "arn:aws:iam::111111111111:role/foo"
             roleSessionName: "jenkins"
       - credentialsProvider:
           assumeRole:
             roleArn: "arn:aws:iam::222222222222:role/bar"
             roleSessionName: "jenkins"

There's a few options for where to insert namespaces here.

Implicit auto-detection

The namespaces could be auto-sensed inside the provider - by extracting the AWS account ID from the ARN of each secret in the ListSecrets operation, and saving that as a new property on the credential.

In this case there is no need to add extra namespace properties to the CasC configuration.

There is a small risk of a name clash from misconfiguration of Jenkins. If two clients are configured to assumeRole into the same target account, the secrets in that account will be fetched twice, producing a name clash. In this situation the administrator would have to have pasted two roleArns containing the same account ID in the CasC file, which should set alarm bells ringing before they even save that config.

Explicit namespaces

The namespaces could be declared in their own section of the CasC config, then referenced from credential providers. This would also make them usable in other places besides the credentials system.

namespaces:
  - 1111111111
  - 2222222222

unclassified:
   awsCredentialsProvider:
     clients:
       # TODO fill in details

Explicit semantic namespaces

If namespaces are extracted to their own section, we are not limited to using the identifier from the underlying system (e.g. the AWS account ID) as the namespace identifier. Instead, we could give each namespace a semantic name like 'staging' or 'production', and somehow map the semantic identifier down to the real identifier within the providers.

namespaces:
  - staging
  - production

unclassified:
   awsCredentialsProvider:
    clients:
       # TODO fill in details

[kuntalkumarbasu]

really looking forward for this feature to be implemented.

[gals-ma]

Any news on this? :)
could be really helpful

[chriskilding]

Hi @gals-ma

I found an eventual solution to this, which is to implement folders support in an extension of this provider: the folder-scoped AWS Secrets Manager Credentials Provider (https://github.com/chriskilding/aws-secrets-manager-credentials-provider-folders-plugin)

Within each folder, you can choose the AWS authentication option. In effect this allows you to have a connection to a different AWS account in each folder you have, which solves both the multi-account problem as well as namespacing.

I have a draft implementation ready in the PR chriskilding/aws-secrets-manager-credentials-provider-folders-plugin#1 - BUT I need someone to beta test it in a real Jenkins (non-production!) installation, to ensure that it works before I release it. So far nobody has been forthcoming to do that, so it has been stuck in draft. But if you are willing to help test it, we can get this over the line into an initial release :)

[gals-ma]

Hey @chriskilding, appreciate your update on this.
I am willing to test this, can you please share installation instructions?

[chriskilding]

Great! You'll need to:

1. Check out the initial-dev branch of the folder-scoped plugin: https://github.com/chriskilding/aws-secrets-manager-credentials-provider-folders-plugin/tree/initial-dev
2. Build the plugin by compiling that branch from source
3. Deploy the plugin .hpi file in your test Jenkins and see how it works

This assumes some knowledge of Java software development, so let me know if that works for you or not