Support for multiple assume roles to get collective inventory

[chandrakanthkannam]

SUMMARY

This change is to implement a ability to get a collective inventory.

Motivation came from our setup, we have a centralized management account from which things can be managed in underlying accounts and we been struggling to attain a central inventory from all accounts in one go, which lead to implement this and we strongly believe this can be useful to community and others can take advantage of it.

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

New option included to the plugin, assume_role_arns: []

This can be taken advantage of if ansible controller is running from a central location and have ability to assume multiple roles in different accounts, this will result in a collective inventory from all the locations

example:

assume_role_arns:
  - arn:aws:iam::<account-id-1>:role/<role-name>
  - arn:aws:iam::<account-id-2>:role/<role-name>
  - arn:aws:iam::<account-id-3>:role/<role-name>

This will give out the inventory from all the 3 accounts, assuming the controller its running from have access to assume these roles.

NOTE: this will still expect a base credential setup, that is, either AWS Keys or EC2 instance profile(when running from an EC2)

ADDITIONAL INFORMATION

This new option conflicts with an existing option, assume_role_arn, we can use either one but not both

To test this here is an example file I used,

plugin: aws_ec2
assume_role_arns:
  - arn:aws:iam::<acc-id-1>:role/<role-name>
  - arn:aws:iam::<acc-id-2>:role/<role-name>
regions:
  - us-west-2
  - eu-central-1
  - ap-southeast-2

keyed_groups:
  - key: tags.Name
    separator: ""

hostnames:
  - 'private-ip-address'

and when running ansible-inventory -i <file-name>--graph -vvv gives me inventory from both accounts across 3 regions, and i used a base credential which has assuming access to these 2 roles used.

Exception:
It fails with conflict error when used both assume_role_arn and assume_role_arns like this in a file

plugin: aws_ec2
assume_role_arn: arn:aws:iam::<acc-id-x>:role/<role-name>
assume_role_arns:
  - arn:aws:iam::<acc-id-1>:role/<role-name>
  - arn:aws:iam::<acc-id-2>:role/<role-name>
regions:
  - us-west-2
  - eu-central-1
  - ap-southeast-2

keyed_groups:
  - key: tags.Name
    separator: ""

hostnames:
  - 'private-ip-address'

error: Conflict here, cannot have not assume_role_arn and assume_role_arns used together

[softwarefactory-project-zuul]

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/0b7fbb9957ac42618c048b65de108482

✔️ ansible-galaxy-importer SUCCESS in 4m 27s
✔️ build-ansible-collection SUCCESS in 12m 37s
✔️ ansible-test-splitter SUCCESS in 4m 54s
✔️ integration-amazon.aws-1 SUCCESS in 20m 26s
Skipped 43 jobs

[softwarefactory-project-zuul]

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/4b2d493369e94e0784ad429631112e2a

✔️ ansible-galaxy-importer SUCCESS in 5m 00s
✔️ build-ansible-collection SUCCESS in 12m 49s
✔️ ansible-test-splitter SUCCESS in 5m 07s
✔️ integration-amazon.aws-1 SUCCESS in 19m 05s
Skipped 43 jobs

[chandrakanthkannam]

Hi Team, can we please get a review on this?

[hakbailey]

@chandrakanthkannam thank you for submitting this PR! It will need a couple of things to be merged, a changelog fragment and integration tests. These tests will need to be marked as unsupported (see example here) since we don't have multiple AWS accounts available to our CI setup, but we still need tests that can be run locally for verification. Let me know if you have questions about either of those requirements, I'm happy to help.

[hakbailey]

Suggested change

	"""
	version_added: 7.2.0
	"""

[hakbailey]

Suggested change

	super().collective_roles(role)
	self.collective_roles(role)

[tremble]

It's already possible to pull from multiple accounts by having multiple inventory files in the inventory folder (or passed with multiple -i arguments):
account_1_aws_ec2.yml
account_2_aws_ec2.yml

As such I'm not sure this is a feature we should add, especially since we can't properly test it.

[hakbailey]

It's already possible to pull from multiple accounts by having multiple inventory files in the inventory folder (or passed with multiple -i arguments): account_1_aws_ec2.yml account_2_aws_ec2.yml

As such I'm not sure this is a feature we should add, especially since we can't properly test it.

Oh interesting. This makes sense now that I'm reading through the general inventory plugins documentation, but it wasn't obvious to me from the ec2 lookup plugin documentation. I wonder if it would be worth updating the plugin doc to make this use case clear?

[tremble]

I wonder if it would be worth updating the plugin doc to make this use case clear?

I would be wary of duplicating too much of the documentation as it'll be liable to bit-rot, but it may be worth linking to the inventory guide and hinting that things like multiple inventory files (even using the same plugin) are possible.

[chandrakanthkannam]

Thank you both for sharing your thoughts on this, I have not tested the scenario myself with multiple inventory files, I will give it a go shortly. Yeah reading through the plugin documentation didn't leave an impression that we can use multiple inventory files.

[chandrakanthkannam]

with placing multiple inventory files in a folder and using the directory path is giving out the collective inventory, that is exactly I'm trying to achieve here with the changes. I guess I don't see any additional advantages ATM with my changes and not taking the directory approach. Anyway thanks @hakbailey for mentioning the missing pieces from the PR, I will keep them in mind for future. And thank you @tremble for pointing out that this is something achievable with the current version itself.

[softwarefactory-project-zuul]

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/140f8ed1b4c54ca0b646d056fc148887

✔️ ansible-galaxy-importer SUCCESS in 4m 10s
✔️ build-ansible-collection SUCCESS in 14m 30s
✔️ ansible-test-splitter SUCCESS in 5m 54s
✔️ integration-amazon.aws-1 SUCCESS in 17m 13s
Skipped 43 jobs

[tremble]

Thanks for taking the time to open this PR. As already discussed, we can already do this using multiple inventory files. As such I'm going to close this PR, especially given we wouldn't be able to properly test it.