-
Notifications
You must be signed in to change notification settings - Fork 336
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
azcollection requirements are so old, which azure-cli package will work with them? #477
Comments
Even cooler - there doesn't appear to be ANY version of azure-cli that matches the requirements for azcollection! azure-cli-2.0.68 wants 6.0 (both of which are nearly two years old at this point) |
@howardjones Thanks for your submit this issue! We are preparing to update all depend packages! once upgrade, I will add comment in here! Thank you very much! |
Does this mean that azcollection will be having the same python module dependencies as azure-cli defined somewhere in recommended requirements file? |
I was wondering this too. At least the same as some version of azure-cli. Currently with the new pip version resolver, pip will just churn and churn forever if you ask it for the azcollection dependencies and azure-cli, since they don't align. |
@howardjones Absolutely! And, as for now, the only solution to combine azcollection and azure-cli on the same machine (or container in my case) is to install CLI binary as package. But this workaround cannot be considered as good, because installation size is about 800MB and contains many other unwanted stuff, like another python installation, its libraries and hundreds of pyc files... Well, I'm really looking forward to see Ansible related functionality aligned with azure-cli, which might lift up the configuration management to the next level. |
I'm not involved in the update that @Fred-sun mentioned, which is the way forward. However I wanted to know if using two virtual environments, one for azcollection, one for azure-cli, would be possible and mitigate the issue for you guys @howardjones and @greystitch in the meantime? There's no hard technical requirement that they both be installed together since azcollection does not directly call the cli. It is what I have done on my machines, although I do azcollection requirements to a venv and azure-cli to the default --user paths. You can then set "ansible_python_interpreter" to the venv during plays that require azcollection, or if using AWX / Tower you can follow those specific instructions for making virtual environments without a need for azure-cli. |
This is what I'm doing currently, but it's not very efficient... the second venv is another 650MB inside the container once azure-cli is installed with pip. |
Gotcha. Yeah, not an ideal situation to duplicate so many packages for version differences. |
@paultaiton Hi, I'm being asked to look into this for another team where they do use the Azure Modules in some Tasks and they use the Azure CLI in others. When you commented that you're splitting up the virtual environments are you calling into other job-templates each with suitable Venvs from a master job-template? From what I gather, the team I'm working with is executing their job directly on the Ansible Worker and is looking for a venv with both capabilities. |
@erikhjensen In theory I believe you should be able to use the fully qualified path to the az executable, and it will pick the right python interpreter environment. That's probably the easiest way to get it working with both. The other option I can think of is to change the "ansible_python_interpreter" on a per play basis, and keep your azcollection tasks and "az" tasks in separate plays. |
@erikhjensen I am doing this with awx at the moment - two extra venvs, one for azcollection (and whatever else we need), and a second for azcli. I also change the
so the job template runs in the azcollection venv, but it calls binaries in the other one which will use their local modules, since they are using that python. If it's for local use, it's worth noting that the package-installed (i.e. apt not pip-installed) azure-cli on Ubuntu from the MS PPA does all this for you in /opt/az. Presumably it does the same in other packages they distribute. |
Hi @Fred-sun, how goes the planning of the dependency updates? I only ask because we had to back out a change which attempted to work around this limitation and I suspect our team is not quite as resourceful as @howardjones as our yum-installed Azure CLI was failing when called from within the awx virtual environments. Fwiw, this is the error we saw within our ansible_az virtual environment.
|
@erikhjensen In general, we install the azcollection dependencies through "pip install-r requires-azure. txt", you can refer to the link --"https://github.com/ansible-collections/azure", Thank you very much! |
Hi @Fred-sun, noted. I believe the concern in the thread is that this list of dependencies is falling far behind the dependencies in Azure CLI's python package and the two packages cannot be installed into the same virtual environment. We will monitor this thread for news of updates to the dependencies. |
Installing requirements-azure.txt breaks azure-cli installation on MacOS, and after finding this issue, I guess we can assume that, yes - azcollection requirements are now so old, that installing Ansible with azcollection now breaks azure-cli on MacOS. For me this renders this collection absolutely useless for local development as I require to use azure-cli for several other things and cannot rely on pip requirements alone. Running different virtual environments just to be able to use ansible against azure is also not an option for us. This issue has already been open since April 1st and nothing seems to have been done to just bump these requirements. Requirements needs to follow these releases closer. Some of them are updated weekly. |
I've been struggling with this for several days, too. I assumed it was something with my environment. Effectively this means I can't use azcollection from Azure Devops as my agent can't login when the azcollection requirements are installed. |
@Liquidmantis You can, but you would need to jump through some hoops with virtualenvs to do it (to run one or both of these tools in its own python venv). This works for us in awx, for similar uses. But it is no fun at all! |
@howardjones Thanks! Yeah, I thought a solution might exist down that path, but that's more than I can take on currently :) Trying to get venvs working well on top of the Azure dynamic inventory plugin doesn't sound like a good time. Fortunately for us, our use case is almost entirely using Ansible on servers, not actual Azure management (so far). Terraform serves that role for us. |
Based on my experience with azcollection I've solved the problem with obsolete dependencies and Azure API moving target by focusing on the single universal I was not sure, if issues tracker of azcollection is an appropriate place to promote an alternative solution ;-) |
@Fred-sun - any progress to share for updating the dependencies? If you are not able to work on it yourself, perhaps you could share advice/direction here and others will be able to assist. Thank you! |
Hi @Fred-sun, any update? Perhaps you could share advice on what needs to be done and others (myself included) will be able to assist. |
Implementation note: many years ago the whole Azure SDK was contained in the single Microsoft library, azcollection was originally implemented by Microsoft employees, the central azcollection/plugins/module_utils/azure_rm_common.py depended on that single Azure SDK ( Some time later Microsoft split they original SDK into multiple pips, see https://pypi.org/project/azure/ and all the different PIPs (divided by "provider", e.g. compute vs. rdbms.postgresql) are now developed and released at different pace. So the implementation of azcollection become hard to maintain, with very unfortunate requirements+dependencies as in Unfortunate architecture (due historical reasons):
@l3ender so you could do two things:
Easier to maintain (better) architecture:
|
Can someone post a workaround? How is anyone using this ansible collection on a modern linux/mac os system where they also have the latest azure cli installed for non-ansible tasks? |
@Ramblurr Related compute and Network resources have been submitted for PR upgrade to the latest, and other resources are under modification. Thank you very much! |
@Ramblurr I'd recommend using the CLI on your host machine and using an Ansible execution environment with the Azure collection to isolate the two. Or, you could containerize both to keep them isolated and prevent dependency intersection. Use ansible-navigator run your_playbook.yml -i hosts \
--pae false \
--mode stdout \
--eei quay.io/scottharwell/azure-execution-env:latest \
--eev $HOME/.azure:/home/runner/.azure |
As this is unfortunately not yet fixed and most likely always be a challenge with the different development speeds i will share my fix. Because i have been struggling on this as well and everyone talks about virtenv but nobody really shows how to use it. What I did is 3 step, I created 2 tasks handling the azure-cli (creation + destruction) and then your main task(s) where you can use the Azure cli commands. Role "install_azure-cli_venv"
Role 'main task'
Role 'remove_azure-cli_venv'
I hope this helps someone struggling with the same :) |
@geekq @l3ender @howardjones @Ramblurr The dependencies in azure-requirements.txt have been updated, but they still conflict with the azure-cli after installation. There is no effective way to avoid such problems for the time being! If you can, only have it run in two separate virtual environments. Thanks! |
All relate dependencies has upgrade, It will contain in version v1.15.0, I will push this to release! So I will close this! Thank you very much! |
@howardjones Since there will be conflict between azure-cli and requirements-azure.txt when installed together, it is recommended to directly install the specified version of azure-cli to effectively avoid this problem, thank you!
|
Dear all, I've tried the latest solution and only the first version works for me: installing first azure.azcollection (I even have to specify In my relatively long experience of pip, I have found that pip is an unstable environment (probably due to its open nature) and results are hard to reproduce and tend to change over time... In the second proposed solution, azure-cli works but azure.azcollection is broken, here is the error message I get while trying to create a new resource group: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: cannot import name 'RecoveryServicesBackupClient' from 'azure.mgmt.recoveryservicesbackup' (/usr/local/lib/python3.8/dist-packages/azure/mgmt/recoveryservicesbackup/__init__.py)
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (ansible[azure] (azure >= 2.0.0)) on alpha's Python /usr/bin/python3. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"} In this second solution, the |
i am having the same problem. |
I'm trying to build a virtualenv for awx/tower that contains azcollection and also the azure cli (since playbooks still need that to do things that the azcollection can't, like creating Service Principals)
However, the current azcli uses a very short script wrapper over python modules
python3 -Im azure.cli $@
The versions (2.11.1 vs 2.21.0 currently) installed by azcollection are not structured this way:
Which version of azure-cli will actually work with these old azure modules, until you move everything to the newer modules?
The text was updated successfully, but these errors were encountered: