{Package} Migrate CLI packaging system to PEP 420 Implicit Namespace Packages #14372

arrownj · 2020-07-15T07:59:35Z

Description

Background

PEP420 is an official python enhancement proposal which introduces implicit namespace package.

There are 3 ways to declare a namespace package.

pkg_resource style
pkgutil style
native style (PEP420)

Currently, azure cli is using pkg_resource style namespace package, which is no longer recommended per python official document. Besides, pkg_resource style namespace package performs slow when importing python modules.

This PR is to migrate azure cli namespace package to native style. Refer Packaging namespace packages to see details about above 3 namespace package styles.

There is also a guide about how to do azure sdk package. Refer Azure packaging to see the details.

We will follow this guide to to do azure cli packaging.

Goal

The main goal of this task to to remove __init__.py file under azure and cli folder in our final distributions on different platforms, includes:

MSI
DEB
RPM
Pypi
Homebrew

Changes

In order to achieve this, several changes are needed as below:

azure-cli-dev-tools

remove installation of all cli related -nspkg packages(PR)

some python sdk dependency

azure-cosmos
azure-multiapi-storage

azure-cli

remove all -nspkg dependency
remove azure-cli-nspkg and azure-cli-command_modules-nspkg
update __init__.py, setup.py and MANIFEST.in according to azure packaging guide
move __version__ and __author__ to __main__.py
update azure_bdist_wheel to fix package __main__.py problem
sdk dependency
fix test cases

Plan

merge azure-cli-dev-tools PR
build new azdev version
rollback azdev updates in this PR
merge PR

Testing Guide

Install azure-cli on different platform, and see whether __init__.py still exists
Make sure all previous tests are passed in both current CI pipelines and Feng's new CI pipelines.
Make sure .tar.gz and .whl files are same with previous ones.

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

The PR title and description has followed the guideline in Submitting Pull Requests.
I adhere to the Command Guidelines.

arrownj · 2020-07-15T08:01:09Z

PEP420

mmyyrroonn · 2020-07-21T05:55:30Z

src/azure-cli/azure/cli/command_modules/eventgrid/linter_exclusions.yml

@@ -0,0 +1,12 @@
+---
+eventgrid partner topic event-subscription update:


This should be in global suppression already.

mmyyrroonn · 2020-07-21T06:39:05Z

scripts/ci/install_azdev.sh

@@ -0,0 +1,12 @@
+#!/usr/bin/env bash


This file should be deleted as well.

Looks like this file is not referenced anywhere.

haroldrandom · 2020-07-24T05:27:16Z

azure-pipelines.yml

@@ -485,7 +485,7 @@ jobs:
  displayName: Build Homebrew Formula

  dependsOn: BuildPythonWheel
-  condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual'))


No need to build at those trigger condition?

yes, will revert these changes before merge it. Just test whether these pipelines can pass.

fengzhou-msft

LGTM, please test the upgrade case as well as the new install case.

zhoxing-ms · 2020-08-19T07:18:26Z

Just be curiously, our relase pipeline also publishes pakage to the following platforms:
ACR, Windows Artifacts, Cloud Shell

Will these changes affect the packages for the above platforms?
If so, have they been tested and evaluated?

arrownj · 2020-08-19T07:37:33Z

Just be curiously, our relase pipeline also publishes pakage to the following platforms:
ACR, Windows Artifacts, Cloud Shell

Will these changes affect the packages for the above platforms?
If so, have they been tested and evaluated?

We don't consider the scenario like that. We build package on Windows, MAC, pypi, RPM, DEB, the 3 target platforms you mentioned are all using these distributions. We have test it on all OS platforms. So I think it can cover the scenario you mentioned.

zhoxing-ms · 2020-08-19T07:41:23Z

We don't consider the scenario like that. We build package on Windows, MAC, pypi, RPM, DEB, the 3 target platforms you mentioned are all using these distributions. We have test it on all OS platforms. So I think it can cover the scenario you mentioned.

Got it~ thanks

jiasli · 2020-08-19T07:53:30Z

src/azure-cli/azure/cli/command_modules/config/tests/latest/test_config.py

@@ -24,6 +24,7 @@ def test_config(self):

        # C:\Users\{username}\AppData\Local\Temp
        tempdir = tempfile.gettempdir()
+        tempdir = os.path.realpath(tempdir)


Let's add a comment for this special case on MacOS.

jiasli

My parts LGTM.

jiasli · 2021-12-27T09:59:08Z

This PR was first included in Azure CLI 2.11.0.

arrownj self-assigned this Jul 15, 2020

yonzhan requested review from jiasli, Juliehzl, jsntcy, mmyyrroonn and haroldrandom July 16, 2020 10:59

arrownj marked this pull request as ready for review July 21, 2020 05:17

arrownj requested review from bim-msft, fengzhou-msft, qianwens and zhoxing-ms as code owners July 21, 2020 05:17

mmyyrroonn reviewed Jul 21, 2020

View reviewed changes

haroldrandom reviewed Jul 24, 2020

View reviewed changes

arrownj added 15 commits July 27, 2020 13:41

init pep420 change

409a919

fix linter error

2eaa836

fix another linter error

13fe8e2

elegant way to handle __main__.py

8d8b52c

remove nspkg

6e23b62

refactor azure_bdist_wheel to fit wheel 0.34.2

9a97ee4

add namespace package __init__.py to sdist

b491f97

fix merge conflict

3687e19

fix generic update problem

2304e2b

fix merge conflict

1a286fd

upgrade telemetry version

dda178c

upgrade telemetry version

efd134e

rollback some useless change

b8f203d

build yum/deb/homebrew etx

4d18448

use template to generate formula

c5d2930

arrownj added 5 commits August 18, 2020 14:08

update homebrew package job timeout value

b35b767

use released azdev

5ccc96c

Merge branch 'dev' into another_pep420

225a8c4

rollback pipelines

12403e8

rollback pipelines 2

ede06e0

fengzhou-msft approved these changes Aug 19, 2020

View reviewed changes

haroldrandom approved these changes Aug 19, 2020

View reviewed changes

zhoxing-ms approved these changes Aug 19, 2020

View reviewed changes

jiasli reviewed Aug 19, 2020

View reviewed changes

jiasli approved these changes Aug 19, 2020

View reviewed changes

refactor config test

320bd06

arrownj merged commit 1cf3072 into Azure:dev Aug 19, 2020

This was referenced Feb 23, 2021

Support PEP 420 -- Implicit Namespace Packages #13293

Closed

No module named azure.cli #3351

Closed

fengzhou-msft mentioned this pull request Mar 18, 2021

Remove the limitation of using azure packages in dependency Azure/azure-cli-extensions#3166

Merged

3 tasks

jiasli mentioned this pull request Jul 5, 2021

WSL: Why Azure CLI is too slow at each command execution? #7447

Closed

This was referenced Aug 19, 2021

No module named azure.cli.__main__; #19027

Closed

azure-cli-nspkg should be removed from requirements-azure.txt ansible-collections/azure#604

Closed

jiasli mentioned this pull request Nov 13, 2021

az account show performance #20314

Open

jiasli mentioned this pull request Dec 23, 2021

Profile __init__() missing keyword argument async_persist when calling az login through default cli invoke() #20796

Closed

jiasli changed the title ~~{Package} Migrate CLI packaging system to PEP420~~ {Package} Migrate CLI packaging system to PEP 420 Implicit Namespace Packages Dec 27, 2021

jiasli mentioned this pull request Dec 27, 2021

Failure on SLES 15 SP3 VM: No module named azure.cli.__main__ #20839

Open

This was referenced Apr 2, 2022

In Azure CLI installed by Homebrew, azure and azure.mgmt still have __init__.py files calling pkg_resources #21910

Open

In Azure CLI source code, azure and azure.cli contain __init__.py files calling pkgutil #21911

Open

jiasli mentioned this pull request Jul 12, 2022

[BUG] Importing setuptools is slow and takes nearly half a second pypa/setuptools#3441

Open

jiasli mentioned this pull request Aug 31, 2023

{CI} Remove --exitfirst to collect all error information at once. #27291

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

{Package} Migrate CLI packaging system to PEP 420 Implicit Namespace Packages #14372

{Package} Migrate CLI packaging system to PEP 420 Implicit Namespace Packages #14372

arrownj commented Jul 15, 2020 •

edited by jiasli

Loading

arrownj commented Jul 15, 2020

mmyyrroonn Jul 21, 2020

mmyyrroonn Jul 21, 2020

jiasli Jul 21, 2020

haroldrandom Jul 24, 2020

arrownj Jul 27, 2020

fengzhou-msft left a comment

zhoxing-ms commented Aug 19, 2020

arrownj commented Aug 19, 2020 •

edited

Loading

zhoxing-ms commented Aug 19, 2020

jiasli Aug 19, 2020

arrownj Aug 19, 2020

jiasli left a comment

jiasli commented Dec 27, 2021

		@@ -0,0 +1,12 @@
		---
		eventgrid partner topic event-subscription update:

{Package} Migrate CLI packaging system to PEP 420 Implicit Namespace Packages #14372

{Package} Migrate CLI packaging system to PEP 420 Implicit Namespace Packages #14372

Conversation

arrownj commented Jul 15, 2020 • edited by jiasli Loading

Background

Goal

Changes

Plan

arrownj commented Jul 15, 2020

mmyyrroonn Jul 21, 2020

Choose a reason for hiding this comment

mmyyrroonn Jul 21, 2020

Choose a reason for hiding this comment

jiasli Jul 21, 2020

Choose a reason for hiding this comment

haroldrandom Jul 24, 2020

Choose a reason for hiding this comment

arrownj Jul 27, 2020

Choose a reason for hiding this comment

fengzhou-msft left a comment

Choose a reason for hiding this comment

zhoxing-ms commented Aug 19, 2020

arrownj commented Aug 19, 2020 • edited Loading

zhoxing-ms commented Aug 19, 2020

jiasli Aug 19, 2020

Choose a reason for hiding this comment

arrownj Aug 19, 2020

Choose a reason for hiding this comment

jiasli left a comment

Choose a reason for hiding this comment

jiasli commented Dec 27, 2021

arrownj commented Jul 15, 2020 •

edited by jiasli

Loading

arrownj commented Aug 19, 2020 •

edited

Loading