From 237f4c4b379480660ca317c230c5775610827296 Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Fri, 31 May 2019 10:26:54 -0700 Subject: [PATCH 01/11] draft of PSGet3.0 proposal (#4) * draft of PSGet3.0 proposal * address feedback * address additional feedback --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 242 ++++++++++++++++++ 1 file changed, 242 insertions(+) create mode 100644 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md new file mode 100644 index 00000000..56a6bd49 --- /dev/null +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -0,0 +1,242 @@ +--- +RFC: RFCxxxx +Author: Steve Lee +Status: Draft +SupercededBy: N/A +Version: 0.1 +Area: PowerShellGet +Comments Due: 7/30 +Plan to implement: Yes +--- + +# PowerShellGet 3.0 Module + +PowerShellGet has enabled an active community to publish and install PowerShell +Modules. +However, it has some architectural and design limitations that prevent it from +moving forward more quickly. +This RFC proposes some significant changes to address this. + +> [!NOTE] +> There are no intentions to break existing PSGet v2 users so existing +> users can stay on PSGet v2 and that will continue to work. + +## Motivation + + As a PowerShell user, + I can discover how to install missing cmdlets, + so that I can be more productive. + + As a PowerShellGet contributor, + I can more easily contribute code, + so that PowerShellGet evolves more quickly. + +## User Experience + +```powershell +PS> Get-Satisfaction +Get-Satisfaction : The term 'Get-Satisfaction' is not recognized as the name of a cmdlet, function, script file, or operable program. +Check the spelling of the name, or if a path was included, verify that the path is correct and try again. +At line:1 char:1 ++ Get-Satisfaction ++ ~~~ ++ CategoryInfo : ObjectNotFound: (Get-Satisfaction:String) [], CommandNotFoundException ++ FullyQualifiedErrorId : CommandNotFoundException + +Suggestion: You can obtain `Get-Satisfaction` by running `Install-PSResource HappyDays`. + +PS> Install-PSResource HappyDays +``` + +## Specification + +### Rewrite of PowerShellGet + +PowerShellGet is currently written as PowerShell script with a dependency on PackageManagement. +Proposal is to write PSResource in C# to reduce complexity and make easier to maintain. +In addition, remove dependency on PackageManagement completely as well as dependency on +nuget.exe. +This module will only use REST APIs to get and publish nupkgs. +This module would be shipped in PowerShell 7. + +### Side-by-side with PowerShellGet + +To ease transition due to the enormity of the breaking change, this module will +have a different names for the cmdlets to allow for side by side use +with the existing PowerShellGet module. +Since the current PowerShellGet 2.x version is a script module and this new one +is a C# based module, they can coexist side-by-side. + +### Cmdlet compatibility + +Function wrappers will be provided to provide compatibility with the two most often +used cmdlets: `Install-Module` and `Find-Module`. +These cmdlets will output a warning to use the new cmdlets. + +### Local cache + +Rather than connecting to PSGallery and other repositories for finding modules, +the user must explicitly request the current metadata of +modules from the repositories and that is cached locally. +`Find-PSResource` (see below) works against this local cache. +This will also enable changes in PowerShell to use `Find-PSResource -Type Command` to look +in this cache when it can't find a command and suggest to the user the module to install to +get that command. +This will be a local json file containing only the latest version of each module. +The cache will be stored in the user path. +There is no system cache that is shared. + +Example cache entry: + +```json +{ + "name": "This is my module", + "version": "1.0.0.0", + "type": "module", + "tags": [ + "Linux", + "PSEdition_Core", + "PSEdition_Desktop", + "AzureAutomation" + ] +} +``` + +An example cache with 5000 resources is ~700KB in compressed json form. + +### Automatic updating of the cache + +On any operation that reaches out to a repository, a REST API will be called to +see if the hash of the cache matches the current cache and if not, a new one +is downloaded. +If the repository doesn't support this new API, it falls back to current behavior +in PSGet v2. + +### Repository management + +`Register-PSResourceRepository` will allow for registering additional repositories. +A `-Default` switch enables registering PSGallery should it be accidentally removed. +The `-URL` will accept the HTTP address without the need to specify `/api/v2` as +that will be assumed and discovered at runtime. +Support for local filesystem repositories must be maintained. +A `-InstallationPolicy` switch accepts `Trusted` or `Untrusted` (default) indicating +whether to prompt the user when installing resources from that repository. + +`Get-PSResourceRepository` will list out the registered repositories. + +`Set-PSResourceRepository` can be used to update a repository URL or installation policy. + +`Unregister-PSResourceRepository` can be used to un-register a repository. + +### Finding resources + +Current design of PowerShellGet is to have different cmdlets for each of the +different resources types that it supports: + +`Find-Command` +`Find-DscResource` +`Find-Module` +`Find-RoleCapability` +`Find-Script` + +Instead, these will be abstracted by a `Find-PSResource` cmdlet. +A `-Type` parameter will accept an array of types to allow filtering. + +With support of the generic `PSResource`, this means we can also find and +install arbitrary nupkgs that may only contain an assembly the user wants to +use in their script. + +If the local cache does not exist, the cmdlet will call `Update-PSResourceCache` +first and then attempt a search. +However, this cmdlet does not update the cache if one exists. + +### Installing resources + +`Install-PSResource` will only work for modules. +A `-AllowPrerelease` switch allows installing prerelease versions. +Other types will use `Save-PSResource` (see below). +A `-Repository` parameter accepts a specific repository name. +If `-Repository` is not specified and the same resource is found in multiple +repositories, then the resource is automatically installed quietly from the +trusted repository with the highest version matching the `-Version` parameter +(if specified, otherwise newest non-prerelease version unless `-AllowPrerelease` +is used). +If there are no trusted repositories matching the query, then the newest version +fulfilling the query will be prompted to be installed. +If there are multiple repositories with the same trust level containing the same +version, the first one is used. + +`-AllowUntrusted` can be used to suppress being prompted for untrusted sources. +`-AllowDifferentPublisher` can be used to suppress being prompted if the publisher +of the module is different from the currently installed version. +`-AllowReinstall` can be used to re-install a module. + +A `-Quiet` switch will suppress progress information. + +### Dependencies and version management + +`Install-PSResource` can accept a path to a psd1 file (using `-RequiredModulesFile`) +or a hashtable (using `-RequiredModules`) where the key is the module name and the +value is either the required version specified using Nuget version range syntax or +a hash table where `repository` is set to the URL of the repository and +`version` contains the [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). + +```powershell +Install-PSResource -RequiredModules @{ + "Configuration" = "[1.3.1,2.0)" + "Pester" = @{ + version = "[4.4.2,4.7.0]" + repository = "https://www.powershellgallery.com" + } +} +``` + +### Saving resources + +With the removal of PackageManagement, there is still a need to support saving +arbitrary nupkgs (assemblies) used for scripts. + +`Save-PSResource -Type Library` will download nupkgs that have a `lib` folder. +A `-AllowPrerelease` switch allows saving prerelease versions. +The dependent native library in `runtimes` matching the current system runtime +will be copied to the same destination. + +This cmdlet has the same parameters as `Install-PSResource` with the addition +of mandatory `-Path` or `-LiteralPath` parameters. + +### Updating resources + +`Update-PSResource` will update all resources to most recent minor version by +default. +A `-OnlyMinorUpdates` switch will allow only updating to newer minor version. +If the installed resource is a pre-release, it will automatically update to +latest prerelease or stable version (if available). + +### Publishing resources + +`Publish-PSResource` will supporting publishing modules and scripts. +Modules will be given as a path to the module folder. +Publishing by name and version is no longer supported. + +### Listing installed resources + +`Get-PSResource` will list all installed resources with new `Type` column. + +```output +Version Name Type Repository Description +------- ---- ---- ---------- ----------- +5.0.0 VSTeam Module PSGallery Adds functionality for working with Visual … +0.14.94 PSGitHub Module PSGallery This PowerShell module enables integration … +0.7.3 posh-git Module PSGallery Provides prompt with Git status summary … +1.0.1 PSAutoMute Script PSGallery Powershell script to Auto Mute you sound devices … +``` + +## Alternate Proposals and Considerations + +This RFC does not cover the module authoring experience on publishing a cross-platform +module with multiple dependencies and supporting multiple runtimes. + +If there is a desire to explicitly update the local cache (like `apt`), we can introduce a +`Update-PSResourceCache` cmdlet with property on a PSRepository indicating whether +it auto-updates or not. +This would not be a breaking change. From fe95b3f07bbe6f69c39052c2abe6795b62975a03 Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Tue, 11 Jun 2019 19:45:39 -0700 Subject: [PATCH 02/11] address feedback --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 166 +++++++++++++----- 1 file changed, 124 insertions(+), 42 deletions(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index 56a6bd49..e927ccfe 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -3,10 +3,10 @@ RFC: RFCxxxx Author: Steve Lee Status: Draft SupercededBy: N/A -Version: 0.1 +Version: 0.2 Area: PowerShellGet Comments Due: 7/30 -Plan to implement: Yes +Plan to implement: Yes, PS7 --- # PowerShellGet 3.0 Module @@ -56,41 +56,39 @@ PowerShellGet is currently written as PowerShell script with a dependency on Pac Proposal is to write PSResource in C# to reduce complexity and make easier to maintain. In addition, remove dependency on PackageManagement completely as well as dependency on nuget.exe. -This module will only use REST APIs to get and publish nupkgs. -This module would be shipped in PowerShell 7. +This module will depend on https://www.nuget.org/packages/NuGet.Client. +This module would be shipped in PowerShell 7 and on PSGallery supporting older +versions of PowerShell. ### Side-by-side with PowerShellGet -To ease transition due to the enormity of the breaking change, this module will -have a different names for the cmdlets to allow for side by side use -with the existing PowerShellGet module. Since the current PowerShellGet 2.x version is a script module and this new one is a C# based module, they can coexist side-by-side. -### Cmdlet compatibility - -Function wrappers will be provided to provide compatibility with the two most often -used cmdlets: `Install-Module` and `Find-Module`. -These cmdlets will output a warning to use the new cmdlets. +Script and module metadata will retain the same format as it exists with v2. ### Local cache -Rather than connecting to PSGallery and other repositories for finding modules, -the user must explicitly request the current metadata of -modules from the repositories and that is cached locally. -`Find-PSResource` (see below) works against this local cache. +Instead of always connecting to PSGallery to perform an online search, +`Find-PSResource` (see below) works against a local cache. This will also enable changes in PowerShell to use `Find-PSResource -Type Command` to look in this cache when it can't find a command and suggest to the user the module to install to get that command. This will be a local json file containing only the latest version of each module. The cache will be stored in the user path. There is no system cache that is shared. +A system wide cache would require elevation or sudo to create and update preventing +it from being useful. Example cache entry: ```json { "name": "This is my module", + "exportedFunctions": [ + "Get-One", + "Set-Two" + ], "version": "1.0.0.0", "type": "module", "tags": [ @@ -102,7 +100,16 @@ Example cache entry: } ``` -An example cache with 5000 resources is ~700KB in compressed json form. +An example cache with 5000 resources (approximately the number of unique packages +currently published to PowerShellGallery.com) is ~700KB in compressed json form. + +The cache would have both latest stable and latest prerelease versions of resources. + +>[!NOTE] +>Need to experiment if it makes sense to have a single cache file for a repositories +>or a different cache per repository for size and perf reasons. + +Perf tests will determine if the cache needs to be in binary form. ### Automatic updating of the cache @@ -111,20 +118,35 @@ see if the hash of the cache matches the current cache and if not, a new one is downloaded. If the repository doesn't support this new API, it falls back to current behavior in PSGet v2. +This means that there is no local cache of that repository and operations will +always connect to that repository. ### Repository management `Register-PSResourceRepository` will allow for registering additional repositories. A `-Default` switch enables registering PSGallery should it be accidentally removed. -The `-URL` will accept the HTTP address without the need to specify `/api/v2` as -that will be assumed and discovered at runtime. +The `-URL` will accept the HTTP address without the need to specify `/api/v3` as +that will be assumed and discovered at runtime (trying v3 first, then falling +back to v2, then the literal URL). Support for local filesystem repositories must be maintained. -A `-InstallationPolicy` switch accepts `Trusted` or `Untrusted` (default) indicating -whether to prompt the user when installing resources from that repository. +A `-Trusted` switch indicates whether to prompt the user when installing resources +from that repository. +By default, if this switch is not specified, the repository is untrusted. +A `-Repositories` parameter will accept an array of hashtables equivalent to +the parameter names. +A `-Name` parameter allows for setting a friendly name. + +A `-Default` switch will set one repository as the default (when not specified +with other cmdlets). +Each time it is specified, it sets the new one as default and the previous default +is no longer default. + +A `-Scope` parameter will support `AllUsers` and `CurrentUsers`. `Get-PSResourceRepository` will list out the registered repositories. -`Set-PSResourceRepository` can be used to update a repository URL or installation policy. +`Set-PSResourceRepository` can be used to update a repository URL, trust level, +or if that repository is the default. `Unregister-PSResourceRepository` can be used to un-register a repository. @@ -146,77 +168,127 @@ With support of the generic `PSResource`, this means we can also find and install arbitrary nupkgs that may only contain an assembly the user wants to use in their script. -If the local cache does not exist, the cmdlet will call `Update-PSResourceCache` -first and then attempt a search. +For each repository, if the local cache does not exist, the cmdlet will call +`Update-PSResourceCache` first and then attempt a search. However, this cmdlet does not update the cache if one exists. +`Find-Module` will be retained to provide compatibility with v2. + ### Installing resources `Install-PSResource` will only work for modules. -A `-AllowPrerelease` switch allows installing prerelease versions. +A `-Prerelease` switch allows installing prerelease versions. Other types will use `Save-PSResource` (see below). -A `-Repository` parameter accepts a specific repository name. +A `-Repository` parameter accepts a specific repository name or URL to the repository: + +```powershell +Install-Module myModule -Repository 'https://mygallery.com' +``` + If `-Repository` is not specified and the same resource is found in multiple repositories, then the resource is automatically installed quietly from the trusted repository with the highest version matching the `-Version` parameter -(if specified, otherwise newest non-prerelease version unless `-AllowPrerelease` +(if specified, otherwise newest non-prerelease version unless `-Prerelease` is used). If there are no trusted repositories matching the query, then the newest version fulfilling the query will be prompted to be installed. If there are multiple repositories with the same trust level containing the same version, the first one is used. -`-AllowUntrusted` can be used to suppress being prompted for untrusted sources. -`-AllowDifferentPublisher` can be used to suppress being prompted if the publisher +`-TrustRepository` can be used to suppress being prompted for untrusted sources. +`-IgnoreDifferentPublisher` can be used to suppress being prompted if the publisher of the module is different from the currently installed version. -`-AllowReinstall` can be used to re-install a module. +`-Reinstall` can be used to re-install a module. + +A `-DestinationPath` parameter allows specifying the target directory instead +of the default one. +This will be in a different parameter set than `-Scope`. + +A `-NoClobber` switch will prevent installing modules that have the same cmdlets +as a differently named module already on the system. A `-Quiet` switch will suppress progress information. +`-Scope` with `AllUsers` and `CurrentUser` will work the same as it does in v2. + +`Install-Module` cmdlet will be retained for compatibility with v2. + +A change from v2 is that installed modules use the full semver (if used) of +the module, so PSReadLine would be in a folder called `2.0.0-beta4` instead of +just `2.0.0` which resolves issues of attempting to overwrite assemblies currently +loaded on Windows. + ### Dependencies and version management -`Install-PSResource` can accept a path to a psd1 file (using `-RequiredModulesFile`) -or a hashtable (using `-RequiredModules`) where the key is the module name and the +`Install-PSResource` can accept a path to a psd1 file (using `-RequiredResourcesFile`), +or a hashtable or json (using `-RequiredResources`) where the key is the module name and the value is either the required version specified using Nuget version range syntax or a hash table where `repository` is set to the URL of the repository and `version` contains the [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). ```powershell -Install-PSResource -RequiredModules @{ +Install-PSResource -RequiredResources @{ "Configuration" = "[1.3.1,2.0)" "Pester" = @{ version = "[4.4.2,4.7.0]" repository = "https://www.powershellgallery.com" + credential = $cred } } ``` +The `repository` property can be the name of the registered repository or the URL +to a repository. +If no `repository` is specified, it will use the `default` repository. + +The older `System.Version` four part version type will be supported to retain +compatibility with existing published modules using that format. + +Due to the differences between semver and `System.Version`, we may have to keep +the two distinct. + +Dependent declared modules not found on the system will prompt to be installed +including module name, version, and size unless `-IncludeDependencies` is +specified which will install without prompting. +Declared dependencies are only searched within the same repository as the original +module to be installed. + ### Saving resources With the removal of PackageManagement, there is still a need to support saving arbitrary nupkgs (assemblies) used for scripts. `Save-PSResource -Type Library` will download nupkgs that have a `lib` folder. -A `-AllowPrerelease` switch allows saving prerelease versions. The dependent native library in `runtimes` matching the current system runtime -will be copied to the same destination. +will be copied to the root of the destination specified. +A `-IncludeAllRuntimes` can be used to explicitly retain the `runtimes` directory +hierarchy within the nupkg to the root of the destination. + +A `-Prerelease` switch allows saving prerelease versions. +If the `-Version` includes a prerelease label like `2.0.0-beta4`, then this +switch is not necessary and the prerelease version will be installed. -This cmdlet has the same parameters as `Install-PSResource` with the addition -of mandatory `-Path` or `-LiteralPath` parameters. +This cmdlet uses `Install-PSResource -DestinationPath`. ### Updating resources -`Update-PSResource` will update all resources to most recent minor version by -default. -A `-OnlyMinorUpdates` switch will allow only updating to newer minor version. +`Update-PSResource` will update all resources to most recent minor version by default. +A `-AllowMajorVersionUpdate` switch will allow updating to newer major version. +A `-PatchUpdateOnly` switch will allow updating only to patched versions. + If the installed resource is a pre-release, it will automatically update to latest prerelease or stable version (if available). ### Publishing resources `Publish-PSResource` will supporting publishing modules and scripts. -Modules will be given as a path to the module folder. -Publishing by name and version is no longer supported. +It will follow the same model as `Publish-Module` in v2. + +A `-DestinationPath` can be used to publish the resulting nupkg locally. +`Publish-PSResource` can accept a path to the nupkg to publish to a repository. + +A `-Nuspec` parameter can be used to specify a nuspec file rather than relying +on this module to produce one. ### Listing installed resources @@ -231,6 +303,11 @@ Version Name Type Repository Description 1.0.1 PSAutoMute Script PSGallery Powershell script to Auto Mute you sound devices … ``` +### Uninstalling resources + +`Uninstall-PSResource` will be available to remove installed resources. +Uninstalling dependencies will be something to consider in the future. + ## Alternate Proposals and Considerations This RFC does not cover the module authoring experience on publishing a cross-platform @@ -240,3 +317,8 @@ If there is a desire to explicitly update the local cache (like `apt`), we can i `Update-PSResourceCache` cmdlet with property on a PSRepository indicating whether it auto-updates or not. This would not be a breaking change. + +The ability to set policy for PSGet is outside the scope of this RFC. + +Automatic cleanup of old resources is out of scope of this RFC. +Keeping with current behavior, installs are always side-by-side. From fe599466ffa412317e2a0150637bf7448c107207 Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Tue, 11 Jun 2019 20:08:33 -0700 Subject: [PATCH 03/11] a few additiona clarifications --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 38 +++++++++++++++---- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index e927ccfe..e462a7dc 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -108,8 +108,7 @@ The cache would have both latest stable and latest prerelease versions of resour >[!NOTE] >Need to experiment if it makes sense to have a single cache file for a repositories >or a different cache per repository for size and perf reasons. - -Perf tests will determine if the cache needs to be in binary form. +>Perf tests will determine if the cache needs to be in binary form. ### Automatic updating of the cache @@ -176,7 +175,8 @@ However, this cmdlet does not update the cache if one exists. ### Installing resources -`Install-PSResource` will only work for modules. +`Install-PSResource` will only work for modules unless `-DestinationPath` is +specified which works for all resource types. A `-Prerelease` switch allows installing prerelease versions. Other types will use `Save-PSResource` (see below). A `-Repository` parameter accepts a specific repository name or URL to the repository: @@ -220,7 +220,7 @@ loaded on Windows. ### Dependencies and version management -`Install-PSResource` can accept a path to a psd1 file (using `-RequiredResourcesFile`), +`Install-PSResource` can accept a path to a psd1 or json file (using `-RequiredResourcesFile`), or a hashtable or json (using `-RequiredResources`) where the key is the module name and the value is either the required version specified using Nuget version range syntax or a hash table where `repository` is set to the URL of the repository and @@ -237,6 +237,25 @@ Install-PSResource -RequiredResources @{ } ``` +The json format will be the same as if this hashtable is passed to `ConvertTo-Json`: + +```json +{ + "Pester": { + "version": "[4.4.2,4.7.0]", + "credential": null, + "repository": "https://www.powershellgallery.com" + }, + "Configuration": "[1.3.1,2.0)" +} +``` + +>[!NOTE] +>Credentials management will be using an upcoming RFC for generalized +>credential management. Not sure how to make that work in json other +>than requiring json to have credential as a json object but that means +>password/secret is in clear text. + The `repository` property can be the name of the registered repository or the URL to a repository. If no `repository` is specified, it will use the `default` repository. @@ -255,6 +274,8 @@ module to be installed. ### Saving resources +This cmdlet uses `Install-PSResource -DestinationPath`. + With the removal of PackageManagement, there is still a need to support saving arbitrary nupkgs (assemblies) used for scripts. @@ -264,12 +285,13 @@ will be copied to the root of the destination specified. A `-IncludeAllRuntimes` can be used to explicitly retain the `runtimes` directory hierarchy within the nupkg to the root of the destination. +>[!NOTE] +>Library support may not be available in 3.0. + A `-Prerelease` switch allows saving prerelease versions. If the `-Version` includes a prerelease label like `2.0.0-beta4`, then this switch is not necessary and the prerelease version will be installed. -This cmdlet uses `Install-PSResource -DestinationPath`. - ### Updating resources `Update-PSResource` will update all resources to most recent minor version by default. @@ -306,7 +328,9 @@ Version Name Type Repository Description ### Uninstalling resources `Uninstall-PSResource` will be available to remove installed resources. -Uninstalling dependencies will be something to consider in the future. + +>[!NOTE] +>Uninstalling dependencies automatically will be something to consider in the future. ## Alternate Proposals and Considerations From c71454770e2ff540792020b5e339d4d2cc0343ec Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Tue, 11 Jun 2019 20:10:13 -0700 Subject: [PATCH 04/11] update feedback time --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index e462a7dc..e4a510e3 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -5,7 +5,7 @@ Status: Draft SupercededBy: N/A Version: 0.2 Area: PowerShellGet -Comments Due: 7/30 +Comments Due: 6/30 Plan to implement: Yes, PS7 --- From 3e904e5565d71f52c9eb8a6cce63836f1142bf6d Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Thu, 13 Jun 2019 14:17:36 -0700 Subject: [PATCH 05/11] add note to return better error message if there is PSGallery outage --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index e4a510e3..e348e4dd 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -332,6 +332,11 @@ Version Name Type Repository Description >[!NOTE] >Uninstalling dependencies automatically will be something to consider in the future. +### PowerShellGallery status + +Upon failure to connect to PSGallery, the cmdlets should retrieve status from +a well known REST API and return a more descriptive error message to the user. + ## Alternate Proposals and Considerations This RFC does not cover the module authoring experience on publishing a cross-platform From a4a986afe83ac788db0e9ccda8fe816a00654afe Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Thu, 13 Jun 2019 14:27:45 -0700 Subject: [PATCH 06/11] Update 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md Co-Authored-By: Dongbo Wang --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index e348e4dd..292ae75a 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -106,7 +106,7 @@ currently published to PowerShellGallery.com) is ~700KB in compressed json form. The cache would have both latest stable and latest prerelease versions of resources. >[!NOTE] ->Need to experiment if it makes sense to have a single cache file for a repositories +>Need to experiment if it makes sense to have a single cache file for all repositories >or a different cache per repository for size and perf reasons. >Perf tests will determine if the cache needs to be in binary form. From cc293e7d9c8bf7b01da7b051f73cb2af0691c9ae Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Thu, 13 Jun 2019 15:08:15 -0700 Subject: [PATCH 07/11] address Mark and Dongbo's feedback --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 42 ++++++++++++++++--- 1 file changed, 36 insertions(+), 6 deletions(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index 292ae75a..8949b85f 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -27,6 +27,10 @@ This RFC proposes some significant changes to address this. I can discover how to install missing cmdlets, so that I can be more productive. + As a PowerShell user, + I can easily install modules without specifying lots of switches, + so that I can be more productive. + As a PowerShellGet contributor, I can more easily contribute code, so that PowerShellGet evolves more quickly. @@ -115,6 +119,11 @@ The cache would have both latest stable and latest prerelease versions of resour On any operation that reaches out to a repository, a REST API will be called to see if the hash of the cache matches the current cache and if not, a new one is downloaded. + +>[!NOTE] +>In the implementation we'll need to handle the case where the module attempts +>to update the cache while it is being read. + If the repository doesn't support this new API, it falls back to current behavior in PSGet v2. This means that there is no local cache of that repository and operations will @@ -132,7 +141,15 @@ A `-Trusted` switch indicates whether to prompt the user when installing resourc from that repository. By default, if this switch is not specified, the repository is untrusted. A `-Repositories` parameter will accept an array of hashtables equivalent to -the parameter names. +the parameter names (like splatting). + +```powershell +Register-PSResourceRepository -Repositories @( + @{ URL = "https://one.com"; Trusted = $true } + @{ URL = "https://powershellgallery.com"; Trusted = $true; Default = $true } +) +``` + A `-Name` parameter allows for setting a friendly name. A `-Default` switch will set one repository as the default (when not specified @@ -140,8 +157,6 @@ with other cmdlets). Each time it is specified, it sets the new one as default and the previous default is no longer default. -A `-Scope` parameter will support `AllUsers` and `CurrentUsers`. - `Get-PSResourceRepository` will list out the registered repositories. `Set-PSResourceRepository` can be used to update a repository URL, trust level, @@ -182,6 +197,10 @@ Other types will use `Save-PSResource` (see below). A `-Repository` parameter accepts a specific repository name or URL to the repository: ```powershell +Install-PSResource myModule -Repository 'https://mygallery.com' + +# or + Install-Module myModule -Repository 'https://mygallery.com' ``` @@ -210,6 +229,7 @@ as a differently named module already on the system. A `-Quiet` switch will suppress progress information. `-Scope` with `AllUsers` and `CurrentUser` will work the same as it does in v2. +Default is `CurrentUser`. `Install-Module` cmdlet will be retained for compatibility with v2. @@ -292,6 +312,9 @@ A `-Prerelease` switch allows saving prerelease versions. If the `-Version` includes a prerelease label like `2.0.0-beta4`, then this switch is not necessary and the prerelease version will be installed. +A `-AsNupkg` switch will save the resource as a nupkg (if it was originally a +nupkg) instead of expanding it into a folder. + ### Updating resources `Update-PSResource` will update all resources to most recent minor version by default. @@ -312,6 +335,9 @@ A `-DestinationPath` can be used to publish the resulting nupkg locally. A `-Nuspec` parameter can be used to specify a nuspec file rather than relying on this module to produce one. +A `-SkipDependenciesCheck` switch can be used to bypass the default check that +all dependencies are present. + ### Listing installed resources `Get-PSResource` will list all installed resources with new `Type` column. @@ -332,6 +358,10 @@ Version Name Type Repository Description >[!NOTE] >Uninstalling dependencies automatically will be something to consider in the future. +### Proxy support + +Each cmdlet will have `-Proxy` and `-ProxyCredential` parameters. + ### PowerShellGallery status Upon failure to connect to PSGallery, the cmdlets should retrieve status from @@ -343,9 +373,9 @@ This RFC does not cover the module authoring experience on publishing a cross-pl module with multiple dependencies and supporting multiple runtimes. If there is a desire to explicitly update the local cache (like `apt`), we can introduce a -`Update-PSResourceCache` cmdlet with property on a PSRepository indicating whether -it auto-updates or not. -This would not be a breaking change. +`Update-PSResourceCache` cmdlet in the future and a property on PSRepository registrations +indicating whether it auto-updates or not. +This would not be a breaking change, but not part of this initial release. The ability to set policy for PSGet is outside the scope of this RFC. From bd58ae8bc1559e6fd635aec1cb2c90c8a3e2c0b6 Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Wed, 19 Jun 2019 17:33:33 -0700 Subject: [PATCH 08/11] address feedback --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 47 +++++++++++-------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index 8949b85f..f80b45d9 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -132,7 +132,7 @@ always connect to that repository. ### Repository management `Register-PSResourceRepository` will allow for registering additional repositories. -A `-Default` switch enables registering PSGallery should it be accidentally removed. +A `-PSGallery` switch enables registering PSGallery should it be accidentally removed. The `-URL` will accept the HTTP address without the need to specify `/api/v3` as that will be assumed and discovered at runtime (trying v3 first, then falling back to v2, then the literal URL). @@ -145,22 +145,22 @@ the parameter names (like splatting). ```powershell Register-PSResourceRepository -Repositories @( - @{ URL = "https://one.com"; Trusted = $true } - @{ URL = "https://powershellgallery.com"; Trusted = $true; Default = $true } + @{ URL = "https://one.com"; Name = "One"; Trusted = $true; Credential } + @{ URL = "https://powershellgallery.com"; Name = "PSGallery"; Trusted = $true; Default = $true } ) ``` A `-Name` parameter allows for setting a friendly name. -A `-Default` switch will set one repository as the default (when not specified -with other cmdlets). -Each time it is specified, it sets the new one as default and the previous default -is no longer default. +A `-Priority` parameter allows setting the search order of repositories. +A lower value has higher priority. +If not specified, the default value is 50. +PSGallery will have a default value of 25. `Get-PSResourceRepository` will list out the registered repositories. `Set-PSResourceRepository` can be used to update a repository URL, trust level, -or if that repository is the default. +or priority. `Unregister-PSResourceRepository` can be used to un-register a repository. @@ -318,8 +318,10 @@ nupkg) instead of expanding it into a folder. ### Updating resources `Update-PSResource` will update all resources to most recent minor version by default. -A `-AllowMajorVersionUpdate` switch will allow updating to newer major version. -A `-PatchUpdateOnly` switch will allow updating only to patched versions. + +A `-UpdateTo` parameter has values `MinorVersion` (as default), `MajorVersion`, `PatchVersion`. +`MajorVersion` allows updating to latest major version. +`PatchVersion` allows updating to latest patch version (e.g. 6.2.1 to 6.2.4). If the installed resource is a pre-release, it will automatically update to latest prerelease or stable version (if available). @@ -369,15 +371,22 @@ a well known REST API and return a more descriptive error message to the user. ## Alternate Proposals and Considerations -This RFC does not cover the module authoring experience on publishing a cross-platform -module with multiple dependencies and supporting multiple runtimes. +These are items are outside the scope of this RFC and version 3.0. +Many of these items can be addressed in a future version of PowerShellGet without +introducing a breaking change: + +- This RFC does not cover the module authoring experience on publishing a cross-platform + module with multiple dependencies and supporting multiple runtimes. + +- If there is a desire to explicitly update the local cache (like `apt`), we can introduce a + `Update-PSResourceCache` cmdlet in the future and a property on PSRepository registrations + indicating whether it auto-updates or not. + +- The ability to set policy for PSGet -If there is a desire to explicitly update the local cache (like `apt`), we can introduce a -`Update-PSResourceCache` cmdlet in the future and a property on PSRepository registrations -indicating whether it auto-updates or not. -This would not be a breaking change, but not part of this initial release. +- Automatic cleanup of old resources when upgrading or removing resources -The ability to set policy for PSGet is outside the scope of this RFC. +- Signing incorporated into `Publish-PSResource` -Automatic cleanup of old resources is out of scope of this RFC. -Keeping with current behavior, installs are always side-by-side. +- `-Trusted` switch to `Install-PSResource` and `Find-PSResource` that pre-validates + the signing cert is trusted on the system From f271cef2421f6a29c779a5bef0117d9beec33006 Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Thu, 20 Jun 2019 08:14:16 -0700 Subject: [PATCH 09/11] add note to notify users of new versions of modules in future --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index f80b45d9..56174864 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -372,7 +372,7 @@ a well known REST API and return a more descriptive error message to the user. ## Alternate Proposals and Considerations These are items are outside the scope of this RFC and version 3.0. -Many of these items can be addressed in a future version of PowerShellGet without +Many of these items can be addressed in a future version of PowerShellGet 3.x without introducing a breaking change: - This RFC does not cover the module authoring experience on publishing a cross-platform @@ -390,3 +390,5 @@ introducing a breaking change: - `-Trusted` switch to `Install-PSResource` and `Find-PSResource` that pre-validates the signing cert is trusted on the system + +- Notify users that they have outdated versions of a module (just patch versions?) From 6be130f176c7c452dafa550b85285152cf2a82fa Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Wed, 3 Jul 2019 13:49:40 -0700 Subject: [PATCH 10/11] address latest feedback --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 50 ++++++++++++++----- 1 file changed, 37 insertions(+), 13 deletions(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index 56174864..56ec83cd 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -73,12 +73,13 @@ Script and module metadata will retain the same format as it exists with v2. ### Local cache -Instead of always connecting to PSGallery to perform an online search, +Instead of always connecting to a repository to perform an online search, `Find-PSResource` (see below) works against a local cache. This will also enable changes in PowerShell to use `Find-PSResource -Type Command` to look in this cache when it can't find a command and suggest to the user the module to install to get that command. -This will be a local json file containing only the latest version of each module. +This will be a local json file (one per repository) containing sufficient metadata +for searching for resources and their dependencies. The cache will be stored in the user path. There is no system cache that is shared. A system wide cache would require elevation or sudo to create and update preventing @@ -133,20 +134,26 @@ always connect to that repository. `Register-PSResourceRepository` will allow for registering additional repositories. A `-PSGallery` switch enables registering PSGallery should it be accidentally removed. +This would be in a different parameter set from `-URL` and `-Repositories`. + The `-URL` will accept the HTTP address without the need to specify `/api/v3` as that will be assumed and discovered at runtime (trying v3 first, then falling back to v2, then the literal URL). Support for local filesystem repositories must be maintained. A `-Trusted` switch indicates whether to prompt the user when installing resources from that repository. +Trusted repositories automatically have a priority of 0. +`-Trusted` is a different parameter set from `-Priority`. + By default, if this switch is not specified, the repository is untrusted. A `-Repositories` parameter will accept an array of hashtables equivalent to the parameter names (like splatting). ```powershell Register-PSResourceRepository -Repositories @( - @{ URL = "https://one.com"; Name = "One"; Trusted = $true; Credential } + @{ URL = "https://one.com"; Name = "One"; Trusted = $true; Credential = $cred } @{ URL = "https://powershellgallery.com"; Name = "PSGallery"; Trusted = $true; Default = $true } + @{ URL = "\\server\share\myrepository"; Name="Enterprise"; Trusted = $true } ) ``` @@ -154,8 +161,8 @@ A `-Name` parameter allows for setting a friendly name. A `-Priority` parameter allows setting the search order of repositories. A lower value has higher priority. -If not specified, the default value is 50. -PSGallery will have a default value of 25. +If not specified, the default value is 25. +PSGallery will have a default value of 50. `Get-PSResourceRepository` will list out the registered repositories. @@ -210,9 +217,10 @@ trusted repository with the highest version matching the `-Version` parameter (if specified, otherwise newest non-prerelease version unless `-Prerelease` is used). If there are no trusted repositories matching the query, then the newest version -fulfilling the query will be prompted to be installed. -If there are multiple repositories with the same trust level containing the same -version, the first one is used. +fulfilling the query will be prompted to be installed from the highest priority +repositories. +If there are multiple repositories with the same priority level containing the same +version, the first one is used and will be prompted to install. `-TrustRepository` can be used to suppress being prompted for untrusted sources. `-IgnoreDifferentPublisher` can be used to suppress being prompted if the publisher @@ -286,11 +294,21 @@ compatibility with existing published modules using that format. Due to the differences between semver and `System.Version`, we may have to keep the two distinct. -Dependent declared modules not found on the system will prompt to be installed -including module name, version, and size unless `-IncludeDependencies` is -specified which will install without prompting. -Declared dependencies are only searched within the same repository as the original -module to be installed. +If the resource requires dependencies that are not already installed, then +a prompt will appear _before anything is installed_, listing all the resources +including module name, version, size, and repository to be installed unless +`-IncludeDependencies` is specified which will install without prompting. +Rejecting the prompt will result in nothing being installed. + +Declared dependencies are searched and installed using the same trust algorithm +as described for `Install-PSResource` above. + +>[!NOTE] +>Installing dependencies is following the `apt` experience in that it prompts +>by default instead of automatically installing dependencies. Since dependencies +>can be quite large, this would be equivalent to `ConfirmImpact=High` which +>would prompt by default. +>Dependency installation works with `-DestinationPath` parameter. ### Saving resources @@ -392,3 +410,9 @@ introducing a breaking change: the signing cert is trusted on the system - Notify users that they have outdated versions of a module (just patch versions?) + +- A way to filter resources to those allowed to be installed on the system + +- Enterprise management of local cache + +PowerShell module loading needs to be updated to [understand semver](https://github.com/PowerShell/PowerShell/issues/2983). From 08bc32943beeaad9b10a67225bb4bcd90867d80d Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Mon, 12 Aug 2019 16:29:23 -0700 Subject: [PATCH 11/11] address feedback --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 33 ++++++++++--------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index 56ec83cd..5ef11688 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -161,7 +161,7 @@ A `-Name` parameter allows for setting a friendly name. A `-Priority` parameter allows setting the search order of repositories. A lower value has higher priority. -If not specified, the default value is 25. +If not specified, the default value is 50. PSGallery will have a default value of 50. `Get-PSResourceRepository` will list out the registered repositories. @@ -241,10 +241,8 @@ Default is `CurrentUser`. `Install-Module` cmdlet will be retained for compatibility with v2. -A change from v2 is that installed modules use the full semver (if used) of -the module, so PSReadLine would be in a folder called `2.0.0-beta4` instead of -just `2.0.0` which resolves issues of attempting to overwrite assemblies currently -loaded on Windows. +v3 will continue to use the versioning folder format as v2 in that it will be +Major.Minor.Patch and not contain the semver prerelease label. ### Dependencies and version management @@ -261,6 +259,7 @@ Install-PSResource -RequiredResources @{ version = "[4.4.2,4.7.0]" repository = "https://www.powershellgallery.com" credential = $cred + allowPrerelease = $true } } ``` @@ -269,20 +268,19 @@ The json format will be the same as if this hashtable is passed to `ConvertTo-Js ```json { + "Configuration": "[1.3.1,2.0)", "Pester": { "version": "[4.4.2,4.7.0]", "credential": null, - "repository": "https://www.powershellgallery.com" - }, - "Configuration": "[1.3.1,2.0)" + "repository": "https://www.powershellgallery.com", + "allowPrerelease": true + } } ``` >[!NOTE] ->Credentials management will be using an upcoming RFC for generalized ->credential management. Not sure how to make that work in json other ->than requiring json to have credential as a json object but that means ->password/secret is in clear text. +>Credentials management will be using [Secrets Management RFC](https://github.com/PowerShell/PowerShell-RFC/pull/208) +> for generalized credential management. The `repository` property can be the name of the registered repository or the URL to a repository. @@ -291,9 +289,6 @@ If no `repository` is specified, it will use the `default` repository. The older `System.Version` four part version type will be supported to retain compatibility with existing published modules using that format. -Due to the differences between semver and `System.Version`, we may have to keep -the two distinct. - If the resource requires dependencies that are not already installed, then a prompt will appear _before anything is installed_, listing all the resources including module name, version, size, and repository to be installed unless @@ -415,4 +410,12 @@ introducing a breaking change: - Enterprise management of local cache +- Full semver support won't happen until there is a semver type in .NET + +- Consider using a merkle tree to validate modules + PowerShell module loading needs to be updated to [understand semver](https://github.com/PowerShell/PowerShell/issues/2983). + +Instead of `Find-PSResource`, call it `Find-Resource`. +Context may be understandable since the cmdlet comes from PowerShellGet, but +`Resource` as a noun is pretty general.