Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating the PowerShellGet 3.0 RFC #237

Closed
wants to merge 14 commits into from

Conversation

SydneyhSmith
Copy link
Collaborator

Publishing an updated version of the PowerShellGet 3.0 RFC based on #185

We are currently targeting a preview of module (not feature complete) for end of year 2019.

@ThomasNieto
Copy link

@SydneyhSmith please add a change log to your comment for what has changed from the original.

@Stephanevg
Copy link

Stephanevg commented Jan 10, 2020

Hi @SydneyhSmith Is this one already available for testing? I have a strong business case for this one, and I Couldn't find anything on the gallery.

@SydneyhSmith
Copy link
Collaborator Author

Hi @SydneyhSmith Is this one already available for testing? I have a strong business case for this one, and I Couldn't find anything on the gallery.

Not yet, we are currently implementing it and targeting a public preview on the gallery (not feature complete) for the end of this month (Jan 2020)--thanks for the patience!

@Stephanevg
Copy link

Hi @SydneyhSmith Is this one already available for testing? I have a strong business case for this one, and I Couldn't find anything on the gallery.

Not yet, we are currently implementing it and targeting a public preview on the gallery (not feature complete) for the end of this month (Jan 2020)--thanks for the patience!

Great, looking forward to it @SydneyhSmith
We are particularly interested in the possibility of register repositories system-wide. With the current available version of the module (2.2.3) this is not possible.
Each user needs to register the repository them self using Register-Repository.

There has been quite some disucssions on this topic here -> https://github.com/PowerShell/PowerShellGet/issues/214 and I had the impression this was taken into consideration for version 3.0 of the module.
Unfortunatley I couldn't really see a reference to registering system wide repositories in the RFC.
Is this something you guys are planning to have as a feature, and I simply missed it?

This module will depend on https://www.nuget.org/packages/NuGet.Client.
This module would be shipped in the PSGallery supporting older
versions of PowerShell.
This module will ship in PowerShell 7.1.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please clarify: Will it ship in addition to the exiting PowerShellGet 2.x or be a replacement? I understand they can work side-by-side but knowing what the 'out of the box' modules are is important for CI scenarios where installation of certain modules is assumed. Since the cmdlet names are different, removing PowerShellGet 2.x would be a big breaking change

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is to ship in addition to PowerShellGet 2.x for the 7.1 timeframe, and before that we will be shipping it in the Gallery as a standalone module to work out issues before we ship in PowerShell


### Local cache

Instead of always connecting to a repository to perform an online search,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whilst this would be nice I do not see it being required for the first minimum, viable version.

@SydneyhSmith
Copy link
Collaborator Author

We are particularly interested in the possibility of register repositories system-wide. With the current available version of the module (2.2.3) this is not possible.
Each user needs to register the repository them self using Register-Repository.

There has been quite some disucssions on this topic here -> PowerShell/PowerShellGet#214 and I had the impression this was taken into consideration for version 3.0 of the module.
Unfortunatley I couldn't really see a reference to registering system wide repositories in the RFC.
Is this something you guys are planning to have as a feature, and I simply missed it?

@Stephanevg thanks for the question, unfortunately this is not something that will ship in the first version of PowerShellGet 3.0, but is something we will highly consider for a vNext...I will update the RFC to clarify this point--sorry to disappoint!

Comment on lines 118 to 120
On any operation that reaches out to a repository, a REST API will be called to
see if the hash of the cache matches the current cache and if not, a new one
is downloaded.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure that this is the best idea....we might want to do something similar to how the update notification feature works (check a file to see if it's already been done in the last 24 hours)

`-Trusted` is a different parameter set from `-Priority`.

By default, if this switch is not specified, the repository is untrusted.
A `-Repositories` parameter will accept an array of hashtables equivalent to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-Repositories should be Repository

### Dependencies and version management

`Install-PSResource` can accept a path to a psd1 or json file (using `-RequiredResourcesFile`),
or a hashtable or json (using `-RequiredResources`) where the key is the module name and the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RequiredResource

Copy link

@Stephanevg Stephanevg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added comments on system wide repository registration - which is crucial for us

Proposal is to write PSResource in C# to reduce complexity and make easier to maintain.
In addition, remove dependency on PackageManagement completely as well as dependency on
nuget.exe.
This module will depend on https://www.nuget.org/packages/NuGet.Client.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this mean that users need to go and download nuget.exe first like we have to do today, or is it planned to add the dependency directly in the module?
If not, it would be nice to have that download part automated as well.

Copy link

@kilasuit kilasuit May 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reads to me that the Nuget client dll's needed will be shipped with the module


- A way to filter resources to those allowed to be installed on the system

- Enterprise management of local cache/the ability to configure a PSRepository on a system-wide level

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this list set in a priority order? Because having the possibility to set a repository system-wide is really some strongly needed for us.
We would benfit way more from having a repository system wide, than from having a message saying we have don't have the latest module installed, or from a cleanup of old modules (which we intentionnally dont remove for backwards compatability / fallback scenarios).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and for v3 GA this is a critical need to address the long standing issue that v2 has had requiring per user setup which doesn't work for systems that require shared management at all

`Update-PSResourceCache` first and then attempt a search.
However, this cmdlet does not update the cache if one exists.

A `-Version` paramter will accept a string of a [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parameter has a typo.


A `-Prerelease` switch will allow include prerelease resources in the results.

A `-ModuleName` paramter will accept a string, and will return commands, DSC resources, and Role Capabilities found in modules with that name.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parameter has a typo.


A `-ProxyCredential` parameter will accept a PSCredential

A `-Credential` paramter will accept a PSCredential.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parameter has a typo.

…xx-PowerShellGet-3.0.md

Moving to Draft-Accepted folder
@TylerLeonhardt
Copy link
Member

Since PSGet v3 dropped the Install-Module cmdlet, this means we can no longer module qualify commands that only exist in v2:

PowerShellGet\Install-Module ...

This works pre-v3, but if I have v3 and v2 installed, it no longer works because v3 does not have this cmdlet.

I think having Install-Module available by default will be less breaking of muscle memory.

on this module to produce one.

A `-SkipDependenciesCheck` switch can be used to bypass the default check that
all dependencies are present.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... all dependencies are present where? on the local system or in the target repository?

Comment on lines 316 to 320
If the resource requires dependencies that are not already installed, then
a prompt will appear _before anything is installed_, listing all the resources
including module name, version, size, and repository to be installed unless
`-IncludeDependencies` is specified which will install without prompting.
Rejecting the prompt will result in nothing being installed.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc talks about resources that have dependencies but does not mention how will we handle modules that depend on a nuget library.

If we can define the nuget dependency in the published module, what happens when the user installs the module with out a -DestinationPath? Are they prevented from installing it because they can't save the nuget package? How would the user know how to handle that situation or should they be responsible to solve it? Or if they do specify a -DestinationPath that is their module folder and end up with libraries all over the root of the folder?

If they do install with the module, where do they go? What would module authors have to do to use those libraries in that module? Will they drop into a standard shared location that's in the $env:path or a module's lib folder that can be referenced with the psd1? Or would they have to hand craft the loading logic in the psm1 or the scripts to process using?

Or is this out of scope and modules authors will continue to embed the libraries inside their module like they do today and distribute it as one (large) module? Or download them on demand from the psm1 with Install-PSResource?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some built in handling of nuget dependencies would be a huge benefit, would love to see that.

In order to open source PowerShellGet 3.0 we will fork the current PowerShellGet repo, rename it as PowerShellGetv2,
and use the existing PowerShellGet repo as the primary support and development contact for PowerShellGet 3+.

## Alternate Proposals and Considerations

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we leave the door open for custom repository providers or other sources? I am really interested in using blob storage as a module repository source. I'm seeing more and more multi account cloud configurations where network connectivity to a central nuget server doesn't work out very well.

@kilasuit
Copy link

@TylerLeonhardt with this

Since PSGet v3 dropped the Install-Module cmdlet, this means we can no longer module qualify commands that only exist in v2:

PowerShellGet\Install-Module ...

This works pre-v3, but if I have v3 and v2 installed, it no longer works because v3 does not have this cmdlet.

I think having Install-Module available by default will be less breaking of muscle memory.

if this Issue PowerShell/PowerShell#8949 can get resolved in v7 this will not be that much of an issue but until then the syntax in this comment PowerShell/PowerShell#8949 (comment) would provide a possible middle ground (& works on all PSversions too)

@TylerLeonhardt
Copy link
Member

Just keep in mind... We do use the module qualified syntax in the PowerShell extension and it still has to support 5.1 for the foreseeable future.

@vexx32
Copy link
Contributor

vexx32 commented May 23, 2020

Pester also makes heavy use of module-qualified commands, from what I recall of how it builds its safe commands table, as well.

Copy link

@kilasuit kilasuit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added my comments to this version of the RFC

nuget.exe.
This module will depend on Nuget Server and Client packages including https://www.nuget.org/packages/NuGet.Client.
This module would be shipped in the PSGallery supporting PowerShell 5.1+.
This module will likely ship in PowerShell 7.1 alongside PowerShellGet 2.x.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer we didn't ship 2.x and 3 in 7.1 but know this will be needed for back compat until PowerShellGet v3 is the dominant version in use.

Unless

You bundle into PowerShellGetv3 a Install-Module wrapper that takes in all the same Params as Install-module in v2 and then spins them off to Install-PSResource - which would be the better User experience and would not break many thousands (perhaps millions ?) of build scripts or at least initiations of those scripts & IMO this needs to be baked in to the GA of PowerShellGet v3.

This will also enable changes in PowerShell to use `Find-PSResource -Type Command` to look
in this cache when it can't find a command and suggest to the user the module to install to
get that command.
This will be a local json file (one per repository) containing sufficient metadata

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depending on the size of the reposistory this likely could become a large file. do you have an estimate of how big this file would be for the PowerShell Gallery as that's the most common repository in use in the wild.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I note this was mentioned further down in the RFC

Comment on lines 76 to 79
The cache will be stored in the user path.
There is no system cache that is shared.
A system wide cache would require elevation or sudo to create and update preventing
it from being useful.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Considering lots of organisations running PowerShell install Modules to the machine level & not the user level we need the option of having this at both levels and with a preference for a location like ProgramData in Windows where it can be stored once and shared between user level and machine level, which I don't think should require admin rights on Windows, I can't speak for the linux side

Comment on lines 101 to 109
An example cache with 5000 resources (approximately the number of unique packages
currently published to PowerShellGallery.com) is ~700KB in compressed json form.

The cache would have both latest stable and latest prerelease versions of resources.

>[!NOTE]
>Need to experiment if it makes sense to have a single cache file for all repositories
>or a different cache per repository for size and perf reasons.
>Perf tests will determine if the cache needs to be in binary form.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are currently over 47k packages (when you consider versions) & many orgs aren't & shouldn't be blindly updating to the latest stable version without being sure this doesn't break them, which is why I think that we need to at least have the latest minor versions for each major version

So that this would show both of the following examples.
PowerShellGet v2.2.4
PowerShellGet v3.1.0

but should also show up
PowerShellGet v2.2.5-preview2
PowerShellGet v3.1.1-preview5

I think that makes most sense for most orgs.

If the file ends up being Binary, instead of JSON then this would stop other ways of inspection of that file via scripted methods that would include anyone wanting to look at deltas between last cache file on disk & any subsequent updates - something for the likes of DSC Resources this is something I have a need in future to do.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also & I just finished testing this, to get a single output from the Gallery now of all the Modules & versions took over 2hours and at well over 5.5k individual calls of Find-Module to the gallery to get all the info required for all modules in the Gallery, so therefore it would be good if we could get the option for both the slim file with the suggestions above, and the full fat file with everything as a choice.

Comment on lines 81 to 98
Example cache entry:

```json
{
"name": "This is my module",
"exportedFunctions": [
"Get-One",
"Set-Two"
],
"version": "1.0.0.0",
"type": "module",
"tags": [
"Linux",
"PSEdition_Core",
"PSEdition_Desktop",
"AzureAutomation"
]
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this makes for a goodish initial spec though there is so much useful additional properties that come back in the AdditionalMetadata field like below that many don't know about that would be good to be also exposed - depending on the file size cost

image

Comment on lines +269 to +271
v3 will continue to use the versioning folder format as v2 in that it will be
Major.Minor.Patch and not contain the semver prerelease label.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This I think should allow use of the pre-release label to enable side by side comparisons of pre-releases like we can for general releases, especially during the developing of these pre-release.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could add the pre-release label in 7.1 and continue to use the old format for everything previous. At least get support for the pre-release label in 7.1 so we have the option to adopt this later.

Comment on lines 306 to 311
>[!NOTE]
>Credentials management will be using [Secrets Management RFC](https://github.com/PowerShell/PowerShell-RFC/pull/208)
> for generalized credential management.
The `repository` property can be the name of the registered repository or the URL
to a repository.
If no `repository` is specified, it will use the `default` repository.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that SecretsManagement will be a dependency for PowerShellGet v3?
Also what other dependencies are expected?
Will these be correctly listed in the off case that a component you intend to use now is in a module that in future is not bundled in PowerShell directly as per this possible change that may come in a future PowerShell Release PowerShell/PowerShell#5681

Comment on lines 325 to 330
>[!NOTE]
>Installing dependencies is following the `apt` experience in that it prompts
>by default instead of automatically installing dependencies. Since dependencies
>can be quite large, this would be equivalent to `ConfirmImpact=High` which
>would prompt by default.
>Dependency installation works with `-DestinationPath` parameter.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to have the ability to push though for CI via use of either the force parameter or -Confirm:$false but should not need both

Comment on lines 332 to 333
We will also introduce a `New-RequiredResourceFile` cmdlet which will create a template file.
If the switch `-AsPSD1` is used it will create a psd1 file, otherwise it will default to json.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the default to json, psd1 as a powershell data file makes more sense

Comment on lines +421 to +422
In order to open source PowerShellGet 3.0 we will fork the current PowerShellGet repo, rename it as PowerShellGetv2,
and use the existing PowerShellGet repo as the primary support and development contact for PowerShellGet 3+.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I so wish this had not been conducted this way as it is a terrible user experience for current users of PowerShellGetv2 including those like me that had it cloned already

Forking and calling it PowerShellGetv3 would have been a much better option, but generally a v3 branch & the branches off of this within the original repository generally makes the most sense for development & should be adhered to in future splits of modules where possible.

@Stephanevg
Copy link

Hi, @SydneyhSmith now that winget got announced , how will powershellget integrate / influence / evolve with winget?
I couldn't find a reference on winget in this RFC.
wouldn't it make sense to think about their relation in this RFC, and define their respective scopes, and where they overlap ?

@vexx32
Copy link
Contributor

vexx32 commented May 25, 2020

@Stephanevg hard to say tbh. The WinGet team have more or less said they're not interested in integrating with PowerShell directly at the present time in this issue (despite quite a few folks asking for some kind of support): microsoft/winget-cli#221

Probably the PS team and the WinGet team will need to have a chat about that before they can make anything clear here.

Updated based on current scope for PowershellGet 3.0 release. 
Includes the following updates: 
- Plans to ship in PowerShell 7.2 
- CompatPowerShellGet compatibility module 
- No `-Filter` parameter for find and install
- Integration with SecretManagement moved to post-GA
- Install-PSResource installs dependencies by default 
- New-RequiredResourceFile cmdlet moved to post-GA
- Save-PSResource `-Type Library` and `-IncludeAllRuntimes` moved to post-GA
- Update-PSResoure `-UpdateTo` parameter moved to post-GA
- Local Cache support moved to post-GA
@kilasuit
Copy link

kilasuit commented Sep 16, 2020

As part of the plan for post GA of v3 can I ask for this RFC to also include details about plans to get v3 (& the compat module) into Windows 10 & Server 2019.

I say this because its 2020, and Windows 10 version 2004 still ships with the awful 1.0.0.1 version by default.

I know that @SteveL-MSFT @joeyaiello amd team will say "it's hard getting things into Windows" but this needs to be part of the plan (even if that takes 6-12 months after PowerShellGet v3 is GA)

@JamesWTruher
Copy link
Contributor

@PowerShell/powershell-committee
Work in the PowerShellGet project is ongoing in that repo and is not tracked in this RFC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.