Multi line continuation

[KirkMunro]

This is an RFC to add support for multi-line commands in PowerShell so that users can wrap commands across multiple lines whether they use named parameters, splatted collections, or the stop-parsing sigil, all while continuing to have Intellisense and tab completion support, without having to use backticks or splatting.

The original idea behind this was posted in PR #176. After a lot of early feedback, and some things learned in the process that were not apparent up front it made sense to switch to this proposal. I apologize for having separate PRs but it changed dramatically enough that it made more sense to close the old one which started out with a different purpose and start from a clean slate.

[dragonwolf83]

I have a few use cases to further the sigil vs enclosures debate.

Scenario 1

First use case is whether or not a sigal will parse correctly when a parameter's value is turned into multiple lines. It sounds like it should work with the sigal. The example below takes the AccountPassword in the RFC and turns it into multiple lines.

New-ADUser @
    -Name 'Jack Robinson'
    -GivenName 'Jack'
    -Surname 'Robinson'
    -SamAccountName 'J.Robinson'
    -UserPrincipalName '[email protected]'
    -Path 'OU=Users,DC=enterprise,DC=com'
    -AccountPassword (
          Read-Host -AsSecureString 'Input Password'
      )
    -Enabled $true

Scenario 2

Take that example further, can we do nested sigals if I want to then turn that Read-Host into multiple lines?

New-ADUser @
    -Name 'Jack Robinson'
    -GivenName 'Jack'
    -Surname 'Robinson'
    -SamAccountName 'J.Robinson'
    -UserPrincipalName '[email protected]'
    -Path 'OU=Users,DC=enterprise,DC=com'
    -AccountPassword (
          Read-Host @ 
              -AsSecureString 'Input Password'
              -Verbose
      )
    -Enabled $true

Scenario 3

The third use case is to use newlines to visually separate out newlines for easier readability. This tends to be more useful if you do multiple lines for several parameters and to space them, either for grouping or to not make the code look so dense. They could also be used for adding comments

New-ADUser @
    -Name 'Jack Robinson'
    -GivenName 'Jack'
    -Surname 'Robinson'
    -SamAccountName 'J.Robinson'    

    -UserPrincipalName (        
        '[email protected]'
    )

   # Get the list of regions for where a user would reside in to put the user into the correct region    
    -Path (            
        $region = Get-Region -Name 'Jack Robinson'
        "OU=Users,OU=$region,DC=enterprise,DC=com"
    )

    -AccountPassword (
        Read-Host -AsSecureString 'Input Password'
    )
    
    -Enabled $true

Get-ChildItem @
    $rootFolder    
    -File
    -Filter '*.ps*1'

Which of these scenarios would work with just using a sigil? Scenario 3 seems like it would only work with an enclosure but not sure about the other 2.

[KirkMunro]

Great questions @dragonwolf83.

Scenarios 1 and 2 would work as described with just a sigil.

Scenario 3 would work with comments injected on lines between the parameters/arguments, but blank lines/whitespace-only lines at the same level (i.e. not blank lines inside subexpressions/brackets/script block enclosures used to define or calculate values -- those would be parsed as part of the subexpression/bracketed expression/script block, so they should be ok) would be interpreted as lines terminating the multi-line command. To also be able to have blank lines among the parameters/arguments, we'd have to go with an enclosure instead.

[KirkMunro]

Just to call it out so that others don't have to go look it up, someone recently requested (in that Issue I just referenced) being able to continue lines with a dot, so that you can do something like this:

$string
    .ToUpper()
    .Trim()

That doesn't work in PowerShell today. Instead, you can end your lines with a dot-reference operator, like this:

$string.
    ToUpper().
    Trim()

Unfortunately, that syntax is not considered as easy to read by many users in the development community who prefer leading lines with a dot-reference operator to terminating them with one. For those users, if we use the @ multi-line continuance operator (aka line splatting operator) at the end of a line, users could do this:

$string @
    .ToUpper()
    .Trim()

With that syntax, the command continues until a statement terminator or a blank line, so an @ at the end of a line tells the parser to treat all subsequent lines as part of the same command until a statement terminator or a blank line is reached.

[KirkMunro]

@rjmholt Agreed, and updated in the PR title and the RFC doc. Thanks!

[KirkMunro]

@SteveL-MSFT This has collected enough feedback and incorporated it into the RFC. I think it's ready for committee review/public comment.

[mklement0]

Late to the party, but a few quick thoughts:

Love the idea of the main proposal (line-ending @) - low on ceremony, easy to grasp. (Stack Overflow answers too would benefit from this; currently, there are many examples of mile-wide one-liners and backtick-fests with broken syntax highlighting.)

Yes, the need to then explicitly signal the end of such a multi-line statement is a new requirement, but:

• people who use the new feature can be assumed to be advanced users who understand this requirement.

• people who use the feature probably care about readability, where using an extra blank line after the command makes sense anyway

• as you state, a terminating ; is sufficient if the desire is truly to place the next command on the very next line

[KirkMunro]

Thanks @mklement0, nice to have the additional confirmation that there are folks who like the idea of a single line-ending @.

It's not a completely foreign/unique concept. YAML uses this with indentation to create multi-line scalars, where the symbol at the end of the line dictates whether newlines will be converted to spaces or kept as newlines.

I'm not holding my breath on this one though, because even though there has been mainly positive feedback from the community so far, based on the notes from this RFC discussion and Jim's comments earlier the folks who call the shots for PowerShell aren't supportive of the idea.

[HumanEquivalentUnit]

Off-topic comment, but @mklement0 :

Stack Overflow answers too would benefit from this; currently, there are many examples of mile-wide one-liners and backtick-fests with broken syntax highlighting

StackOverflow uses Google Prettify which falls back on "generic C-like language" for PowerShell - at least it dit a while back, I can't see that's changed - googlearchive/code-prettify#458 and amroamroamro's comment; new language additions to PowerShell seem unlikely to make StackOverflow highlighting much better except by coincidence.

[mklement0]

Thanks, @HumanEquivalentUnit.
Yes, there is no PowerShell-specific syntax highlighting (if there were, line-continuation backticks wouldn't be a highlighting concern), but the fallback highlighting mostly works well, except in two cases:

• unquoted ` chars. - the need for which this RFC mostly eliminates
• string literals that end with \" or \' (no direct solution, but you can post an off-screen #" or #' comment to fix that, if you want to go the extra mile) - this typically only comes up with Windows paths ending in \, and can be worked around (omit trailing \ or switch to /)

Here's proof that what this RFC proposes would work well with the existing highlighting:

[joeyaiello]

@PowerShell/powershell-committee reviewed this today with @SteveL-MSFT, @JamesWTruher, and me. Generally, we would prefer to implement inline splatting (which is already approved) over a new sigil which would have comparable function but require a completely new construct.

Also, given the difficulty that folks already have with here-strings today (and our desire to implement here-docs), we don't to continue overriding @<foo>.

We still need quorum to make a final decision here, but overall we don't think this is likely going to be approved.

[joeyaiello]

I don't necessarily agree with the statement that backticks should be avoided, especially when used in conjunction with PSSA and the AvoidTrailingWhitespace rule.

I know that's contentious in the community, but a lot of the remaining justification is based on the assertion that backticks aren't desirable, thought it was worth calling out.

[vexx32]

Backticks tend to create fragile code, that is generally intolerant to further maintenance. Trailing whitespace is an easy example, but there are no use cases I've seen where their use is excusable.

If there were a dedicated line continuation token, that's a different story. Since backticks are general-purpose escape characters, I don't think it's appropriate to really use backticks for this purpose. It's the kind of tool that, sure, you can use in a pinch, but should never be a long-term solution. Sooner or later someone's going to break it, and they ways in which these break unfortunately tend not to halt execution, so you can easily end up with things breaking in more ways than just a red error message on the screen, with commands called with only half their parameters, quite by accident.

It also doesn't help that backticks are one of the hardest to spot characters available on a standard US keyboard; very easy to forget to add or remove them when editing commands written in this way.

[dragonwolf83]

Backticks tend to create fragile code,

Agreed. It is a solution but not the best solution to the problem.

This past month I had to modify a vbscript where they use underscores for the continuation character. I made the change, included it at first, but realized I could remove unused code and suddenly my ending line is no longer at the end. Did I add the underscore correctly? Nope!

This is a common pitfall of requiring a line-continuation character after each line.

[KirkMunro]

I've modified the wording to make it more clear that many members consider it to be a syntactical nuisance so that it's not written as an absolute. Thanks for the feedback @joeyaiello, my changes will be in my next commit.

[joeyaiello]

Assuming we implement inline splatting, we believe that this could be implemented using a PSSA rule. E.g. it could say that any number of parameters or column width over some value could trigger a rule with an auto-format to turn the in-line list of params and their values into a splatted inline hashtable.

[KirkMunro]

That may be helpful, but some questions back to you about this:

1. What about folks who don't use PSSA?
2. Would that be in Visual Studio Code only? Or other tools?
3. Once you convert, then what? If someone wants to add a parameter, will they get the same Intellisense? In any PowerShell scripting tool, or only in Visual Studio Code?
4. Depending on the syntax you go with for inline splatting, I may want to avoid that entirely, so hopefully it's configurable. I'll use backticks over @@{...} anyday.

[joeyaiello]

This is mostly the same point as your first in this list, and we think usage and discoverability could be fixed in tooling. Furthermore, new operators are notoriously difficult to discover, so I'm not convinced that you wouldn't be in the same boat with a new sigil.

[mklement0]

Splatting and multi-line continuations are really distinct use cases, and you shouldn't have to use the former to achieve the latter:

• (a) If I want to specify my known parameters as usual, but across multiple lines in a readable fashion, there should be an easy way to do that, and the trailing @ offers just that.

• (b) If I want to construct the (variable) set of parameter values programmatically, I'll use splatting.

(a) strikes me as far more common than (b)

[mklement0]

Another way of looking at it: if we had good support for (a), as proposed, then perhaps we wouldn't need the inline splatting variant at all, as its primary purpose seems to be to (indirectly) address (a)

[KirkMunro]

Another way of looking at it: if we had good support for (a), as proposed, then perhaps we wouldn't need the inline splatting variant at all, as its primary purpose seems to be to (indirectly) address (a)

Exactly this. Inline splatting feels like the wrong solution to the problem. Splatting's intent is not for code readability.

[joeyaiello]

It's a little annoying, but you can still do this with inline splatting by using a subexpression ($()):

Get-Foo @@{
    a = 1
    b = $(
              statement1;
              statement2;
         )
}

[joeyaiello]

This is true, but we believe it's the right work to do: splatting should be better.

[KirkMunro]

Better, yes. Able to support splatting in members or the results of methods, absolutely, there's a PR for that, and that improvement should be merged in. But when you push inline splatting as a way to make line wrapping better, you're violating the single-responsibility principle.

Splatting is for command invocation with dynamic parameters that are chosen based on runtime execution.

Line continuation is about more than just cmdlet or advanced function invocation. Take chained method invocation, for example, as shown earlier in this RFC.

Also, I just remembered what I was trying to say about the stop-parsing sigil...with inline splatting as the way we wrap things, that's only good for cmdlet/advanced function parameters. If I want to wrap arguments passed after a stop-parsing sigil, I can't, no matter how long the line gets. But with this proposal, I would be able to do that.

[joeyaiello]

This one's problematic because it's an anti-pattern to use positional parameters in maintained scripts. While it's perfectly fine to use them on ad hoc / interactive basis, all parameters should be named in scripts (and that point is even more applicable if you're talking about invocations that are long and complex enough to need something like multi-line continuation).

[KirkMunro]

I recognize that; however, in practice, it seems to me that anti-pattern is very common in maintained scripts.

[mklement0]

Indeed it is very common, and I don't think it's an anti-pattern:

Leaving the unfortunate -Path / -LiteralPath schism aside (a separate debate), something like:

Test-Path $somePath

is not only perfectly reasonable, but arguably preferable to the needlessly verbose

Test-Path -Path $somePath

That is, thoughtfully defined cmdlets:

• (a) choose obvious parameters for their positional defaults
• (b) lock those positional defaults in, as a contract, going forward (often there's just one suitable parameter)

With that in place, both interactive and script use benefit from the concision of not having to spell out what is intuitively implied.

[joeyaiello]

This is a breaking change because it changes what the trailing backtick means today. Users could absolutely be splitting lines after @ in the real-world.

[KirkMunro]

This is not breaking. Here is output from preview 6:

Windows PowerShell 5.1 recognizes it as a parser error as well, and since it's a parser error, this is non-breaking as things stand in PowerShell today.

Also, this doesn't change the meaning of the backtick (a single ` token). That behavior remains the same. The portion of this RFC that proposes using enclosures suggests we add new enclosure tokens (@` and `@), and while those tokens have a character in common with the line continuation token (`), they are not that token.

[bpayette]

It seems that the actual root problem here is that backtick as a line continuation character is hard to see. So fix the editor to highlight backtick line continuations. Problem solved. @TylerLeonhardt How hard would this be to do in VSCode?

[dragonwolf83]

It seems that the actual root problem here is that backtick as a line continuation character is hard to see.

I strongly disagree with that. Some might prefer it being more readable, but that is not the crux of the issue. VBScript used underscores, it was more visible but also far more ugly, hard to read, and fragile.

This can be summed up in two questions:

• Why do we need a line continuation character for each line?
• Are there not better solutions to allow wrapping long statements?

This RFC proposes 2 alternative solutions:

• One line continuation character at the beginning to end of whitespace
• An enclosure continuation character to more clearly indicate start and stop

Why are either of these solutions better?

• Easier to read
• Cleaner code
• Less fragile by allowing changes within the enclosure without having to remember to add a special character at the end!
• Easily allows chaining methods as mentioned by Kirk.

I think the RFC does a good job covering why this is better than inline splatting.

[KirkMunro]

@BrucePay This isn't just about backtick visibility. It's about more flexibility and easier maintenance when it comes to spanning commands across multiple lines, without having to `
put `
line `
continuance `
characters `
after `
every `
line.

What you propose may make backticks more visible in Visual Studio Code, but what about everywhere that PowerShell is used -- other shells, other editors, websites, blogs, printed books, etc.? That's not the issue here though, as @dragonwolf83 pointed out above.

Also most folks using en-us keyboard layouts don't realize how much of a pain it is to have to type ` in an editor when using a keyboard layout that supports accented characters.

[Pxtl]

Help me out here - for the enclosure case, why is a special character necessary?

For a single expression, doesn't the standard parens provide enough information that it could be extended to cover multi-line expressions?

(Invoke-MyCommandlet 
 -arg1 "foo" 
 -arg2 "bar" 
 -arg3 "baz" 
)

also works for piping

(
  Invoke-SqlCmd -serverInstance myserver -database mydatabase -query 'select * from mytable'
  | Export-CSV -path $myPathVar -NoTypeInformation
)

and dot stuff

$myObject = new-object SomeNamespace.SomeClass
($myObject
     .foo()
     .bar()
     .baz
)

Basically just ignore any linebreaks that occur inside parens.

After all, all of the above cases would be valid as one-liners with the parens, so the parens are already valid syntax.

(Invoke-MyCommandlet  -arg1 "foo"  -arg2 "bar"  -arg3 "baz")

(Invoke-SqlCmd -serverInstance myserver -database mydatabase -query 'select * from mytable' | Export-CSV -path $myPathVar -NoTypeInformation)

($myObject.foo().bar().baz)

All three of those are currently valid powershell, but would be invalid if you split them onto multiple lines, because of a missing close parens. So to me it seems obvious - an unclosed parens means ignore the line-ending. The only downside I can see is that you'll get much more confusing errors if you leave an unclosed parens, which is the problem we see in C#.

[jhoneill]

Help me out here - for the enclosure case, why is a special character necessary?

Enclosures are both easier to see and less fragile than "continue" characters on each line. And some bracket construction would good IMHO , and I don't think it is excessively difficult to build. {} does
it for a script block already. @{} does it for a hash table already.
() is problematic because it denotes a value

(Invoke-MyCommandlet 
-arg1 "foo" 
-arg2 "bar" 
-arg3 "baz" 
)

is the the result of running the command. It could be implemented today but wouldn't give something which can receive a value. i.e.
get-stuff | (invoke-MyCommandlet ... isn't allowed However

dir | & "C:\Program Files\blah\blah\blah" is so something like

 dir | &( "C:\Program Files\blah\blah\blah.ps1" -arg1
foo
-arg2 
bar 
)

Should be workable.

[mklement0]

less fragile than "continue" characters on each line

True, but note that the RFC's main proposal is a single continuation char. on the first line only.

() is problematic because it denotes a value

More specifically, (...) turns a command into an expression, which has three side effects:

• It can only be used as the first pipeline segment (as stated)
• The command runs to completion first, with all output collected in memory before it is sent through the pipeline.
• If the command outputs a collection as a whole, that collection is enumerated.

Similarly, &(...) (same as: & (...)) is not an option, because it has established semantics that conflict with the proposed one; that is, whatever (...) evaluates to is interpreted as a command name by &:

&(Write-Output 'get-date')  # same as: & 'Get-Date'

[jhoneill]

less fragile than "continue" characters on each line

True, but note that the RFC's main proposal is a single continuation char. on the first line only.

Which from my reading and @JamesWTruher 's comment above isn't a good idea.

This makes PowerShell white space sensitive and requires empty lines. That just seems like a non-starter for me

As fixes go, creating inline splatting (which seems to be the committee's preferred option) is fairly horrible. To avoid writing

command -param1 a `
-param2 b

Write

command @@{
  param1 = "a" 
  param2 = "b"                        
}

Does anyone like that syntax ? When my line gets too long I go back and remove - signs, add = signs and quote strings which didn't need to be quoted before.

Very simply , we have a layout which conveys "end-of-command" : end-of-line. Many languages have a way of saying "this end of line is not end-of-command" _ in vbscript ` in PowerShell etc.

The "rolling continuation" says, in effect "temporarily change the layout convention to indicate end of command. If I indent, or begin the next line with -paramName or <> continue the line". A marker at the end removes the need to do anything to keep the continuation going.

Similarly, &(...) (same as: & (...)) is not an option, because it has established semantics ...

Peril of posting late at night. I should have seen that, but think of &( ) as a style rather a concrete proposal, ¬( ) or €{ } or ~[ ] or something of that kind ...

[Pxtl]

@mklement0 @jhoneill

Okay, now I'm starting to see the drawbacks. I'm not sure about the forcing evaluation, but for the pipelining problem can't that be worked around by wrapping the whole thing in ()? Like, if you need multiline throughout every part of your pipeline, wrap the whole thing in parens?

Like I can still do

$quux = (invoke-foo
  | invoke-bar
     -barParam1 $myBarParam #see, invoke-bar is multiline and is the second step in the pipe!
  | invoke-baz
)

and if we don't want the output

(invoke-foo
  | invoke-baz
) | out-null

In those cases either way the results would've been collected before moving on regardless of the parens, right?

[jhoneill]

@Pxtl
I think that would work.

Actually it doesn't really a big change. IF the value of the of the what is run inside the brackets is does the job then a rule in the form " if there is an an unclosed ( at the end of a line and it is the last unnclosed thing" (quotes, {} and @{} can all span line ends), "then treat the line break as white space"
Would do it .

It's mildly nuts that in 5 (outside the ISE) and 7
(get-childitem [enter]
puts up the
>>
Prompt for "unfinished line" but if you put anything on the second line it causes an error. You can't have a second command or a continuation of the first.

Pulling the break out of

(dir | 
where ...)

or

(2+
2)

or

(1,2,
3)

would be harmless. I think.

[mklement0]

can't that be worked around by wrapping the whole thing in ()?

I don't think (...) is the way to go:

• In commands, () have behavioral implications beyond just precedence grouping (as discussed before - evaluation in full up front, forced enumeration), I think it would be confusing to make them serve as a syntax modifier as well, not least because it then requires the awkwardness of wrapping the entire pipeline to avoid behavioral changes. (), $() and @() are operators, and that's all they should be.

• More generally, the issue at hand relates to the syntax for a command's arguments, not the command as a whole (or the whole pipeline), and it makes sense to focus the syntax on that, as the RFC does.

¬( ) or €{ } or ~[ ] or something of that kind ...

The latter point applies to this proposal too; on a side note, with both proposals you could end up with unwieldy constructs such as @(( ...)) and @(¬( ...)).

Which from my reading and @JamesWTruher 's comment above isn't a good idea.

Noted, but no decision has been made.

The "rolling continuation" says, in effect "temporarily change the layout convention to indicate end of command. If I indent, or begin the next line with -paramName or <> continue the line".

If I read the main proposal correctly, the proposed significant whitespace is (rightfully) only the blank line after the command signaling the end of the command, not the whitespace on the lines that are part of the command.

A marker at the end removes the need to do anything to keep the continuation going.

We already have such a marker, which you may use as an alternative to the blank line: ;, the statement terminator / separator.

Also note that no marker may be needed at all, depending on the context; to quote from the RFC text:

command-terminating tokens include a pipe symbol, a redirection operator, a closing enclosure, a semi-colon, or a & background operator.

Although, @KirkMunro, come to think of it: I don't think redirections (>) can imply command termination, given that you can technically write commands such as Get-Date >tmp.txt -UFormat %s (however ill-advised that is).

[Pxtl]

@mklement0

But numerous behavioral operators are already enable multiline things. | and + let you put the next operator on the next line. @() lets you split a list declaration across multiple lines, ditto @{} for hashtables. It seems like a natural extension of the language to extend the same courtesy to the humble () expression.

awkwardness of wrapping the entire pipeline to avoid behavioral changes

I'm just not seeing how that's so awkward. After all, if we used :; as a multiline operator, would

   :$result = $_ | Get-ChildItem
       | Foreach-Object {
       ....
       } | Do-MoreStuff
       | Yadda-Yadda
   ;    

really be that much less awkward than

   $result = ($_ | Get-ChildItem
       | Foreach-Object {
       ....
       } | Do-MoreStuff
       | Yadda-Yadda
   )    

I'm just not seeing any specific failure-cases here. Where exactly does this not work? Is it just that we're creating an implied return of $null to an operation that previously returned void

[jhoneill]

If I read the main proposal correctly, the proposed significant whitespace is (rightfully) only the blank line after the command signaling the end of the command, not the whitespace on the lines that are part of the command.

A marker at the end removes the need to do anything to keep the continuation going.

We already have such a marker, which you may use as an alternative to the blank line: ;, the statement terminator / separator.

If you made the ; mandatory then whilst it wouldn't have the elegance of matching {} [] () <> (the last absolutely wouldn't work) but it would at least have a marker. Blank lines having syntactic meaning - any white space having syntactic meaning ... you either have to go the whole hog and have python and yaml, or you don't do it all - like everything else.

numerous behavioral operators are already enable multiline things. | and + let you put the next operator on the next line. @() lets you split a list declaration across multiple lines, ditto @{} for hashtables.

A better way to think of this is (outside of the ISE) ALL operators allow their trailing operand to appear on the next line

$x =
>>42

{0} hoo" -f
>> "boo"

etc , is a list making operator so lines can split at a ,

>"{0} {1}" -f "hoo",
>> "boo"

(hash tables are backwards because they need the line breaking ; if two members appear on the same line - in effect each assignment in a hash table is a command)
These are all fairly easy because the line parser knows "this line still has something unfinished."
Telling the parser "this CR you're about to see ... ignore it" is easy.
"Ignore CRs and join the lines until something tells you to stop" is harder but not impossible - that's what the ; does in some languages.

[joeyaiello]

We on the @PowerShell/powershell-committee discussed this offline with @KirkMunro and agreed that taking a dependency on whitespace conventions and look-ahead is not something we want to pursue. For this scenario, users should leverage backticks as the line continuation character.

We recognize that the backtick can be difficult for some users to see, and we want to pursue some investigation around how to make it more visible e.g. in the VS Code extension or in PSReadline's syntax highlighting.