End support for PowerShell v3/v4 in VSCode-PowerShell v2.0.0

[rjmholt]

Currently, supporting PowerShell v3 and v4 creates several challenges for us:

• To get PSReadLine into the extension, we need changes coming in PSReadLine 2.0, which does not support v3/v4.

• We are forced to use the v3 AST to support v3, which means:

  • No support for classes or enums
  • Unable to use modern AST APIs

• Incompatibility in the APIs between v3, v4 and v5 mean that there are bugs that are hard to fix (and even hard to reproduce, since most of us don't have access to a machine running PowerShell v3 or v4).

• Of the two CI systems we currently use and the others we are looking to invest in, none supports PowerShell v3/v4 natively.

• v3/v4 usage is relatively low, and decreasing. With Windows Server 2019 around the corner and Server 2012 mainstream EOL also coming up this year, even more users will be moving to a newer PowerShell version.

• The code base is fragmented and hard to maintain because:

  • We have to compile against .NET Framework 4.5.1 and .NET Standard 1.6, when we would ideally compile against only .NET Standard. (This also spurs us toward better cross platform support)
  • We also have compile time PowerShell version targets.
  • Both of which lead to crazy .csproj files like this one.
  • The .NET Framework/Standard stuff means the backend codebase is riddled with #ifdefs.
  • There are also #ifdefs for v3/v4 support.
  • There's also version conditional logic in the build and run tooling, but that's easier to work with.

This issue contains most of the discussion that's already been had about dropping v3/v4.

My suggestion is:

• Include a deprecation warning for ~6 months, starting when v2.0.0 of the extension is released as stable.
• Actually drop support for v3 and v4 6 months after that date.
• Increment the major version number when we do it.
• Ensure that PSScriptAnalyzer can analyze PowerShell feature support by version so that people can still write scripts for v3 and v4 using the VSCode extension (but with the extension host running v5+).

Some additional things:

• An alternative to some of this is to refactor much of our code to abstract away the differences. However, that could be a costly effort and lead to significant regressions.
• We've also been talking about incrementing the major version of the extension with the addition of PSReadLine support. It seems a little cruel to me to drop v3/v4 support just as we add PSReadLine, but it also makes sense that both changes are big enough to mean a major version increase.

UPDATE

The addition of PSReadLine depends on a new version of PSReadLine that requires PowerShell v5.1 or later. This is by far our biggest motivating factor for ending support for PowerShell v3/v4.

[bergmeister]

PSSA supports v3/4 using pragmas and conditional compilation. This way you can use the latest API in in the versions of PowerShell where they make sense, i.e. a rule analysing classes does not make sense on ps v4.
If this is too much pain, then you should consider having a v2 of the extension in parallel (with the pain of merging important commits back to v1) but dropping support is not ideal. I can understand that it would be nice for us developers but keep in mind that it is also the job of a developer to put up with that complexity to make life as easy as possible for the user.
This issue is about VSCode but please also keep in mind that other parties might be using PowerShellEditorServices as well (I plan make either a WPF or Monaco based editor integration experience)

[SeeminglyScience]

@bergmeister that's what we currently do, but @rjmholt did a great job at outlining the issues with that. One of the biggest factors for me is the fact that we will only be able to support the latest version of PSReadLine, which requires PowerShell 5.1. Adding to that is the sizable amount of refactoring required to get PSRL working (often requiring newer PowerShell API's). Meaning we'd be supporting two wildly different experiences and code bases.

Also I suspect that the vast majority of folks are either stuck on version 2, or they're running 5.1 with very little in between. I'd be inclined to agree with you if we had the ability to support version 2, but we haven't since the beginning of the project (afaik). Granted this is anecdotal and based mostly on my own experiences and discussions with others, but I would be incredibly surprised if versions 3/4 made up more than 5% of PSES usage between them.

[bergmeister]

We are forced to use the v3 AST to support v3

What else do you expect? A user gets as many features as appropriate to the used version. By dropping support for v3 completely means that you change the feature coverage to 0%

Incompatibility in the APIs between v3, v4 and v5 mean that there are bugs that are hard to fix (and even hard to reproduce, since most of us don't have access to a machine running PowerShell v3 or v4).

Ok but it does not mean you have to fix all bugs in v3/4, you can still suggest users to upgrade should there be a blocking issue. Getting PowerShell v3/4 for testing purposes is as easy as creating a WinServer 2012/2012R2 VM, which takes less than 1 min to setup in Azure.

v3/v4 usage is relatively low, and decreasing. With Windows Server 2019 around the corner and Server 2012 mainstream EOL around the corner, even more users will be moving to a newer PowerShell version.

Customers use OSs that are supported by MS and they still run v 2/3/4 out of the box and with updates. Most of my customers don't even know there is a newer version of PowerShell. Users then have to take what they get in a corporate environment as they do not have admin rights to install stuff on their machine.

The code base is fragmented and hard to maintain

As I said that is sometimes part of our job but of course there is a limit at some point...

Your last argument about PSReadline is more interesting because it would be a technical and unavoidable reason. I think we need to split the discussion/deprecation into 2 parts:
For the VSCode extension it might be OK to drop support because there is still the ISE but PSES is supposed to be a general purpose library for usage by any 3rd party (including non-Microsoft!). If I embedded an editor into my app (for easier inline editing) that used PSES then I need this to work on any ps version that is installed on the client machine without having to force them to upgrade since scripts can break with the upgrade!

[rjmholt]

I've opened this issue just so that we can have an open discussion about pros and cons where anyone can weigh in. This issue properly belongs in the Editor Services repo, but this repo has more visibility so seemed like the right place to start the discussion. I'll open an issue in PSES to link back to this one.

@bergmeister you make excellent points, and we should also consider the possibilities of forking PSES and refactoring it to handle old versions properly.

We do have a responsibility as developers, but our time is finite and we should plan accordingly -- time spent maintaining a more complex, ultra-backward compatible codebase will mean less features, bugfixes and performance improvements. As PowerShell develops and changes, we will have to support that too, and we're going to have to decide what gets less attention or is dropped (which could just translate to "we support everything, but not as well").

As an open source project, users are also contributors, and to my mind the responsibility is shared.

[SeeminglyScience]

Ok but it does not mean you have to fix all bugs in v3/4, you can still suggest users to upgrade should there be a blocking issue.

If that's acceptable, why not use an old version of the extension if you need support for v3/4?

Getting PowerShell v3/4 is as easy as creating a WinServer 2012/2012R2 VM, which takes less than 1 min to setup in Azure.

That's a good solution for @rjmholt and @tylerl0706, but not for anyone else contributing to the project.

Customers use OSs that are supported by MS and they still run v 2/3/4 out of the box and with updates.

I have no doubt that tons of people are still running v2. But we already don't and can't support that one. What percentage are running 3/4 though?

Your last argument about PSReadline is more interesting because it would be a technical and unavoidable reason. ... For the VSCode extension it might be OK to drop support

Keep in mind PSRL is not a VSCode feature. The changes required are entirely within PSES and PSRL, any PSES enabled editor with a terminal will get the feature.

[TylerLeonhardt]

Lots of great discussion here. I'm on the side of revving the major version and back-porting important bug fixes back to the v1.

Sorry for being terse - traveling :)

[vexx32]

While some % of people are still on v3/v4 of PowerShell, i think that in large proportion these are not on machines where anyone will be doing any regular developing.

The ISE exists for one-off or occasional development work on those machines... But I think any serious PowerShell developer or sysadmin will be writing their code on 5.1, and many are starting to at least tinker with PS Core.

In other words, retaining overly complicated and time consuming support for these older platforms is unlikely to really benefit anyone we can maintain support for developing for these systems through version awareness (no classes when developing for PS v4, etc.), without needing to actually support development using these systems with the latest versions of these tools.

If you want similar functionality, there's nothing wrong with allowing people to still download and use the older versions that still support PowerShell versions 3 and 4, but I think it could be pretty hindering to both developers and users of these tools alike to unilaterally support as many older versions as we can.

It's a common problem: sure, we can spend exorbitant amounts of time and effort supporting older versions for as long as humanly possible. But I don't think we should.

[billkindle]

Still need time IMHO. Or at least give an option to switch to an older mode temporarily. Personally I have hoarded of systems that are on 2008 for another year or two. Yeah they are going to be replaced and yeah maybe still have ISE. I’m torn. I guess as long older versions would remain available for the fringe cases....

[Matalus]

I can understand the push to remove obsolete versions, but I'll make this argument, PowerShell tends to get used as a DevOps language, and at least in my case and I would wager many others. you don't always get to choose which version of PS the customers you're working with are using. While most of what I do is in v5, I know I have customers that are still running v2 and they have zero plans to update ever.

[rkeithhill]

As has been mentioned, the extension already does not support v2.

[vexx32]

Just because the tool is developed for PS 5.1 doesn't mean it can't support testing for compatibility for earlier versions of PS.

You just won't have access to the latest version for creation of scripts on these machines themselves. You can still verify that the scripts will work on the older machines, though.

You may not be able to choose what version of PS your customers are using, but there's no reason you have to develop in the exact same environment as your customers.

[dimitertodorov]

VSCode is mainly used on our development machines where it would be silly to run anything below 5.1

I develop a lot of Binary modules, and having C# interop work better would be great.
In particular, this issue would be great to close off.
PowerShell/PowerShellEditorServices#14

For testing, we always use remote machines for QA, and that is not dependent on having VSCode installed.

[DougPlumley]

I'm all for dropping it, our whole organization upgraded all servers to v5.

[AspenForester]

This system admin will not be above (mis)using this as an argument to management to get servers in the field upgraded. But in greater seriousness, I completely agree that development workstations should and tend to be running the current PowerShell.

Thank you for opening up this discussion!

[lehtoj]

All of my personal Windows machines at work have PowerShell 5.1. It is unfortunate fact that Microsoft has yet to update all their enterprise products to support PowerShell versions beyond 4.0, so that's what we still have to target when writing scripts.

Better support for 5.1, 6 and beyond would still be more than welcome. As long as dropping support for v3/v4 does not hinder our ability to target said versions too much, I say go for it.

[illegalOperation]

My customers that are still using legacy PowerShell are not using VScode. I know there must be some edge cases where some servers can't be upgraded to current WMF (I'm looking at you old exchange) but I feel that if an IT shop is using VScode they are already on the ball on being current. My customers that still rock older WMF are happy as a clam to use the ISE.

[Halkcyon]

@lehtoj Doesn't Win10 / Server2016 ship with v5 as a default? I'm not sure of any enterprise that has plans to implement v6 since it provides no benefit to them.

[lehtoj]

@TheIncorrigible1 Well yes, I have yet to see a single Windows Server 2016 in the wild though. Support for v6 and beyond would be mostly for personal tinkering until a viable business case for it emerges.

[dragonwolf83]

This would be a different discussion had PS Editor Services been the framework for ISE. The ISE was and is the editor for PowerShell v2-5.1. This is a new framework and cannot be expected to just slide in perfect support to older versions of PowerShell that was not designed for the new functionality the framework supports.

Let's pose this question as if this was ISE. Has any version of ISE supported running older versions of PowerShell? AFAIK, you can only get the new ISE when you install the latest WMF. So had the ISE come out with some new uber feature, you wouldn't get it anyway unless you were on the latest.

Going forward, PowerShell is a side-by-side model. We have Windows PowerShell 5.1 and multiple versions of PowerShell Core v6.0 installed on the same box. PSES needs to maintain backwards compatibility support to 5.1 for the long-term. I think new features can still light up in newer versions, but not drop support for several years.

[TylerLeonhardt]

Just to do a bit of moderation:

@dimitertodorov: Issue PowerShell/PowerShellEditorServices#14 can be addressed quicker and with less pain if we drop v3/v4 🙂

@TheIncorrigible1, @lehtoj value prop for PS 6 is not really a discussion point here but xplat is a benefit for those that manage a mix of linux/windows. Also it's a lot faster. If there's something you're missing from PS Core, please open an issue 😄

@dragonwolf83 the plan is that PSES will continue to support 5.1 for the long-term.

[Halkcyon]

@dragonwolf83 Stop beating a dead horse. ISE is deprecated and a different beast.

[TylerLeonhardt]

Also to be clear, for those that still use v3/v4 you would still have 2 options:

• any v1.x PowerShell Editor Services editor like VSCode's extension (will still receive certain bug fixes)
• ISE which will be in Windows already (maintenance mode)

Also as folks have mentioned, you can still write a script that works on v3 while using a PSES editor (vscode) and 5.1 or 6.

In fact, PSScriptAnalyzer will have rules, I believe, that say "hey you're not compatible with this version" to help that scenario.

[dragonwolf83]

@TheIncorrigible1 by your definition, so is PowerShell v3-4. Besides, you missed the point. I'm not advocating ISE forever. I'm saying that Microsoft has no obligation to backport a new way to do an Editor to older versions. It is a nice to have but if it hinders development, I vote for the path @tylerl0706 mentioned above.

[bergmeister]

Also as folks have mentioned, you can still write a script that works on v3 while using a PSES editor (vscode) and 5.1 or 6.
In fact, PSScriptAnalyzer will have rules, I believe, that say "hey you're not compatible with this version" to help that scenario.

The UseCompatibleCmdlets rule already does this and the next release will contain more version files. Future versions will also analyse the types compatibility and miscellaneous other things.

To summarise: I am rather worried about the PSES dropping support rather than the VSCode extension.
However, having said that, this would mean that it would be at least nice if both versions get installed when installing the extension and at runtime the extension decides which one to use, to avoid having to manually dig out the old one. Keep in mind that people deliberately keep test boxes not updated to test that their PowerShell modules also work on older versions of PowerShell and therefore will need to visit the box from time to time for debugging purposes.
PSSA has a WMF4 image in the build matrix here to run tests against v4, which means that other people do not need any VM setup for a PR.

@SeeminglyScience @tylerl0706 @rjmholt I have one more constructive idea to solve the PSES problem with the API problem Why don't you just use the latest Parser of PSCore itself (in general or for old versions like 3 or 4) since PSES seems to be just a .Net solution? I did a proof of concept a few months ago, where I was parsing a script to an AST on a Linux box that did not even have PowerShell installed at all. This is possible because one of the NuGet packages of the powershell-core feed (I think it was either PowerShell.SDK or System.Automation.Management) actually contains the whole Parser code and not just the reference assemblies.

[SeeminglyScience]

@bergmeister

I am rather worried about the PSES dropping support rather than the VSCode extension.

I get the impression that you are worried this change is being made without other PSES implementations taken into consideration. We may disagree on this change, but I'd like to reassure you that when it comes to PSES changes, my primary concern is how it could effect future implementations. VSCode is a great example of what an implementation can do but the power of PSES comes from it's extensibility first and foremost. It is with that in mind that I've been fighting for this change for some time.

However, having said that, this would mean that it would be at least nice if both versions get installed when installing the extension and at runtime the extension decides which one to use, to avoid having to manually dig out the old one.

I'm not sure there is enough usage to justify implementing this. Personally I think the same about forking/porting fixes but I think that's a reasonable enough compromise for those worried by the change.

Keep in mind that people deliberately keep test boxes not updated to test that their PowerShell modules also work on older versions of PowerShell and therefore will need to visit the box from time to time for debugging purposes.

I think the big disconnect here, for me, is why VSCode (or any other editor) needs to be installed on these machines? Wouldn't it be preferable to debug from a PSSession? Or with something preexisting (ISE)?

PSSA has a WMF4 image in the build matrix here to run tests against v4, which means that other people do not need any VM setup for a PR.

That's great for final checks before merge, but doesn't help as much with development.

Why don't you just use the latest Parser of PSCore itself
...
This is possible because one of the NuGet packages of the powershell-core feed (I think it was either PowerShell.SDK or System.Automation.Management)

The SDK does contain all the code required for that, but in order to reference it you need to target netcoreapp2.0. Right now we build two sets of assemblies, one for CoreCLR and one for FullCLR. We can't reference the SDK in the FullCLR build which is where we would want to use a newer API, and we already reference it in the CoreCLR build. Adding to that, PSES is hosted in a PowerShell session so even if we had a later version of SMA we could reference in FullCLR we can't override the one already loaded in the AppDomain.

I think there are significant enough reasons to look into ripping out the code for SMAL.Parser and adding some custom logic at some point. But that doesn't solve this problem, a lot more than the parser has changed since 3.0. Off the top of my head the debugger API has changed significantly from 3.0 to 5.1.

[bergmeister]

@SeeminglyScience

The SDK does contain all the code required for that, but in order to reference it you need to target netcoreapp2.0

This was true for version 6.0.1 but with 6.0.2 they did it properly using netstandard2.0, shouldn't this allow you then to use it in Windows PowerShell where it gets compiled for the fullclr? AppDomains should normally allow you to use a different version even when being hosted in Windows PowerShell.

We will probably never agree on PSES support but at least having some integrated legacy mode to keep basic functionality would be nice. I will ask management for details/clarification but I can imagine that if they get the choice between taking something that works on any version or taking something that only works on the latest version but receives great improvements, they are likely going to prefer the former option then. Our clients are unfortunately often in a big corporate environment that is not very flexible.
Personally I am all for using the latest and greatest stuff from a client and development point of view whereever possible but this is a tricky subject since it is affecting clients and not just us.

[SeeminglyScience]

This was true for version 6.0.1 but with 6.0.2 they did it properly using netstandard2.0, shouldn't this allow you then to use it in Windows PowerShell where it gets compiled for the fullclr?

Maybe I'm getting it from the wrong place. Is there a way to reference the SDK other than this myget feed?

[essentialexch]

I can't keep up with all the issues!

It sounds like you've already made decisions, but I'll throw my 2 cents in here:

As long as I have to support old versions of Exchange, I need to be able to develop against old versions of PowerShell. Right now there are four versions of Exchange in play -- Exchange 2010, Exchange 2013, Exchange 2016, and Exchange 2019 (unreleased). So I develop against PS 3/4/5/5.1 and generally do it on those lab servers. Removing VSCode/PSES support for PS 3/4 will cause significant pain for me.

(And for those of you that are not aware, Exchange does not support you upgrading WMF versions. You use the version shipped with the OS where Exchange is installed.)

[vexx32]

You can still use the old version(s) of the extension without the new features... this is more to break away from that so we can have new features on the versions of PS that support them.

[D3MaxT]

I also vote for only supporting 5.1 and Core with 2.x.

For older versions, use 1.x or use ISE or PowerShell Studio.

[DarkLite1]

Agree, it would be best to have the old extension support all PowerShell version below 5.1 and the new extension then supports from 5.1 upwards. If this would be implemented, it might be cool if a small pop-up would tell the user, if he is on 5.1 or higher, to download the newer extension.

[WSLUser]

I'd like to add my feedback here as well. I'm in a environment where I use a VM to do all my dev work and the only approved Windows VM is 7...So upgrading PS is impossible for me. If PS Core had better support for 7 then I could be in support of dropping it and using PS Core instead but PS Core has no plans to support Win 7 due to it going EOL soon (which my org doesn't care about, they'll still use it anyways since it's just a VM and not the main box anyways.) I'm sure I'm not the only one in a similar situation so support is still needed but separating out the extension by whether you're on 10 or Core vice older is a good idea as long as support for both is still on going for at least the next 5 years. By then, my org may actually decide we need to upgrade to 10 (hopefully).

[vexx32]

You can upgrade Windows 7 machines to PowerShell 5.1. There's no need to suffer through on v2; it's really not worth it. It's much more productive to use 5.1, there are a ton of convenience features that make it a lot simpler to work with.

[essentialexch]

It may be possible to update Win7 to PowerShell 5.1, but it may not be allowed.

[Halkcyon]

@vexx32 cough -in, -notin, @().Where(), @().ForEach(), just to name a few that make my life easier and code more readable. Plus all the bugs that have been remediated in that time so you're not working around weird issues like if blocks falling through because you did a right-hand $null compare.

@swngdnz I can see zero reason it would not be allowed. That is an issue with management more than anything technical.

[vexx32]

If it's not allowed, it's time to read up. PS v2 is such a security risk that it's a commonly-tested exploit vector for machines that have 5.1 installed and haven't fully removed v2 yet. Lots of malicious PS scripts explicitly try to use PSv2 because it's harder to protect against attacks from there, not to mention actually properly manage. If memory serves, that was before the real logging solutions were implemented, so most of them can do whatever they want without being particularly traceable.

It's not a good policy, not a safe policy by any measure of the imagination, to retain PSv2 on any company devices. I'm fighting a similar fight at my location, though in my case I'm just glad these old machines are dying off quickly enough and our replacements are all windows 10. 😄

[WSLUser]

I was able to upgrade but my $PSModulePath is fried. I'd hate to revert, save it somewhere, and then upgrade again. Any way to get it back?

[Halkcyon]

@DarthSpock PSModulePath is an environment variable. You can restore this from the "Advanced system settings" menu or using the .NET [Environment] class:

[Environment]::SetEnvironmentVariable(
    'PSModulePath',
    ("$Env:USERPROFILE\Documents\WindowsPowerShell\Modules;" +
     "$Env:ProgramFiles\WindowsPowerShell\Modules;" +
     "$Env:SystemRoot\System32\WindowsPowerShell\v1.0\Modules;"),
    [EnvironmentVariableTarget]::User
)

[rjmholt]

Hi all, just want to let you know that I've been doing some work to help with the compatibility story here:

PowerShell/PSScriptAnalyzer#1133

The next release of PSScriptAnalyzer will (with the right configuration) be able to determine that a command, .NET type or syntax in your script is not available by default on another platform (like PowerShell 3/4). So writing a script in PowerShell 5.1 or 6, you'll get advance warning that [System.Collections.ArrayList]::new() or Get-FileHash $myFile or "Stuff" | Out-File ./file.txt -NoNewline won't work in PowerShell 3/4.

I'm still writing up some documentation on how to use it, but hopefully we'll have a blog running through an example soon enough.

[rjmholt]

FYI everyone, I've just published the blog post.

The compatibility rules come with the latest release of the PowerShell extension for VSCode, so all you need is to write a configuration file to run them, which is explained in the blog post.

[SydneyhSmith]

In the next release of the PowerShell extension we will be dropping support for v3/v4. For more information:
https://devblogs.microsoft.com/powershell/powershell-editor-services-roadmap/