@TBBle I only just realised that there's lower L in your nickname instead of capital i when I saw that pinging you didn't work :D

Just wanted to say, I'm aware of this, but haven't been through it yet (it's a long patch).

One thing that jumps out from the description: why can't we check if the container version is newer than the host OS? It should be an easier check now, since if we're working with Windows build versions, it's a straight numerical comparison, without any complications for "ltscXXXX" or "XXHY" strings.

At worst, if given a build tag we don't recognise and that doesn't match the local system build version or release ID, we can bypass the test with a warning, assuming the user knows what they're doing and we are just late in an update.

I'd also suggest we keep rejecting the known Windows suffixes, as it's a small list that only grows by one every six months or so, but perhaps only those that are actually still supported upstream. If useful, we could parse them out of https://mcr.microsoft.com/v2/windows/servercore/tags/list as anything in that list without _, - and that doesn't start with 10. is a release ID for which server core images exist, but since we will need the mapping between release id -> Windows build version at some point anyway, I don't think just hard-coding this list will be an issue, since that's what everyone else (even MS: a + b) ends up doing, since there's no 10.0.X.latest tags.

Side-note: I think it'd be better to not name the bug being solved in the name of the PR, since it doesn't create the right links (i.e. this PR is linked to #138, but not #164).

Edit: Looking at the description again, I realise that maybe this PR doesn't do what I expected it to do, so I'll have to actually read it before I can comment further. The description doesn't match what I had in my mind based on #138 (comment)

why can't we check if the container version is newer than the host OS?

We can't do that because we still refer to images using release id (1809, 20H2 and so on). If we started referring to images by their build number, you're right, it would be easy to compare them.

I'd also suggest we keep rejecting the known Windows suffixes, as it's a small list that only grows by one every six months or so, but perhaps only those that are actually still supported upstream. If useful, we could parse them out of https://mcr.microsoft.com/v2/windows/servercore/tags/list

Yep, we can do that.

The description doesn't match what I had in my mind based on #138 (comment)

True, I did a bit different thing here. I am not doing a buildnumber -> release id lookup anywhere. Instead I ask the host system for its release id and use that to lookup image (the same way as asciidoc

I've split part of this PR into #170 so current PR becomes smaller and easier to review.

Okay, I've added reading tags list from https://mcr.microsoft.com/v2/windows/servercore/tags/list. But we still can't check whether target tag is newer than host system because https://mcr.microsoft.com/v2/windows/servercore/tags/list is not sorted chronologically.

But we still can't check whether target tag is newer than host system because https://mcr.microsoft.com/v2/windows/servercore/tags/list is not sorted chronologically.

I'm not sure why you'd want them to be pre-sorted? We're only comparing host BuildID against the X in 10.0.X.Y, since (apart from one or two historical issues) Y is irrelevant, and only X needs to match for process isolation.

We know that the host BuildID either matches an X from the list (14300, 14393, 16299, 17134, 17763, 18362, 18363, 19041, 19042), aka (Windows Server 2016 Tech Preview 5, ltsc2016/1607, 1709, 1803, ltsc2019/1809, 1903, 1909, 2004, 20H2); or it's between those numbers and hence an out-of-date Insider build, or it's greater than the largest, in which case it's a current Insider build (or if it's 19043, then it's 21H1 and we're still waiting for an image to go with it, so we can pretend that's "Insider" for our purposes).

I suspect the only reason 10.0.14300.1030 is still tagged is because it was the first wide-spread Windows Containers base container, and it's very easy to find old instructions and "How to" guides for "Windows Containers" that have that tag explicitly present. And (in Hyper-V isolation) it still works, so why not?

Later, we can look at automatically attempting to find such "Insider" images on https://mcr.microsoft.com/v2/windows/servercore/insider/tags/list. Annoyingly, that won't work right now for Windows Server 2022 Preview on the MS Evaluation centre, as that's 10.0.20348.1, and there's no Insider container image base for that yet. It seems to take a week or so for Server Insider container images to appear after a Server Insider release, and that's out-of-scope for this work anyway.

I'm not sure why you'd want them to be pre-sorted?

We need to solve two tasks:

1. Automatically pick an image that matches current host. Before this PR, ue4-docker always used ReleaseId massaged through WindowsUtils.getReleaseBaseTag function for that. After this PR ue4-docker uses DisplayName whenever it exists (20H2 and newer) and otherwise fallbacks to ReleaseId. This could be alternatively solved in a less-human-friendly way by using "10.0.<buildNumber>.<UBR>". This way we would be 100% guaranteed to match host version, but produced tags would be awfully long. Just imagine it: docker run adamrehn/ue4-minimal:4.26.1-10.0.19042.985-vs2019 🤢
2. Decide whether it is safe to work with -basetag provided by user. Before this PR, ue4-docker had hardcoded (ordered!) list of valid tags. After this PR, we allow any tag that is present on https://mcr.microsoft.com/v2/windows/servercore/tags/list. But now we have no idea if specified tag has the same kernel as our host OS until we actually download it and try to run. And I don't think that writing ad-hoc regex to parse all formats that are already published is constructive.

On 1, we can do that with the mapping of BuildId => ShortContainerTag, and just totally ignore DisplayName, ReleaseId, or anything else from the registry, in favour of sys.windowsversion.split('.')[2], which is host's actual numeric BuildId. This is exactly what Microsoft does, as I pointed out earlier, in their own code to solve the same problem.

On 2, the tag format is pretty fixed, or at least it has been for the last half-decade or so. It's either 10.0.BuildId.Patch, 10.0.ReleaseId.Patch[-_]SomeMetadata, ShortContainerTag, or ShortContainerTag[-_]SomeMetadata.

We don't care about the SomeMetadata versions, so we can trim any provided tag at the first - or _, and then if it's 10.0.ReleaseId.Patch, compare BuildId to the host BuildId, and if it's anything else, look it up in the fixed, stable but growing slowly mapping of ShortContainerTag => BuildId

Sorting/ordering never matters because we can't take an inequality match on BuildId. We either match something, or it has to be Hyper-V isolation because it doesn't match the host's BuildId, at least until such time as MS says that Feature Updates can be ignored, irrespective of "newer" or "older".

And if we want to know something is newer than the newest known version, we can always go from ReleaseName to BuildId unless the tag is too new for our system, so we either have a numerically-ordered BuildId, or "the tag is too new for our system" in which case we treat it as an Insider build, and users will have to either force the isolation mode, or use the 10.0.BuildId.Patch basetag until a real image is pushed.

If we're really concerned that MS are going to change their tagging format to celebrate the second half-decade, we can grab the manifest for the named image, and check what OS version it claims. It's about a 30-line JSON file, in a fixed format, so that's not exactly a huge challenge. But that still seems like way more effort to implement and maintain than updating a table mapping tags and build IDs every six months or so (plus once every three years for an LTSC release).

This latter approach might be useful for supporting both Insider and custom base images, though, so long-term perhaps it's worth implementing, or finding a library that implements it already.

Okay, I've tried going that route, but I don't believe in it. Too much hassle with all these regexes, reverse lookups and very small benefits. We're effectively trying to reproduce Hyper-V logic that determines what images are compatible with what host OSes instead of allowing Hyper-V to do its job and fail with a loud error message if user tries to do something inappropriate. I already have 4x amount of changes compared to current PR and it's still not finished. I'm going to close this PR.

You can take a look at what I did so far: slonopotamus@ea77722

Sorting/ordering never matters

Ordering matters because we have three scenarios:

1. Host matches target. Use process isolation.
2. Host is newer than target. Use Hyper-V isolation
3. Host is older than target. Error, no fun for you.

Now, the question is: how to compare host version with target image tag.

Currently, ue4-docker just has a hardcoded ordered list of well-known versions, uses it to determine order and completely rejects unknown tags.

That doesn't scale if we want to allow insider tags, so we need to drop such logic. Now, what we have is build number of the host (good thing) and unordered list of tags from Docker registry (bad thing). We don't know which build number is inside those tags. We could try to attack this problem with regexes + well-known list, but usually regexes only increase number of problems we have.

Now, let's step back and remember why we are doing all this stuff. We're doing it only to print error message that user tries to use Hyper-V in an incompatible way quicker than Hyper-V itself. Hyper-V would do that build number check anyway when ue4-docker tried to run the image. Is it worth the effort? I don't think so.