Update the release cycle

[thufschmitt]

The 6-weeks release schedule that we've been using for several years doesn't really work-out that well (the best proof being that the Nixpkgs maintainers decided to sidestep it completely and freeze the stable Nix version in the stable NixOS releases).

Rework it to match more closely the reality and round a few sharp edges:

• Align the “frequent” release cycle with the calendar
  • The 6-weeks release cycle is hard to keep track of. A monthly
    release will make it much easier to remember the release date.
• Officialise the support for a stable version maintained for as long as NixOS stable
  • This is already the case in practice, it just happens that the
    “stable” Nixpkgs version is whichever version was deemed
    stable-enough at the time of the NixOS release.
    Officialise that by cutting a new major release alongside each NixOS one.

Note that this breaks whatever semver compatibility Nix might pretend to
have, but I don't think it makes sense any way.

cc @NixOS/nix-team @NixOS/nix-issue-triage

Also cc @flokli and @RaitoBezarius as the maintainers of the Nix package in Nixpkgs

[fricklerhandwerk]

Generally in favor of adjusting the release process to get higher quality releases that are less work for everyone, but still regular.

Not convinced that changing the version pattern will be helpful. I find the proposal rather confusing. We already note when releases were published, so all the information is there. And we need those two axes: time and compatibility.

I also don't think it matters all that much whether it's 4, 6, or 8 weeks. All that matters is that old stuff doesn't break, broken things get fixed, and new stuff works as intended. The only other constraint is having someone to do the work, and it gets easier if the work is less.

So maybe rather than a different release cycle we need better quality assurance? Which would mean diverting existing capacities and building up new ones.

[cole-h]

Does this need to go through the RFC process considering the 6 week schedule was decided through one?

[roberth]

"frequent" [...]

• The 6-month release cycle is hard to keep track of.

@thufschmitt I think you meant 6-week here.

RFC

For increasing the frequency of frequent releases, I find this unnecessary, because it's described as a detail, and more frequent releases are in the spirit of the existing RFC, which argues that more frequent releases are better.
Also teams should be able to innovate in their process, or we'd risk stagnation.

For changing the meaning of the version's major number, this seems rather significant.

Officialise that by cutting a new major release alongside each NixOS one.

I wouldn't do it, but if I'm the outlier, I'd suggest doing year-month versioning instead, and adopting a Nixpkgs-like release workflow for producing an extra stable Nix 24.05 for NixOS 24.05. Our level of QA should permit that, but currently we might want to skip Nix 24.04, and release 24.05 early so that we have a two month stabilization window.

[edolstra]

I really don't see how doing a release every month instead of every 6 weeks improves things for downstream. How does that address any of the QA issues? If quality is an issue, we should be making fewer releases, not more, and make sure they get tested properly. For instance, a release every 6 months or even every year.

[edolstra]

I don't understand what this means. What are XX and YY? Year and month? And what do the +1.0 and +1.1.0 denote?

[roberth]

If quality is an issue, we should be making fewer releases, not more

Are you doing manual QA before each release? If so, I would consider that to be the problem.
If you're referring to the QA done by users on nixos-unstable, then that's very honest, but killing the frequent releases will remove that source of feedback.

When changes are allowed to linger unreleased, this massively increases the feedback cycle and our ability to iterate on fixes, or even features.
Waiting a long time for a release is frustrating to contributors, reducing engagement.
We risk massive barrages of bugs, as in 2.4.
Dependent projects will lag behind, and we'll lag behind their requirements - especially those with a lot of contact area, such as Hydra, CIs and NixOps (TBD).

So no, we're not decreasing the frequency.

whether it's 4, 6, or 8 weeks

we need better quality assurance

I'd like to highlight this.

[RaitoBezarius]

I won't comment on the process until there's a conclusion because I don't have a lot of time, and I'm mostly on the Nixpkgs side, I believe I said in multiple venues, multiple times what I had to say on what I see in Nix releases in Nixpkgs, as a release manager, as an active NixOS contributor and as someone running production with large stores, etc. (a 'power user' or whatever).

I will just bring one data point that is quite important to me, I completely gave up on asking the Nix maintenance team for backports to stable Nix because my perception is that they're not taken seriously for Nixpkgs needs.

Furthermore, I admit that when I have to do things like https://discourse.nixos.org/t/nixpkgs-cli-working-group-member-search/30517/17?u=raitobezarius and then that @Ericson2314 tells me that he will take care of it and that it's not taken care of, after this public mention, this really does not help.

All of this to say that I hope the dimension 'doing backport of important fixes in stable channels' will be carefully (re?)considered and improved in whatever new process which will came out of this.

Thank you.

[edolstra]

Are you doing manual QA before each release?

I don't. However, a shorter release cycle means there is even less chance for regressions on master to be noticed, so it would increase the pain for downstream packagers.

We risk massive barrages of bugs, as in 2.4.

Yes, it's really a choice between having a lot of pain infrequently or a bit of pain frequently. Better QA is always good, but wanting new features is incompatible with the goal of having no regressions. It would be easy to have no regressions if we declare a feature freeze and put all new features in a "nix-ng" or "nix-next" branch.

[9999years]

wanting new features is incompatible with the goal of having no regressions

Huh?? That's what tests are for. The uncaught regressions are a symptom of Nix's lack of tests.

[thufschmitt]

This was written a bit in a hurry just to trigger the discussion, happy to see it succeeded :)

The one important point for me is that:

1. Some (undefined proportion of) people want something that just works™ and stays out of the way
2. Some (undefined proportion of) people want all the latest shiny features

and we should satisfy both.

I'm hoping that the next Nix survey will give more insight on the “undefined” part here.

---

With that said:

The 6-weeks schedule was meant to try and satisfy both at the same time (with the idea that enforcing frequent releases would enforce a better quality and testing and cause more stability overall).
This partially happened, but not fully (both because QA is still lagging way behind and because the surface area of Nix made this possibly too optimistic).

The result is that we effectively have a 2-layers schedule, with both

• The official 6-weeks schedule followed by a bunch of people who care about it
• An unofficial 6-months schedule set by nixpkgs.nix.

I think this is a good thing (forget about the 1-month thingy, that would make life slightly easier imho, but it's very minor), provided that:

1. We're explicit about it
2. We commit to it fully and willingly (by putting way more emphasis on the stability of the release that will end-up in Nixpkgs; by increasing our stability efforts overal, etc..)

[thufschmitt]

I've edited the proposal based on the comments here, making it much closer to the previous text (just more explicit about support).

Cherry-picking from the discussion:

@fricklerhandwerk : Not convinced that changing the version pattern will be helpful. I find the proposal rather confusing.

Fair enough. I think that having a clear marker for stable(er) releases would help a lot, but if it's confusing let's drop it for now.

@roberth: I wouldn't do it, but if I'm the outlier, I'd suggest doing year-month versioning instead, and adopting a Nixpkgs-like release workflow for producing an extra stable Nix 24.05 for NixOS 24.05

That could make a lot of sense, indeed. And match people's likely expectations that (by default) a given Nix version matches a Nixpkgs version (could also increase the confusion because the don't have to match though).

@RaitoBezarius : I completely gave up on asking the Nix maintenance team for backports to stable Nix because my perception is that they're not taken seriously for Nixpkgs needs.

That is a very important point, and my biggest gripe with the current policy. It states that we “typically” backport up to the Nixpkgs version, which is very weak and non-enforcing. It also has this painful “backports never skip releases” rule which makes every backport 5x as annoying, and vastly increases the chances of missing one.
Setting clear and reasonable expectations for the maintainers is crucial here.

@edolstra : it's really a choice between having a lot of pain infrequently or a bit of pain frequently.

The primary focus should be on reducing that pain altogether (which is partially out-of-scope here).
That being said, that is true, with two big caveats:

1. This proposal is actually giving people that choice
2. By having frequent releases, we can get a much bigger pool of users to battle-test Nix in between two “LTS” releases so that migrating from one to another is much less painful (“the QA done by users on nixos-unstable” that @roberth mentions)

@edolstra : a shorter release cycle means there is even less chance for regressions on master to be noticed, so it would increase the pain for downstream packagers.

Let's be frank: right now, releases are mostly just tags we stick at a random commit once in a while. This is by design (because master is supposed to be “kept in a releasable state at all times”), but means that changing the release frequency doesn't change much.

[zimbatm]

You can have both by:

1. branching (instead of tagging) every X period, and declaring that this branch is the next nix-unstable.
2. nixpkgs contributors package the latest commit and feed back the hydra failures into the backlog for the release.
3. Nix developers can keep developing on the master branch and backport fixes into the nix-unstable branch.
4. When the backlog is done and both parties are happy, announce a release.

[Ericson2314]

Huh?? That's what tests are for. The uncaught regressions are a symptom of Nix's lack of tests.

Agreed. I just opened #10320 for what I think is the easiest first step to for us to improve our test coverage.

[edolstra]

Huh?? That's what tests are for. The uncaught regressions are a symptom of Nix's lack of tests.

We can't base our release policy on a hypothetical world where Nix has 100% test coverage. Until we have that, there will always be regressions.

[9999years]

Agreed -- that's why writing tests to improve coverage should be our top priority for avoiding regressions.

[Ericson2314]

I think there is just specific happenstance where lazy trees PRs happened to be some fairly extension refactors hitting some areas (language filesystem interactions) that happened to be worse tested than other areas. An unfortunate coincidence.

On one hand, we shouldn't "blame" lazy trees on a picking an area that was a priori less tested --- sometimes that is just where the work needs to be done! On the other hand, we shouldn't let our experience with that persuade us that testing is futile and won't help.

[thufschmitt]

@zimbatm that's quite interesting. That feels like having -rc versions (which is kinda what we have in practice where the .0 versions are more buggy than acceptable).

I'm not too keen on the “when everyone's happy” part because without deadline it risks to just be delayed forever. But there can be a middle-ground.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-56/43035/1

[Ma27]

fwiw NixOS/nixpkgs#305951 may help with enabling more people to catch regressions early.