Provide Mesa3d V23.1.7 libraries for Windows

[gzotti]

This requires also new environment variables for starting in "Mesa mode".

Description

This is intended to fix problems around outdated Mesa V11 libraries which come with Qt.
It provides OpenGL 4.5 CoreProfile and would allow to run advanced features (OpenGL3.3) on really old systems.

I cannot currently say whether this deploys correctly. And, when uploading, there was a warning from Github:

remote: warning: See https://gh.io/lfs for more information.
remote: warning: File util/mesa3d/x64/libgallium_wgl.dll is 57.89 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote:
remote: Create a pull request for 'update-mesa' on GitHub by visiting:
remote:      https://github.com/Stellarium/stellarium/pull/new/update-mesa

So it may not be the best idea to accept this in this format.

Fixes: #2803

Screenshots (if appropriate):

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• This change requires a documentation update

How Has This Been Tested?

Test Configuration:

• Operating system: Windows 11
• Graphics Card: Geforce 3xxx (irrelevant), Intel Core-i12xxx.

Checklist:

• My code follows the code style of this project.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation (header file)
• I have updated the respective chapter in the Stellarium User Guide
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

[github-actions]

Great PR! Please pay attention to the following items before merging:

Files matching src/**/*.cpp:

• Are possibly unused includes removed?

This is an automatically generated QA checklist based on modified files.

[10110111]

would allow to run advanced features (OpenGL3.3) on really old systems

Is it really worth the trouble? ShowMySky is slow even on some GPUs (which was the reason for introduction of reduced resolution mode), won't it be unusable on a software renderer?

[10110111]

File util/mesa3d/x64/libgallium_wgl.dll is 57.89 MB; this is larger than GitHub's recommended maximum

Does it maybe contain debug info that could be stripped? 58 MiB is an enormous size for a code-only binary.

[10110111]

Who is pal1000? The official license of Mesa states Brian Paul as the copyright holder.

[gzotti]

He who provides ready-made Windows builds.

[10110111]

Does the copyright really get transferred due to building the opensource package?.. I suppose the license on the builder's project page refers to the build scripts, rather than Mesa binaries.

[gzotti]

I just took the License file as found there.
Around 2015 I managed to build Mesa myself (involving a cross-compilation on Arch I think). Since 2018 I was unsuccessful, did not need it, and ultimately was happy to find this ready-made release. It's MIT in any case.

[gzotti]

It's not only about ShowMySky. I agree I would turn that off on weak systems. But in light of your fixes of Landscapes, and probably other changes, and #2803, it may be interesting.

[gzotti]

File util/mesa3d/x64/libgallium_wgl.dll is 57.89 MB; this is larger than GitHub's recommended maximum

Does it maybe contain debug info that could be stripped? 58 MiB is an enormous size for a code-only binary.

No idea. The old version included with Qt (Mesa 11 IIRC) has 20MB, and we don't have to host that.

Packaging experts could maybe just download the three libraries directly from pal1000's site on building/packaging. I hope my CMakeFile.txt changes work, but I agree hosting a file which also develops further may not be the best solution.

[10110111]

hosting a file which also develops further may not be the best solution

Well no, I don't think it's a good idea to blindly follow the further development. We do need to fix the version to the one we've tested.

[gzotti]

hosting a file which also develops further may not be the best solution

Well no, I don't think it's a good idea to blindly follow the further development. We do need to fix the version to the one we've tested.

Not blindly. Just stick to one version until we need more or some improvement was made in Mesa. The older builds are still there. Just that one must download the 7z file and extract 5 files.

Or is it possible/preferrable to host those files on our website? There is no real need to git them.

[10110111]

Or is it possible/preferrable to host those files on our website?

I think not. We don't host, say, Qt. Nor did we host CalcMySky before we switched to custom builds of it. So if we can just fetch the binaries for the particular version of Mesa, this is OK.

[alex-w]

I think adding a downloadable archive of DLL's (here: https://github.com/Stellarium/stellarium-data/releases, see https://github.com/Stellarium/stellarium-data/releases/tag/qt-5.6 for example) and writing instructions for developers in BUILDING.md will be better solution. Of course, it requires some changes in cmake and .appveyor.yml files.

[alex-w]

Please check binaries from here: https://ci.appveyor.com/project/alex-w/stellarium/builds/48022354

[alex-w]

Correct link: https://ci.appveyor.com/project/alex-w/stellarium/builds/48022746

[gzotti]

Tried that. Umm, it still somehow contains the old dll, Mesa11 with OpenGL 3.0.

[alex-w]

@gzotti please test binaries from https://ci.appveyor.com/project/alex-w/stellarium/builds/48037720

[gzotti]

Tried again. The Qt6 build still contains the old dll, Mesa11 with OpenGL 3.0. I exchanged manually with the new files for immediate upgrade to OpenGL4.5. The Qt5-64 build works! Qt5-32 build: maybe tomorrow, download limit exceeded...

The old library appears considerably faster though. Maybe as soon as higher features are available, the CPU must work harder. @10110111 ? I am not sure what is currently 3.3+-only.

Given that --mesa-mode is mostly targeted towards old PCs as last resort, we could leave it with the old version and describe a manual update. The additional env variables apparently are irrelevant for the old library.

I think some users have demands of running the program in VMs or other GPU-less environments with powerful CPUs. Here the newer Mesa may be useful. This could also mean again Qt5 with old (and upgrade possibility) and Qt6 with new version...

[10110111]

The old library appears considerably faster though.

Well, maybe the newer one is built with less optimization enabled?.. I don't remember any systematic slowdown in llvmpipe over the years.

I am not sure what is currently 3.3+-only.

Only dithering and ShowMySky currently require glInfo.isHighGraphicsMode (which implies GL3.3+).

running the program in VMs

Modern VMs (at least VirtualBox) can provide HW acceleration via the host GPU.

Generally, I think all this development of kludges is not really useful beyond Qt5 version. Qt6 already cuts most of incapable Windows machines away, so we could just remove the support for software rendering in Qt6 — or leave the one that Qt provides. High graphics mode features only target realism and visual quality, not the astronomy, so I don't think providing software rendering for them is so much important.

[alex-w]

Tried again. The Qt6 build still contains the old dll, Mesa11 with OpenGL 3.0. I exchanged manually with the new files for immediate upgrade to OpenGL4.5. The Qt5-64 build works! Qt5-32 build: maybe tomorrow, download limit exceeded...

I didn’t replace Mesa for Qt6 builds

[alex-w]

Link to binaries: https://ci.appveyor.com/project/alex-w/stellarium/builds/48041621

Is target milestone correct?

[gzotti]

The old library appears considerably faster though.

Well, maybe the newer one is built with less optimization enabled?.. I don't remember any systematic slowdown in llvmpipe over the years.

I would assume that - given that Mesa does not provide binaries - this semi-official source does all to release an efficient build. But who knows? (Do you want to build optimized binaries?)

I am not sure what is currently 3.3+-only.

Only dithering and ShowMySky currently require glInfo.isHighGraphicsMode (which implies GL3.3+).

running the program in VMs

Modern VMs (at least VirtualBox) can provide HW acceleration via the host GPU.

If the host has no GPU? I think of VMs on Xeon servers.

Generally, I think all this development of kludges is not really useful beyond Qt5 version. Qt6 already cuts most of incapable Windows machines away, so we could just remove the support for software rendering in Qt6 — or leave the one that Qt provides. High graphics mode features only target realism and visual quality, not the astronomy, so I don't think providing software rendering for them is so much important.

OK, I see two options to continue:

1. Upgrade as @alex-w just completed AppVeyor integration. Advantage: complete, and @10110111 can upgrade OpenGL usage at least in the Qt6 family. (Must keep OpenGL ES2 compatibility on Qt5!) Disadvantages: Package size increase, further slowdown, esp. for slow-to-begin systems.
2. No upgrade of actual libraries, but just the improved wrapper call with more env variables prepared for a user-made upgrade of new Mesa binary libraries, overwriting the Mesa11 library provided by Qt. The upgrade process will be described in User Guide 5.5 and is in users' responsibility.

[gzotti]

Link to binaries: https://ci.appveyor.com/project/alex-w/stellarium/builds/48041621

Thanks. Qt6 tested, works. Commit message was likely wrong, should have been "Use Mesa23".

Is target milestone correct?

That depends on our decision speed. I said 23.4 because of 23.3 feature freeze.
There are no string translations involved. I can write a SUG section on upgrading this weekend.

[alex-w]

Thanks. Qt6 tested, works.

Good!

Commit message was likely wrong, should have been "Use Mesa23".

Yes, you're right.

That depends on our decision speed. I said 23.4 because of 23.3 feature freeze. There are no string translations involved. I can write a SUG section on upgrading this weekend.

But by the fact this is not a new feature, so, 23.3 is OK for me.

[10110111]

If the host has no GPU? I think of VMs on Xeon servers.

How likely are they to run Windows—and run Stellarium in these obscure environments? I'd say it's their problem to solve, not ours. Servers already require some expertise to maintain, so their administrators must be capable enough of setting up OpenGL runtime.

[10110111]

No upgrade of actual libraries, but just the improved wrapper call with more env variables prepared for a user-made

I don't think one more wrapper is needed. If a user does need a newer Mesa (which I don't think they do), it's sufficient to provide the instructions to do everything that doesn't require recompilation. Again, it's a very rare use case that I'd not spend a lot of time on. Software rendering is slow enough to not really expect fancy graphics from it.

So my option is:
3. Leave things as they are, maybe providing instructions on how to upgrade and enable newer Mesa.

[gzotti]

I may have seen more requests for weird problems or setups than you have. Classroom setups, remote use, ... You may think those running Xeons know what they are doing. Some may have one because it sounds cool or they expected more productivity when they are sitting next to a 19" fan array. I cannot follow their decisions.

There must be an option --mesa-mode also with Qt6 versions. It can remain crippled with OpenGL3.0 as now but with documentation on upgrading, or we deliver a big extra binary that most people likely really don't need.

Installing the upgrade causes a slowdown, but adds features which users may balance based on what they need.
Qt5/mesa users may profit from dithering. Qt6/mesa users may also get ShowMySky, at least on low framerates.

@alex-w , @10110111 please vote 1 or 2 as described above.

EDIT: this was written before reading @10110111 option 3. For all practical purposes, option 2 does everything which option 3 does (the changed wrapper just sets 2 more envvars which are ignored by the default solution), with less instructions for upgraders.

[10110111]

You may think those running Xeons know what they are doing. Some may have one because it sounds cool or they expected more productivity

Why should we fix the problems that these people created because of their incompetence? Game producers don't try to make their products run on every imaginable machine. High graphics mode of Stellarium is more of this type. Anything related to astronomy that may be needed for classrooms or remote use is still available in the legacy mode.

Besides, if you try to make Stellarium look good but feel slow in a classroom, you make a bad impression of the software on the students, which are used to video games being smooth.

we deliver a big extra binary that most people likely really don't need

This already bad because of this PR, because the repo clone takes longer now. If we refrain from storing the binaries, we can squash (or remove) this branch to avoid this.

Qt5/mesa users may profit from dithering. Qt6/mesa users may also get ShowMySky, at least on low framerates.

Let those who want fancy graphics use decent (not even fancy!) hardware.

option 2 does everything which option 3 does (the changed wrapper just sets 2 more envvars which are ignored by the default solution), with less instructions for upgraders.

This spreads the upgrade over the project, while with option 3 we'd only have it in the instructions, without additional maintenance for the code. I expect this upgrade to be ignored by most of the users of Mesa mode, simply because the whole software rendering mode is a workaround to get astronomical information, not graphics.

[gzotti]

You may think those running Xeons know what they are doing. Some may have one because it sounds cool or they expected more productivity

Why should we fix the problems that these people created because of their incompetence? Game producers don't try to make their products run on every imaginable machine. High graphics mode of Stellarium is more of this type. Anything related to astronomy that may be needed for classrooms or remote use is still available in the legacy mode.

Stellarium is not a game. The game industry drives GPU development and sales of the latest hardware for ever more graphic explosions and damage. I aim at max compatibility starting with €50 hardware.

Besides, if you try to make Stellarium look good but feel slow in a classroom, you make a bad impression of the software on the students, which are used to video games being smooth.

Modern students should have laptops that are less than half their own age. Most should have at least some Intel HD in their Core-i5 3xxx+ GPUs.

we deliver a big extra binary that most people likely really don't need

This already bad because of this PR, because the repo clone takes longer now. If we refrain from storing the binaries, we can squash (or remove) this branch to avoid this.

This is why I asked very early where to put those binaries (#2803 (comment)). Unfortunately the answer did not include useful instructions for me. If I delete everything and force-push, will that remove the cruft, or what actions can one take to let the repo completely erase the DLLs?

Qt5/mesa users may profit from dithering. Qt6/mesa users may also get ShowMySky, at least on low framerates.

Let those who want fancy graphics use decent (not even fancy!) hardware.

As said, there may be situations where no GPU is available and people are happy with good graphics at 7fps@UWQHD.

option 2 does everything which option 3 does (the changed wrapper just sets 2 more envvars which are ignored by the default solution), with less instructions for upgraders.

This spreads the upgrade over the project, while with option 3 we'd only have it in the instructions, without additional maintenance for the code. I expect this upgrade to be ignored by most of the users of Mesa mode, simply because the whole software rendering mode is a workaround to get astronomical information, not graphics.

What you write here is already more than the change in CLIprocessor code. I only added two environment variables which become active when somebody copies the files. I don't see any further maintenance burden. Instructions in the Guide are required for 2 and 3.

BTW this PR aims at solving #2803 where you all agreed on replacing old Mesa libs...

[10110111]

Stellarium is not a game.

And this is the reason it still has a legacy mode. Better graphics certainly implies higher requirements on hardware, moving Stellarium towards the game-like class. In the future the requirements of the high-graphics mode will be more taxing on software renderer, so the users won't like such use anyway.

Unfortunately the answer did not include useful instructions for me. If I delete everything and force-push, will that remove the cruft, or what actions can one take to let the repo completely erase the DLLs?

Yes, the clone will get lighter if you somehow remove the commit that introduces large files. This can be achieved by squashing all the commits here into one (since 7e63de3 removes the files), or by removing the branch.

What you write here is already more than the change in CLIprocessor code.

How so? My suggestion is to only update the docs and leave the code alone.

But anyway, you asked to vote, I voted. If you really want it, I'm OK with option 2.

[gzotti]

And now I found this...

https://fdossena.com/?p=mesa/index.frag

A single 36MB dll with Mesa 20.1.8. Renamed to equal Qt's name, and it provides OpenGL3.3.
Wow: Running 3x side-by-side, I see

• 23.2 16fps (8fps full-screen)
• 23.2+ (Alex' latest build with that 3-dll solution of Mesa23.1.7) 12fps (6fps full-screen)
• 23.2+ (this branch) with fdossena's Mesa20.1.8 24fps (14fps full-screen)

OK, this means we can fully delete this branch without a trace (what are the git commands to cut away the history?), and all that needs to be done is writing instructions for DLL exchange. I am not sure about license from fdossena, so I am not sure if we can just deliver that instead of Qt's Mesa11 dll. It's just a build result, though, so maybe MIT license applies.

[10110111]

23.2+ (this branch) with fdossena's Mesa20.1.8 24fps (14fps full-screen)

Suspiciously fast ;) Did you check that you're actually running via Mesa (opening the log from Stellarium GUI to make sure there's no mistake)?

what are the git commands to cut away the history?

If your branch is current, you can reset it to the state of master with

git reset --hard master

Then just force-push (this will likely close this PR if there're no commits on top of origin/master).

A single 36MB dll with Mesa 20.1.8.

This is still quite a lot for the source repo. Can we store it in a separate repo, like stellarium-data?

[gzotti]

23.2+ (this branch) with fdossena's Mesa20.1.8 24fps (14fps full-screen)

Suspiciously fast ;) Did you check that you're actually running via Mesa (opening the log from Stellarium GUI to make sure there's no mistake)?

Of course. Needed the logs in F1 panel to identify which is which... I have an insane i7-12700H (14 cores/20 T) which of course helps on that. It seems 14 threads are active (66%CPU). My Geforce 3070 makes 46fps fullscreen with about 30% load while CPU is around 12%. (I have 1146 SSOs, these take their time and yes, we should seriously start multithreading here.)

what are the git commands to cut away the history?

If your branch is current, you can reset it to the state of master with

git reset --hard master

Then just force-push (this will likely close this PR if there're no commits on top of origin/master).

OK, if @alex-w agrees. After that reset I will add the SUG changes again.

A single 36MB dll with Mesa 20.1.8.

This is still quite a lot for the source repo. Can we store it in a separate repo, like stellarium-data?

If @alex-w is OK with that, we can just replace the existing Qt5.6 dll. Or leave it to users to upgrade with instructions.

I cannot say why, but suddenly even Mesa11 on 23.1 reported OpenGL3.3. (???)

[10110111]

I cannot say why, but suddenly even Mesa11 on 23.1 reported OpenGL3.3. (???)

Something seems broken... or maybe it already did this before on high-end GPUs?

[10110111]

Or leave it to users to upgrade with instructions

Given the performance improvements, I think it's worth doing this ourselves. But I wouldn't hurry for the nearest release, because this still needs some testing, including by the beta users.

[alex-w]

If @alex-w is OK with that, we can just replace the existing Qt5.6 dll. Or leave it to users to upgrade with instructions.

Why not? Or you can re-use mesa-3d tag/release and update a .appveyor.yml file

P.S. updating SUG should be enough IMHO

[github-actions]

Hello @gzotti!

Please check the fresh version (development snapshot) of Stellarium:
https://github.com/Stellarium/stellarium-data/releases/tag/weekly-snapshot

[github-actions]

Hello @gzotti!

Please check the latest stable version of Stellarium:
https://github.com/Stellarium/stellarium/releases/latest

[github-actions]

This pull request is outdated, such, the problematic code replaced by new tool or feature...

[gzotti]

This buried attempt has been superseded by #3414.