Memory tweaks and proper Windows support

[kubilus1]

• Should work on Windows without WSL2 now. Basically install WinFSP first.
• Tweaks to how we cache and handle tiles in memory
• Add caching of individual satellite images
• Fix some issues where multiple reads after header can occur but we aren't referencing mipmap0

For now, users should ignore the run.bat file and run this from the command prompt or powershell:
Install prereqs:

py -m pip install -r requirements.txt
py -m pip install memory-profiler
py .\autoortho\autoortho-winfsp.py <ORTHO PATH> <CUSTOM SCENERY MOUNT POINT>

Replacing <ORTHO PATH> with the ortho foler and <CUSTOM SCENERY MOUNT POINT> to the path in your XPlane custom scenery dir to mount this.

---

This discussion was created from the release Memory tweaks and proper Windows support.

[kubilus1]

Currently this is in pre-release and needs some cleanup, but should work for most use cases.

[betampona]

On the last few posts you told us on one:
If on Windows run with
py autoortho/autoortho-winfsp.py
and on another one :
py .\autoortho\autoortho-winfsp.py
which one is the right one?

Currently I'm trying with both but I get the same error message [Errno 2] No such file or directory
Here is what I am typing:
py .\autoortho\autoortho-winfsp.py C:\users\dominique\ortho\z_ortho G:\X-Plane 11\Custom Scenery\z_autoortho
I'm on 2 different disks

Translated with www.DeepL.com/Translator (free version)

[ekc007]

I had the same problem. You have to try the second one and be at first in the right directory "cd C:\autoortho-0.0.5".
then you would use the second command but there is a typo..it should be "autoortho-winfspy.py". This way he finds the path but in my case the error pops up "module ... not found". So I tried to install (found this solution over the internet) winfsp module with typing "pip install winfspy". Then "py .\autoortho\autoortho-winfspy.py C:\users\dominique\ortho\z_ortho G:\X-Plane 11\Custom Scenery\z_autoortho"

Do this all before creating the directory "G:\X-Plane 11\Custom Scenery\z_autoortho". It will be created automatically and you can check it thereafter

[ekc007]

For me there were also problems with recognizing the paths if the is anywhere a space in between. in this case put the path in "path"

[betampona]

I went through the command prompt to get the right directory cd C:\autoortho-0.0.5
Then I installed the winfsp module by typing "pip install winfspy".
then I typed the paths py .\autoortho\autoortho- winfspy .py C:\users\dominique\ortho\z_ortho G:\X-Plane 11\Custom Scenery\z_autoortho paying attention to the possible spaces
I did not create a z_autoortho in the custom scenery
see the picture I end up with the same error.
For path I do not know the manipulation

Translated with www.DeepL.com/Translator (free version)

[kubilus1]

New pre-release version is available: https://github.com/kubilus1/autoortho/releases/tag/0.0.6

This changes a few things around, but hopefully should be a bit easier to use:

• A configuration start screen is now present.
• The orthophotos directory config option should point to your extracted orthos directory
• XPlane scenery dir should point to the Custom Scenery dir in your Xplane install
• This will automatically attempt to setup dirs based on these locations and should warn you if something doesn't look right.
• An option to override the maptype is present. So you could start with 'Arc' tiles and switch to 'BI' tiles.
• Lots of tweaks and bug fixes.

Windows users should:

• Make sure they have WinFSP already installed
• Make sure they have an up to date Python installed (with pip)
• Extract the autorotho-0.0.6.zip and execute with 'run.bat'

Linux users should:

• Make sure fuse is installed and setup.
• Make sure an up to date Python and pip is installed
• Install pre-reqs: python3 -m pip install -r requirements.txt
• Run with python3 autoortho

So far my testing shows this works on Linux and Windows for both Xplane 11 and 12. I'll do some more cleaning up before I make this a proper release.

[betampona]

Eureka it works, I did a test it looks really good on a BI 17 tile.
Can we have a button on the desktop to quickly launch the run.bat and fly when launching X-plane 11, this would avoid going to the disk.
I often start from the old arrival airport to a new destination, could we have an archive of the old location to make it quicker to launch x-plane?
You have done a great job that few people thought possible.

Translated with www.DeepL.com/Translator (free version)

[kubilus1]

Autoortho will cache the satellite imagery as it downloads it, which should make subsequent runs faster. Currently I do not cache the XPLane formatted tiles, but that's something I can consider. These take up a bit more space so I'll have to figure out how to make that efficient.

Glad it is working for you!

[ivotaHR]

1. PS C:\autoortho-0.0.6\autoortho> python3.exe .\autoortho.py
   Windows detected
   ChunkGetter got: 0
   chunk_getter: <getortho.ChunkGetter object at 0x000001F32CCB4550>
   Config file found C:\Users\Ivica.autoortho reading...
   Config file found C:\Users\Ivica.autoortho reading...
   Updating config.
   {'orthos': 'C:/autoortho-0.0.6/autoortho/0/z_utah', 'Browse': '', 'scenery': 'C:/X-Plane 11/Custom Scenery', 'Browse0': '', 'showconfig': True, 'maptype': 'BI'}
   {'general': {'# use gui config at startup': '', 'gui': 'True', '# show config setup at startup everytime': '', 'showconfig': 'True'}, 'paths': {'# ortho photos path': '', 'orthos_path': 'C:/autoortho-0.0.6/autoortho/0/z_utah', '# x-plane custom scenery path': '', 'scenery_path': 'C:/X-Plane 11/Custom Scenery'}, 'autoortho': {'# override map type with a different source': '', 'maptype_override': 'BI'}}
   Wrote config file: C:\Users\Ivica.autoortho
   Wrote config file: C:\Users\Ivica.autoortho
   Config file found C:\Users\Ivica.autoortho reading...
   Config file found C:\Users\Ivica.autoortho reading...
   Preparing directory structures....
   Diretories are ready!
   root: C:/autoortho-0.0.6/autoortho/0/z_utah\textures
   mountpoint: C:/X-Plane 11/Custom Scenery\z_autoortho\textures
   Running in Windows WinFSP mode.
   Started tile clean thread. Mem limit 2147483648
   Started tile clean thread. Mem limit 2147483648
   NUM TILES CACHED: 0. TOTAL MEM: 52 MB
   NUM TILES CACHED: 0. TOTAL MEM: 52 MB
   TILE CACHE: MISS: 0 HIT: 0
   TILE CACHE: MISS: 0 HIT: 0
   Starting FS
   Stopping FS
   FS stopped
   Traceback (most recent call last):
   File "C:\autoortho-0.0.6\autoortho\autoortho.py", line 569, in
   main()
   File "C:\autoortho-0.0.6\autoortho\autoortho.py", line 544, in main
   autoortho_winfsp.main(root, mountpoint,
   File "C:\autoortho-0.0.6\autoortho\autoortho_winfsp.py", line 867, in main
   fs.start()
   File "C:\Users\Ivica\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\winfspy\file_system.py", line 172, in start
   raise WinFSPyError(f"Cannot mount file system: {cook_ntstatus(result).name}")
   winfspy.plumbing.exceptions.WinFSPyError: Cannot mount file system: STATUS_INVALID_PARAMETER
   PS C:\autoortho-0.0.6\autoortho>

[ErikSchlijper]

How did you fix this? I'm having the exact same issue!

[ivotaHR]

It is working!!!
Thank you We need Global scenery pack.

[kubilus1]

That's exactly what I've been working on ;)

[betampona]

Hello, I have a test in IFR with the zibo between Nice and Paris in BI zoom 17, I had misses of tiles due to the speed of the plane compared to the reloading of the tiles although I have a flow of 60 mb/s.I am doing a test on France in BI 15 without the zoom 18 near the airports, for the moment I am downloading on Ortho4xp, then I will make you a return.For VFR I think it's okBut for IFR you need a zoom level lower than 17 unless you find a way to make the zoom fluctuate according to altitude and speed.Greetings.Translated with www.DeepL.com/Translator (free version)envoyé : 16 octobre 2022 à 15:26de : ivotaHR ***@***.***>à : kubilus1/autoortho ***@***.***>cc : betampona ***@***.***>, Comment ***@***.***>objet : Re: [kubilus1/autoortho] Memory tweaks and proper Windows support (Discussion #3) It is working!!! Thank you We need Global scenery pack.—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[kubilus1]

I found that zoom level 16 with a higher airport zoom level is a good compromise between speed and quality. Particularly at 1080p resolutions.

This tool already will get the minimum possible amount of data dynamically by effectively retrieving only the resolution of data that XPLane requests at any given time. If you are flying higher, or something is far, you don't get the full resolution of a tile.

At higher zoom levels, you effectively have many more tiles per area, which means more time and downloading will occur. This very well may be waste in a lot of cases. XPlane will only actually read in a limited amount of data per tile at any given time. You can see this in the XPlane developer tools tile explorer. It will indicate the resolution it actually has read in. 4096x4096 is full resolution, but often you see much lower. If your graphics settings are not maxed out for textures you may never actually read in full texture resolution of a tile.

60mb/s is on the slow side for internet, that combined with high ZL (ZL17) and perhaps you are far from the tile servers. Your internet may also have latency that isn't measured in the download speed. Autoortho will cache your downloaded tiles though, so subsequent runs should be a bit faster.

Another factor to consider is your CPU. This tool will generate the format XPlane needs on the fly. A slower CPU will mean slower processing of new tiles and possible delay.

[ivotaHR]

When we fly ,does the cache (2GB usage-limit) automated deleted, updated??

[kubilus1]

The 2GB memory limit is a target for retention of the on-the-fly processed tiles. It will automatically attempt to remove tiles when it can. The on-disk satellite imagery cache though does not automatically get cleaned up.

[ivotaHR]

For me ZL16 is top of my choice. No need better. I am on 1080p. I can use due data traffic, cpu usage ZL15 or less.

[betampona]

I just did a test in BI15 with the airports in ZL18 (I thought I had removed them)
Overall the flight went well LFMP LFPG 500 nautical miles flight time 1h20, I was with the zibo at FL 360 ground speed 480kts.
I had a few pale blue tiles on the right side of the screen at half height, limit between the near and far scenery.
By continuing the flight this blue tile disappeared and the scenery came back.
On the right side no problem.
Do you have any idea about this problem?
The quality of the BI15 is suitable for high altitude flights.

My fibre at the box is 90 Mbit/s or 11.25 Mb/s but my computers are in the attic and I have 70 Mbit/s or 8.75 Mb/s through PLCs
In France there are 2 fibre contracts, the first one is 400 Mbit/s but in reality we have 100 Mbit/s, the second one is 1Gbit/s but it must be much more expensive and I have strong doubts about the speed.

Translated with www.DeepL.com/Translator (free version)

[kubilus1]

The blue areas are basically what XPlane does when it is waiting to read a DSF file. You will get the same result if you try to load orthos from a slow hard drive, for instances.

I'm not sure a fully understand how you are describing the computer in your attic, but if this is over a slow local network, that could cause extra delays, and perhaps the blue tiles you saw.

[betampona]

Hi kubilus1, you specify us option in run.bat:
An option to override the maptype is present. So you could start with 'Arc' tiles and switch to 'BI' tiles
what is the advantage of this option?

[kubilus1]

This allows you to try out different tile set types. BI is from Bing maps. Arc is from ArcGIS, etc.

[betampona]

You managed to make autoortho work under windows, what made you progress in the realization of the project compared to this summer when everything was blocked, you always used winfsp and python ?

[kubilus1]

So there are a lot of little reasons. To start, understand that Linux and similar operating systems have the concept of userspace filesystems through a library called FUSE. This makes it very simple to interact with a filesystem and create filesystem for various purposes. This is the technique that this project is based off of.

WinFSP provides a somewhat analogous tool for Windows, but works differently. It is advertised though to have a FUSE compatibility layer. This is initially what I tried out for Windows compat. It works in some modes, such as with file explorer and editing tools but for a variety of little reasons XPlane would refuse to recognize WinFSP-FUSE file system mounts.

So it is possible to write against WinFSP at a lower level, but that is a lot more work, and typically done in C or C++. Which I can program, but that's a lot lot more effort. Plus I don't use Windows, so it's not a huge amount of value to me. I also did not have much of a system to test out Windows changes.

I discovered a project though that wraps WinFSP for python and determined it may be feasible to create a new layer using this just for Windows users. This is the current implementation. Plus I found a little bit of time and energy to work on it.

[betampona]

Thanks for the reply, I have just switched to 1Gb/s fibre instead of 100Mb/s, I have gained in launch time from 4mn to 2mn40, I am in ZL15 with ZL18 on the airports. I'm going to test the scenery, see if I don't have any more blue tiles synonymous with delay on the scenery. As I'm flying at 480 kts or 900 km/h at FL from 300 to 380 the tiles renewal are very close in time. Last time with the old fibre the blue tiles were at minimum.
Greetings.

Translated with www.DeepL.com/Translator (free version)

[betampona]

For the flight no problem of downloading tiles in BI 15 for IFR that seems to me good, to test BI16.
I have attached a picture of the download speed and periodicity, would you be interested?

[betampona]

After cleaning, I re-downloaded and installed everything, see photo.
I looked under C: with powerShell the version of python 3.10.8 and pip 22.3
I'm not able to access the implementation interface with run.bat.

[betampona]

If you are talking about autoortho.log it is empty, as for the run.bat the python screen does not stay on the screen.When I go to the autoortho folder of autoortho-0.1.0 and click on autoortho.py I launch the autoortho.setup image, I enter the direction of the ortho folder and the direction of the custom scenery but when I click on Fly the python image disappears, on the other hand I have the creation of an .autoortho or in the paths part I have the directions saved, but as the python screen does not stay in run.bat when I launch x-plane 11 there is no download.envoyé : 6 novembre 2022 à 16:38de : kubilus1 ***@***.***>à : kubilus1/autoortho ***@***.***>cc : betampona ***@***.***>, Comment ***@***.***>objet : Re: [kubilus1/autoortho] Memory tweaks and proper Windows support (Discussion #3) Did you see anything in the log file? I would make sure you still have WinFSP setup. The run.bat is pretty basic, you could try manually executing those commands and see what happens.—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[kubilus1]

Instead, try to open 'powershell' go to the directory you have extracted autoortho and type the commands that are in the run.bat file.

First:
py -m pip install -r requirements-win.txt

Followed by:
py autoortho

[betampona]

I typed the 2 command lines in autoortho-0.1.0 see the picture.
run.bat is still not active

[kubilus1]

Okay, this is helpful, I think I see the issue. The quick fix is for you to just do a py -m pip install fusepy.

[betampona]

On the attached picture I typed in powershell the direction to run.bat.
it appears below C:>py -m pip install -r requirements-win.txt an error line in red, ERROR: Cloud not open requirements file: Errno 2 No such file or directory: "requirements-win.txt"
is this normal?

[betampona]

py -m pip install fusepy solved the problem, I did a test everything is ok.

[betampona]

Hello, I have just looked at the download speed of the scenery during a first flight using the task manager. I never exceed 25MBits/s and it takes about 6 minutes to launch the scenery in x-plane 11, yet in fibre I have 300Mbits/s. Where can this limitation come from?
If you do the same test with the task manager what value do you have?

[kubilus1]

I suspect the bottleneck is the satellite image service itself (bing maps, ArcGIS, or whatever else). It's possible CPU is your limit. A test you can do is to re-load the same flight. For me this will take about 5-6 min the first time, but then about 1-2 min following times. If you see it taking around 6 min every time and your CPU is maxed, then you are CPU bound.

[ivotaHR]

https://imgur.com/a/8CXukOV

TIP:
Is possible to make tool like this for make ortho dsf files (to collect files from route)? Or we should wait full world ortho dsf scenery package.

[kubilus1]

So Ortho4XP can make these of course but you have to ahead of time. It's also a pretty complex and difficult to use tool for beginners.

[betampona]

I am in the same case as you, I put 5 to 6 mn the first time then 1 mn if the cache is loaded. I am away from home for the moment but there is a command in powershell to optimize the download (internet search)Envoyé depuis mon appareil Galaxy -------- Message d'origine --------De : kubilus1 ***@***.***> Date : 12/11/2022 02:33 (GMT+01:00) À : kubilus1/autoortho ***@***.***> Cc : betampona ***@***.***>, Comment ***@***.***> Objet : Re: [kubilus1/autoortho] Memory tweaks and proper Windows support (Discussion #3) I suspect the bottleneck is the service itself. It's possible CPU is your limit. A test you can do is to re-load the same flight. For me this will take about 5-6 min the first time, but then about 1-2 min following times. If you see it taking around 6 min every time and your CPU is maxed, then you are CPU bound. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[kubilus1]

Yes, this is not surprising then. I am working on a change that should reduce the amount of downloaded data a bit more as well, so perhaps could speed this up somewhat, but don't expect anything too dramatic.

[kubilus1]

Here's a pre-prelease version that should somewhat reduce the downloaded data: https://github.com/kubilus1/autoortho/releases/tag/0.1.2

[ivotaHR]

Here's a pre-prelease version that should somewhat reduce the downloaded data: https://github.com/kubilus1/autoortho/releases/tag/0.1.2

I tested it seems all ok.

[betampona]

I compared the downloads of version 01.0 and 0.1.2
the times are taken when run.bat starts to load.
On Alice Springs YBAS version 0.1.0, 1st download 5'33" to get the scenery when the cache is created 53" to get the scenery.
for version 0.1.2, 1st download 5'17" for the scenery and 46" when the cache is created.
I noticed that the number of tiles in the cache has decreased.
The best would be to have the possibility to download only the tiles in jpeg format of 256 by 256
A remark on ortho4xp the jpeg tiles are 4096x4096 and on autoortho they are 16 times smaller.

Translated with www.DeepL.com/Translator (free version)

[kubilus1]

Thank you for testing that. So basically a 5% speed improvement when downloading images from scratch and a 13% improvement when already cached. That's about what I expected. The only really practical way to decrease downloads more would be to reduce the ZL that is used.

Your airport ZL in particular, since you start on the ground, you typically are downloading the maximum size.

I think it may be technically possible to speed up the tile processing as well, but probably won't make much difference. Some time is taken by XPlane even if you don't use any of this stuff, afterall.

[betampona]

I did a test with ZL13 on Alice Springs it takes 22 seconds to get the set and 17" if the cache is loaded.
I made an attempt by mixing ZL16+18 and ZL13 in the same folder, I got the scenery in 22" autoortho stayed there, it did not load ZL16+18 because it started with the 12 tiles in ZL13 and as they were present in x-plane11 it blocked the other part of the download.
Yet it would be best to start in a degraded ZL13 form and then finish the download in ZL16+18

Translated with www.DeepL.com/Translator (free version)

[betampona]

Flight between YBAS and YPED FL370 speed 450 kts
as soon as I start at low altitude I have the appearance of a multitude of checkerboards on the ground very close to the plane, I no longer have the scenery in ZL16 with a certain depth, the scenery in ZL16 is just in front of me afterwards it's blurrier and in the distance it's blurred but it's normal on x-plane.
At FL370 level see the pictures, in front of me ZL16 in sawtooth on very little ground surface, the ZL16 should go to the normal limit allowed by x-plane.

Translated with www.DeepL.com/Translator (free version)

[kubilus1]

What versions are you using? Probably from the source satellite photos. Take a look at your map source and zoom in and out and you'll probably see a similar effect. I could possibly make the minimum fetched zoom level tunable.

[betampona]

For orthophotos I use your AUS BI16 pack
Here is a picture at FL370 with the autoortho-0.1.2 version

The following photo is with autoortho-0.1.0 at the same height and about the same place, there is no checkerboard and the ZL16 fills well its zone which is dedicated to him, it is much more pleasant visually even during the rise, it is of the traditional orthophoto that one finds on x-plane.

[kubilus1]

Thanks for that comparison. Easy fix, effectively ver 0.1.2 allowed a min fetched ZL of 11 where 0.1.0 allowed a min ZL of 12. I think we'll want 12 to be the default.

[betampona]

Have you released a new version?

[betampona]

I forgot to tell you that on the 1st photo autoortho-0.1.2 the undercoat is sweating on the ZL16 layer.

[betampona]

Hello, I see you are working on the code.
What are these changes?
Can we expect a new version soon?

[ivotaHR]

I am looking for new version , too.

[kubilus1]

Good stuff is on the way. Just doing a little final testing and cleanup. This will be a fairly significant update:

• GUI based ortho package downloader and installer
• Replacing the DDS file creation tooling (2X speed up when processing files!)
• Re-organizing code, etc
• New satellite source choices

[kubilus1]

FYI new release has been cut!

See: https://github.com/kubilus1/autoortho/releases/tag/0.2.0
Discussion thread: #20