Adjusting the Fog of War Vision Grid for a rotated plane (and rotated camera)

[WolvenBard]

Hi there!

First off, I wanted to say fantastic job with this asset. I was a very early adopter and purchased this during your first release, but only recently have started to integrate it into my project.

Now a little bit about my current issue:

Context:
My game world uses a grid that exists on the X - Z plane (instead of the traditional X - Y plane for most 2D games).
Furthermore, my orthographic camera is facing this plane at a 45 degree rotation around the x axis like so:

Long story short, I do this in order to create a "2.5"d perspective that allows sprite objects tilted at 45 degrees easily sort in front of each other like a 3d game would. This is a very similar approach to what games like Enter the Gungeon do. Furthermore, I'm actually manipulating the projection matrix inorder to force an oblique projection so that I can get around any vertical distortion (though we can ignore the projection matrix alteration for now).

I'm trying to get your Fog of War functionality to work for this set up.

Problem
From what I understand, the logic behind the fog of war works like so: each tile of the grid is tracked along with its fade/visibility value. That grid is then translated to a texture which is then blit on the camera.

Now this works perfectly fine assuming the game world grid is on the X - Y plane. However, as mentioned above, my game world grid lays on the X - Z plane.

My highschool math lessons are failing me here, and I'm having trouble understanding how I have to manipulate the FogOfWarVisionGrid in order to account for my set up. My best attempt so far is trying to manipulate the X Y values within the GetVisionTextures method of the FogOfWarVisionGrid script, but I can't seem to get it right.

Any help/guidance would be greatly appreciated :) That said, I totally recognize this is beyond the scope of what your tool is made for (traditional 2d games).

[OndrejNepozitek]

Hello!

I have some questions for you:

• Do you have an example scene that you could share with me to better understand how your setup works?
• Do you use Cell Swizzle on your tilemaps to display them in the X-Z plane?
• What happens when you enable the FoW? Is everything in fog? Is nothing in fog?

[WolvenBard]

Hi!!

Thank you so much for such a prompt reply.

1. I can certainly mock up an example. Because this is an active game development, I'll have to either strip away a bunch of content or simply duplicate my setup in a separate project. So I'll do that and get back to you ASAP. If by any chance you're comfortable, I'd also be more than happy to showcase over a video conference (though I totally understand if you don't want to do that).
2. I do not use cell swizzle. I simply rotate the grid object by 90 degrees (a euler rotation value of 90, 0, 0). In order to get the generator to work, I'm actually using a custom handler that has a template structure where the grid is rotated, and furthermore I also as part of the post process I iterate through all the generated room objects (under the "Rooms" object) and swap their Y and Z values so that they end up at the correct position.
3. When the FoW is enabled, everything is fog (black in my case). If I remove the rotation of the grid (so that it is back to X and Y plane), and remove the rotation of the camera (have it face the X Y plane directly), the FoW looks perfectly normal. So basically the FoW vision grid seems to be generating perfectly fine, but its just existing on the X Y plane.

[WolvenBard]

I also just conducted a brief test by using cell swizzle (I set the grid swizzle to XZY and all the tilemaps orientations to XZ).
While this lets the generator correctly position all my room objects (I dont have to use my post process step to re-position rooms anymore), the FoW still doesnt work.

[OndrejNepozitek]

You might try this quick hack: Currently, in the FoW shader, for each rendered pixel, its tile position is computed so that we can compute the fog value. To compute this tile position, I use a view projection matrix of the current camera. There are these two lines of code in the FogOfWar.shader file:

// Compute world position rounded to whole tiles
float floorX = - _VisionTexOffset.x + worldpos.x - 1;
float floorY = - _VisionTexOffset.y + worldpos.y - 1;

As you can see, I use worldpos.y to compute the Y coordinate of the tile. But if you use the XZ plane, all the pixels will probably have the same Y coordinate which will cause the FoW to not work correctly. You can try to replace these lines of code with this:

// Compute world position rounded to whole tiles
float floorX = - _VisionTexOffset.x + worldpos.x - 1;
float floorY = - _VisionTexOffset.y + worldpos.z - 1;

Note that one y coordinate changed to z. Disclaimer: I haven't tested the code so it might not work. I can investigate it later if needed.

[WolvenBard]

Ah thank you for pointing me to the shader!

This is very close! After this change, the FoW works correctly but only if my camera is now pointing directly down at the XZ plane. However, the camera needs to be rotated at 45 degrees on the X axis (looking at the XZ plane at a 45 degree angle). I guess I need to adjust this line slight further to account for that?

[OndrejNepozitek]

Do you have a screenshot of how it currently looks like with the camera rotation?

[WolvenBard]

After changing that line of code to worldpos.z this is how it looks like if my camera points directly downwards (90 degrees):

However, if I used my desired 45 degrees, it is still black like so:

[WolvenBard]

At 45 degree rotation, I'm moving the camera along the local Y axis, but I can't seem to spot any red

[WolvenBard]

Apologies, I stand corrected. At a 45 degree rotation, I managed to find the red overlay but at a position way below. It also may look like its a bit bigger in size? I had to change the orthographic size to capture it in frame (but ill double check now to confirm that it is actually bigger):

[WolvenBard]

Actually no, it looks likes like its about the same size (I set the size back to 16, same as the picture 2 above - the one at 90 degrees):

So looks like its just getting shifted along the local Y axis then?

[WolvenBard]

I brute-forced some offsets to the worldpos.z value...
float floorY = - _VisionTexOffset.y + worldpos.z - 2690 - 1;

and managed to get it overlayed on the same room. However, the fog will move if I move the camera closer or further away from the target (along the camera's local z axis).

Also, it looks like the rotation might indeed be incorrectly scaling it vertically (it looks taller now):

[OndrejNepozitek]

I think that you might now have to somehow combine worldpos.z and worldpos.y to compute the correct floorY tile coordinate. Basically what I think is the only thing stopping the FoW from working is this conversion between pixel coordinates and tile coordinates. Once that's working, the whole FoW should work.

[WolvenBard]

I believe you're right, I just need to find the correct relationship here.

While I still haven't figured out how to combine that, I did at least figure out how to incorporate my custom projection matrix (I use an oblique projection matrix). Before, if I had my custom projection matrix enabled, I wasn't able to find the red overlay anywhere. However, after making this change in the FogOfWar.cs script to reset the projection matrix before querying it, it seems to work (although I still have the issue above where it is terribly offset):

Before:

            var viewMat = camera.worldToCameraMatrix;
            var projMat = GL.GetGPUProjectionMatrix(camera.projectionMatrix, false);
            var viewProjMat = (projMat * viewMat);
            var offset = (Vector3) (Vector3Int) visionTextures.Offset;
            offset -= new Vector3(1, 1);

After:

            // Cache the custom oblique matrix
            var originalMatrix = Camera.main.projectionMatrix;

            // Reset the projection matrix
            Camera.main.ResetProjectionMatrix();

            var viewMat = camera.worldToCameraMatrix;
            var projMat = GL.GetGPUProjectionMatrix(camera.projectionMatrix, false);
            var viewProjMat = (projMat * viewMat);
            var offset = (Vector3) (Vector3Int) visionTextures.Offset;
            offset -= new Vector3(1, 1);

            // Revert the projection matrix back to the custom oblique one.
            Camera.main.projectionMatrix = originalMatrix;

[WolvenBard]

Looking into the offset a bit more, I discovered that even if I brute force figure out the "correct" floorY tile coordinate, it behaves like it is in 3d space. If I move around, the opening (lack of fog) moves around too:

FoW-Test.mp4

[WolvenBard]

Re-uploaded video as it wasn't working before

[OndrejNepozitek]

I think that this repository https://github.com/chriscummings100/worldspaceposteffect might be worth looking into. There's a slightly different approach to computing world coordinates in image effects so maybe it works even with this camera rotation. I probably won't be able to try that right now but I should have some time in 1-3 days.

[WolvenBard]

Thank you for showing me this, I really appreciate all your help :)

I've been spending a bit of time picking apart the shader in that repo, using his method of getting the worldPos, but looks like I'm seeing similar results. That said, I'm not to familiar with shaders, so not sure if I'm missing something important.

[OndrejNepozitek]

I've got some news. It seems like tilemaps (or maybe all 2D objects?) are not "recorder" to the depth texture which is used when converting screen coordinates to world coordinates.

You can see that in the video below. The checkerboard texture should be aligned with the tilemap but it desyncs as soon as you start rotating the camera.

without_plane.mp4

BUT if I add a 3D plane to the scene which is aligned with the tilemap, the screen to world conversion starts working again.

with_plane.mp4

Now there's this obvious problem that you probably don't want to have a 3D plane in your scene.. But I think it's progress.l

The bad news is that I'm very new to shaders too so I don't really know how to utilize this information to get a fully working solution. What you can at least do is try this setup in your game. Add this additional 3D plane to your scene, align it with your tilemaps and see if it works.

[WolvenBard]

Excellent find, that may indeed be true. I'm trying to do a bit more research myself about tilemaps + depth textures.

As for your example showcasing with/without the plane, it looks like you've pulled down that other project right?

Is there a special material on the plane? I'm trying to replicate your findings with your FogOfWar example scene with just a simple plane object, but I'm having no luck. See the video below:

FoW-WithPlane_FogOfWarExampleScene.mp4

[OndrejNepozitek]

Try to add this line in something like Start() method:

GetComponent<Camera>().depthTextureMode = DepthTextureMode.Depth;

I imported the last released version of Edgar, added this single line and the 3D plane and the camera rotations work.

[OndrejNepozitek]

It looks like you just need to have the correct material for your tilemaps and then you don't need any plane.

As per Unity docs: (https://docs.unity3d.com/Manual/SL-CameraDepthTexture.html)

DepthTextureMode.Depth texture
This builds a screen-sized depth texture.

Depth texture is rendered using the same shader
passes as used for shadow caster rendering (ShadowCaster pass type). So by extension, if a shader does not support shadow casting >>(i.e. there’s no shadow caster pass in the shader or any of the fallbacks), then objects using that shader will not show up in the depth texture.

Make your shader fallback to some other shader that has a shadow casting pass, or
If you’re using surface shaders, adding an addshadow directive will make them generate a shadow pass too.
Note that only “opaque” objects (that which have their materials and shaders setup to use render queue <= 2500) are rendered into the >depth texture.