Space Cannon Energy Ball projectiles are not always visible

[mcphedar]

Describe the bug
In the 4/10 playtest. The space cannon's energy ball projectiles cannot always be seen. This issue is inconsistent between players, with some seeing the balls as they fly and others being unable to. Their visibility also depends on the world position and potentially the direction they are fired in. All players can see the balls forming at the energy cannon, but when the projectile VFX starts, they disappear. Sometimes, upon player respawn or reload the balls became visible again.

Steps to reproduce
Steps to reproduce the behavior:

1. Build o3de-multiplayersample
2. Open the NewStarBase level in the editor
3. Play the game (ctrl-g)
4. Observe the various energy cannons placed around the level. Some may fire visible projectiles, but most will not.

Expected behavior
You should be able to see the projectiles as they travel

Actual behavior
Visibility is inconsistent

Screenshots/Video
(https://user-images.githubusercontent.com/67011188/232088761-6d97b8d3-eb13-404f-9a2a-5437aad63972.mp4)

Found in Branch
03de-multiplayersample stabilization/2305

[mbalfour-amzn]

From the PopcornFX folks:
"I suspect a culling issue (plugin bug maybe) it's definitely not an issue in atom.

Is there a way to prevent the energy ball effects from getting batched together?
Unfortunately no, there are several options:

• self-sorting issues (flickering of several draw calls from the same emitter): we can use "Camera sort offset" to fix that
• sorting issues between other emitters from PKFX or in the scene: we have a experimental system in place "draw calls slicing", producing sub draw calls depending on the view angle which can help . However I am not sure if atom sorts draw calls or world entities. If world entities are the one being sorted instead of draw calls, slicing won't help
• "Global sort override" can be used as a global sort offset (for example, render all popcornfx draw calls behind atom fog, glass, ..) which obviously won't work great depending on the situations"

[HugoPKFX]

Hi,
We'll look into this asap

[mbalfour-amzn]

So, I defnitely suspect a culling issue for the invisible particles. From a PIX GPU capture, when the ball isn't visible, there is nothing submitted for it. I'm not sure where exactly the culling happens though, since CAtomFrameCollector::EarlyCull / LateCull always return false. Is there somewhere else in PKFX that would cull it?

"Global sort override" can be used as a global sort offset

I can't find anywhere that camera sort offsets or global sort overrides are implemented. In CPopcornFXFeatureProcessor::Render(), the draw packet sort key is hard-coded to 0. I've locally tried changing that value globally to other values (1, 1000, etc) but that doesn't fix either of these problems though.

I am not sure if atom sorts draw calls or world entities. If world entities are the one being sorted instead of draw calls, slicing won't help

Atom sorts draw calls. They're sorted first by sort key, then by depth. For MPS, nearly everything uses a sort key of 0, so for all intents and purposes everything is being sorted by depth. This leads to the concern I have with the batching. Imagine we have 4 energy ball effects spaced across 80 meters, and all 4 are visible. In the energy ball effect, let's say it's made up of one billboarded material that's 2 meters high, and one billboarded material that's 1 meter high. What I'm seeing in PKFX is two submissions, one for each material. The first one has a bounding box of 81 meters (80 meters between centers, plus half of 2 meters for the billboard size). The second one has a bounding box of 80.5 meters (80 meters between centers, plus half of 1 meter for the billboard size). The submitted depth is 81/2 = 40.5 for the first, and 80.5/2 = 40.25 for the second. This causes multiple problems. One is that the second billboard in the effect will sort closer simply because it's smaller. The second is that we're getting a depth of ~40 meters for all 4 effects, even though they might be at 10 m, 30 m, 50 m, and 80 m in the scene.

The biggest concern I have is the culling that I suspect is happening but can't find, but the batching just seems like it's always going to have problems if we have the effect spread out too far apart as well.

[mbalfour-amzn]

A couple of other things to notice in this video:

1. The arrows on the jumppad effect also appear to be missing, even though the rest of the effect is there.
2. When the energy ball is missing, it is completely missing - no billboards, no distortion effect, no ribbons, no light. So whatever the problem is, it seems to be related to the particle emitter, not to an individual renderer.

[mbalfour-amzn]

I believe I've got a solution to the missing energy ball effect - it appears to be bugs in the EnergyBallComponent in the MultiplayerSample, not a culling issue. I can reproduce the problem from the same angle with the same code, but just with different amount of other things in the level that change the timing of how the client & server communicate, so it looks much more like a timing problem than a rendering problem.

I'm still investigating, but I believe the code sometimes calls enable/disable in the wrong order for the particle effect, or disables multiple times before enabling, or something. I should have more details on Monday.

The sort ordering problem still exists though, that one is definitely a separate problem from the missing energy balls.

I also ran into a sporadic crash in the Editor while testing which is ultimately due to the fact that PopcornFXIntegration::_Clean() will cause any existing emitter pointers to get deleted, even though other components might still have references to them. I can submit a separate bug for this if you'd like. IMO, the solution to the crash could be to do one of the following:

• (better) Change the SpawnEffect*() methods to return a shared_ptr or other type of lifetime-preserving thing instead of a direct pointer so that nothing can hold onto a direct pointer and try to use it outside of its lifetime
• (medium) Change PopcornFXIntegration to check for isEnabled inside of all of its API calls and do nothing if it isn't. This would fix the problem in the specific case I observed, since it looked like it was probably an order of operations problem where the PopcornFXIntegration got the signal to exit game mode before some of the other entities had a chance to clean themselves up, so the other entities still tried to do things with the emitter pointers on the way out of game mode. isEnabled was definitely false at the point of the crash that I saw. The main concern I'd have with this fix is that there still might be other use cases in which the emitter pointers get held onto past their lifetimes that I just didn't observe with this one specific crash.
• (worse) Change every method in PopcornFXIntegration that takes in an emitter pointer to call IsEffectAlive() before using it. This looks like it would be pretty terrible though, because it would do an O(N) lookup of that pointer on every single API call.
• (worse) Add an EBus notification whenever an emitter pointer is destroyed, so that anything that has an emitter pointer can at least have a chance of knowing that it's destroyed. This would work, but it puts a burden on every system that calls SpawnEffect() to listen for this, which would be easy to get wrong.

[mbalfour-amzn]

@HugoPKFX I finally tracked down the specific timing issue as far as I can take it. The problem is somewhere in PopcornFX code. Attached is a level that demonstrates the problem. Basically, if Kill() is called on a particle effect on the first tick, it will never make it to an OnDeath() state and so any calls to Restart() will get stuck waiting for the death to occur. If Kill() is called any time after the first tick, it seems to work correctly.

The attached level has a Script Canvas script that exposes the number of ticks to wait before calling Kill(), the number of ticks before calling Restart(), and the total number of ticks before looping. When the kill ticks is set to 1 (the first tick), the ball never appears. When it is set to 2, the ball disappears and reappears correctly.

This problem can be worked around either by using Restart(false) (to call terminate instead of Kill) and/or using Terminate() instead of Kill() to stop the effect, but calling Kill() on the first frame ought to work.

ParticleClientOnly.zip

[mbalfour-amzn]

@HugoPKFX I've submitted 3 issues over in the O3DEPopcornFXPlugin repository to capture everything I've described above:

• Sporadic Editor crash on editor-game mode transitions PopcornFX/O3DEPopcornFXPlugin#41
• Calling Kill() on the first tick puts particle emitters in a permanent undead state PopcornFX/O3DEPopcornFXPlugin#42
• Sorting is difficult / impossible to control for particles with transparency PopcornFX/O3DEPopcornFXPlugin#43

[HugoPKFX]

@mbalfour-amzn Thanks for creating these issues with detailed explanations, we are setting up the multiplayer project to repro these (will reply on each issues too)

[AMZN-Gene]

Fixed in stabilization