MEGA65: sprite priority dictated by RRB gotox token

[lgblgblgb]

VIC-IV seems to provide an extra control over sprite priorities than the usual VIC-II way. The GOTOX token colour RAM byte 0, bit 6 and bit 2 provides extra control: if bit 6 is set, the following characters are treated as background (so sprites can show up in the front, even if sprite is set to background). Bit 2 is similar but then treated as background. This whole stuff is not yet emulated by Xemu at all.

Any feedback is welcome, currently I am kinda that:

• how it works exactly
• how it can be implemented in the emulator in an efficient way keeping all the mechanics more less intact

Some questions about the first problem (how it works at all):

1. when either bit 6 or bit 2 is set, this mode will remain to the end of the scanline, regardless of multiple GOTOX tokens maybe? I guess no, further GOTOX tokens must be set those bits if the effect is still wanted to be used
2. So then, $D01B is totally ignored if either bit 6 or bit 2 is set?
3. I am totally clueless how some can control the "sprite layer" (let's call that way), since multiple RRB layers may dictate different policies for the very same pixel position multiple times ...
4. What happens if both of the mentioned bits are set?
5. If xpos already passes a sprite, then an RRB GOTOX token is used to be at the start of the raster buffer again with different bits settings of sprite priority, what would happen? MEGA65 also only renders sprites for real at the very end of scanline processing when there is no more possibility to use GOTOX or anything? I mean, if sprite pixel is already rendered (passed the xpos position where there is a sprite already) then how to modify it? If this is the case, that's kinda surprising and more like how Xemu works, having the sprite rendering pass after all the scanline content is done otherwise even with all the RRB stuff.

The Xemu problem:

I am not sure if it can be implemented in Xemu at all in a same way, even if I understand how it should work (which is totally not the case currently, it's waaaaay more complicated for my brain ...). The problem: Xemu's performance is quite acceptable because it avoids to do different things at a time, which also applies to VIC-IV emulation. For example, sprite rendering is done in a single pass after all the scanline info is rendered already. This way, there is no need to interleave code with scanline "content" rendering and sprite rendering which would result much-much-much more poor emulation performance. This is because the content rendering can be a much tighter and more compact loop without too much condition checking at all (which is always pricey to have conditional jumps).

This is done with an auxiliary array called is_fg[]. It's filled by the scanline rendered, by pixel position: value 0 means background ("transparent") pixel, other means foreground. At the end of the scanline content rendering, the sprite rendering pass follows, which uses the is_fg[] array: if sprite is set to background priority, sprite pixel is rendered only if is_fg[pos] value is zero. This array is also used to check sprite-background pixel level collision in an sane way.

I'm very unsure how this scheme should/can be modified to fit our needs.

Preliminary implementation idea

Use is_fg[]. Sprite rendering currently uses this information: being 0 as a transparent (background) pixel for foreground, any other value is some pixel (non-background). If I modify this, to be 0 = background, other number = "layer number" (maybe incremented at earch "RRB layer"?), then I can have a value that sprite renderer can check if the is_fg[] at that position is how to be compared with a certain layer number. Modifications:

1. On sprite rendering, rather then checking if is_fg[] is zero or non-zero should be modified with comparisons.
2. On filling is_fg[]: must be modified to fill with the actual "layer" identifier when rendering non-transparent / foreground pixel
3. Current "layer id" must be maintained well
4. Sprite renders should know what layer id it should compare (ie, which layers they are front - but behind further layers)
5. Of course these all should not bother the functionality of the emulator when not in use, also: sprite collision can be an issue and should be checked!

[archiecrux]

Since I have some experience in this area, I would like to share my findings with you.

1. You are correct. Each GotoX sets/clears these relevant bits for the characters that follow.
2. These bits in the GotoX only apply to the sprite for which the corresponding bit is set in $D01B. This way, you can use the sprite background priority register to determine which sprites belong to this specific 'sprite layer'.

[archiecrux]

You can control the char/spr priority using the two bits in the GOTOX byte. If both bits (in the GOTOX) are clear, the sprite is drawn behind the char pixels, and in front of the background colour of the char, like VIC II does. If you set both bits, the sprite is only displayed above the char pixels, and behind the background colour pixels (when chars are solid).

The two 'control bits' behave like toggle switches. With bit6 you can determine the priority of the char pixels, with bit2 the priority of the background colour of the char, separately.

[lgblgblgb]

@archiecrux To be honest, the long prelude from me is meant to be as a kind of guide for myself when I have more time to dig into this (my usual way to write my thoughts into the issue first not to forget the problems later when I really have time to deal with the issue). But in any way, thanks for your comments, they will be very useful! <3