feat: view as surface

[Julusian]

Related: #1352 #2189

This adds the ability to view the button grid as a particular surface.
For now this is limited to reducing the buttons shown to the ones displayed by a surface, it will later be extended to support showing 'custom' layouts and graphics for surface types that define them.

The selection is remembered across page reloads/refreshes.

There is a toggle button added above the grid to quickly enable/disable this for the current surface, the main configuration for this tool is a panel on the right.

The 'Surface' dropdown offers every surface as shown in the surfaces tab of companion. We already cache the dimensions of these, so know what size grid to show, even when they are not connected

In some cases, we don't have dimensions (surface hasn't been connected since we started storing that), then a warning is displayed and no size change takes effect

You can also choose 'Custom' in the 'Surface' dropdown, which enabled the 'Surface Type' dropdown which is configured with every surface type that companion natively supports.
You can also manually adjust the x&y offsets to position as desired.

[dnmeid]

I'm not sure how much I like this. I always try to inline developments with #2387 and the UX there does not relate to single surfaces. As far as I see it at the moment, and please correct me if I'm wrong, this looks like a super complex way to hide some buttons. It adds another tab and many options and complex logic and stuff which leads to confusion when done wrong. "Why I can't see my buttons anymore?" "You have to check that..."
Both related issues are quite old, have not been asked for in the last two years and are in my opinion contrasting the general idea of Companion's UI. We want to make things universal, unified, easy. If the control grid does not match your single streamdeck, so what? People are used from Excel that cells are just there, nobody complains about the complexity of unused cells. The requests for surface sized views are from a time when we didn't have the infinite grid. Now people are starting more and more to use also controls at locations where no physical surface is.
I guess limiting the grid to one surface is a feature nobody really needs. OK, some would like to have it, but they don't need it. Some still are mixing up programming of buttons and use of buttons. On the other hand it just adds complexity.
If there is a real need of limiting views, which I don't see, I would prefer a different solution. Something more elegant, more easy, definitely not another tab with a lot of elements.

[bryce-seifert]

I like this idea personally, I think it will be helpful as I think button position is often something users can struggle with trying position surfaces on an infinite grid. I like the sanity check of "let me see exactly what's on this surface as the user sees it" while programming buttons.
Did some testing, and everything seems to be working as intended.

One possible suggestion, and there may be a more elegant way than mocked up below, but potentially some more visible flag that says "you're viewing as this surface" that way people don't confuse the view for missing buttons.

[dnmeid]

some more visible flag

From my side, even more NOs. Sorry to be the bad guy here, but that is a UX nightmare. Modes are bad UX.

What about this mockup:

Would this be easy enough to know where to put your button?

[Julusian]

I like this idea personally, I think it will be helpful as I think button position is often something users can struggle with trying position surfaces on an infinite grid. I like the sanity check of "let me see exactly what's on this surface as the user sees it" while programming buttons.

This is exactly the use case I am aiming for here.
Particularly the next portion of 'complex' layouts will target the case of "I am using surface X which doesn't conform to a grid pattern, and don't know/remember where each button falls in the grid"

I'm not sure how much I like this. I always try to inline developments with #2387 and the UX there does not relate to single surfaces.

I don't think this is in contrast with some of the ideas proposed in #2387. It is aiming to solve a similar scenario as that screenshot of yours.
but instead of forcing everything into a grid shape, it embraces that surfaces aren't perfect grids.
It isn't intended to be the only way you view the grid, but a way of checking alignment/positioning and making tweaks for a specific surface.

Im curious, how would you display this in that grid? It doesn't quite align to a grid nicely, so I think will be confusing to display like that. (there is an lcd segment for each button around it, meaning that portion is a 2x4, but then with centered buttons at the ends)

This is actually already supported in satellite.

[dnmeid]

1. I can't see how the limitation to a part of the grid would embrace non-grid surfaces
2. I thought the grid vs. non-grid discussion is done. We talked a lot about how non-grid surfaces can be transfered to Companion's grid and how this could be made visible to the user. see Companion beyond 3.0 #2387
3. Nobody seems to disagree that the setup tab is too complex for the problem it should solve and generally: please not another tab somewhere just because it has to go somewhere.

Don't get me wrong, I'm a big fan of every functionality, but I think this it the wrong UI approach to that problem.

[Julusian]

I can't see how the limitation to a part of the grid would embrace non-grid surfaces

For now it is showing just the portion of the grid, and it may remain doing so for some surface types. But stage 2 of this change is to add 'full' layout definitions so that when viewing as a surface you get a view like this:

This view will be able to position the buttons however it wants to, so when showing a non-grid surface the buttons can be aligned like the physical device.

I would argue that this simple mode still has value, both for consistency so that this works for every surface, even if we don't have a full drawing definition. And as a way to easily see what portion is used by a surface.
For that latter part, the other approach would be to as you have drawn, and a way to 'jump to surface X'. But when you have a bunch of surfaces potentially in overlapping, I do wonder how hard it will be to interpret.

I thought the grid vs. non-grid discussion is done.

Yes, we have agreed that surfaces will choose something for how they map to the grid, but I don't see a conclusion on how the user should know what that is other than through trial and error or interpreting some semi to scale, semi stretched representation of the surfaces.
This is an alternate approach to present that grid in a friendlier and easier to interpret way only when the user wants to see it like that.

Nobody seems to disagree that the setup tab is too complex for the problem it should solve and generally: please not another tab somewhere just because it has to go somewhere.

I almost did this as a popover thing like the zoom control, but thought it would be too long/complex to be reasonable to put there. I could try doing that if you would prefer?

[dnmeid]

Yes, we have agreed that surfaces will choose something for how they map to the grid, but I don't see a conclusion on how the user should know what that is other than through trial and error or interpreting some semi to scale, semi stretched representation of the surfaces.

One of the points of my proposal is, that controls don't need to be 1x1. Let's map your Xencelabs. I think it fits 2x7 quite good.

The top button/dial is covered by four cells, the bottom button is covered by two cells. When you place a control on one of those cells it will interact with the overlapping surface hardware element. The user can either place his Companion control on any of the touching cells without guesswork or enlarge it to 2x2 for the top dial or 2x1 for the bottom button. I think this is simple, straight forward, and quite reusable on different surfaces.

I almost did this as a popover thing like the zoom control, but thought it would be too long/complex to be reasonable to put there. I could try doing that if you would prefer?

If the problem that is to be solved is: "I want to see my surface and how it will look" or "I only have one streamdeck and I get lost if I see more cells than there are buttons, I think your stage 2 can be a solution. But I would take a different aproach. I see this as an surface emulator, something which renders like the real device and behaves like the real device. I can imagine to have surface emulators also inline in the GUI but not replacing the button grid but being an equivalent tab. The button grid by default selects buttons for editing and has a hot-press option. The emulators default to press and can have an select option. Maybe what is standard and what is option can also be a user preference for the emulator. Then this would be complete inline with the existing emulators and no need of additional configuration or additional modes.

[dnmeid]

And by the way I think placing controls is difficult if you have a few different surfaces with different offsets and that is the point where you need to see all of them and not one of them.

[Julusian]

I think this is simple, straight forward, and quite reusable on different surfaces.

I kind of agree but not fully.
That is the case (to users) until multiple of those buttons it overlaps has contents. And once that happens, it will be some guesswork as to which one will be used. (Maybe we execute the actions for all of them, but only one will win on feedback)

And for the author of the surface integration, they will need to define those overlaps.
Either each integration will need to handle the merging/selection for presses & drawing or companion itself needs to become aware of this to handle it for them.

I see this as an surface emulator, something which renders like the real device and behaves like the real device.

Unless rotaries and things work in that, it can't really be called an emulator. A viewer maybe, but for those with rotaries it would only be useful as a programming tool not for operation. (Unless someone has a plan on how rotaries should work in an emulator)
The old emulator has not looked like a streamdeck in quite a while, there were many requests to change that because of the wasted screen space, and very few if any to restore the old styling, which suggests that an operation tool like that isn't in demand. While there havent been many requests for something like this, I think this will be very useful when working with weirder surfaces in the future.

And by the way I think placing controls is difficult if you have a few different surfaces with different offsets and that is the point where you need to see all of them and not one of them.

I had thought yesterday that it would be nice to be able to select a surface-group in this view, rather than just individual surfaces, but didn't want to tackle that yet.
So I am definitely open to this being continued to view multiple at once. Whether that is a rigidly defined thing, or becomes a more freeform 'desk simulator' I don't know.

This is reading to myself like I am fighting the concept of a grid, but I'm not intending to. And its not changing any of the technical details behind things, just an alternate way of viewing the grid.

[dnmeid]

Unless rotaries and things work in that, it can't really be called an emulator. A viewer maybe, but for those with rotaries it would only be useful as a programming tool not for operation. (Unless someone has a plan on how rotaries should work in an emulator)

That shouldn't be a real problem. Rendering like it looks on the device, use mouse or touch rotation movement to rotate it. Extra bonus: use scroll wheel to rotate. Done.

The old emulator has not looked like a streamdeck in quite a while, there were many requests to change that because of the wasted screen space, and very few if any to restore the old styling, which suggests that an operation tool like that isn't in demand. While there havent been many requests for something like this, I think this will be very useful when working with weirder surfaces in the future.

The current emulator should not be called an emulator in that regard, I'd rather call it a webpanel.

So I am definitely open to this being continued to view multiple at once. Whether that is a rigidly defined thing, or becomes a more freeform 'desk simulator' I don't know.

What would be the difference then from my surface view? Only that you don't see unused cells or anything else? In my proposal the surface view actually has the main purpose of positioning the surfaces, but I'm open for a simulator view. Overlapping surfaces is a challenge for that, maybe one could select / deselect the surfaces or groups to show on simulator. I don't think we should use too much effort for this feature because Companion is about using equipment with real surfaces or virtual surfaces and the main GUI is for programming that stuff and not for running it. A simulator can help while programming but the main focus of the GUI should stay programming.

[bryce-seifert]

Do we have any telemetry on how many people actually use Companion with multiple surfaces / overlapping surface layouts?

Maybe I’m the odd one out, but I don’t utilize surfaces in that way and would consider it more of an advanced use case. The Surface mockup with them all overlayed is confusing to me, in terms of how I think about the grid and organize my Companion systems. but maybe seeing an actual implementation would make it make more sense.

And data could prove me dead wrong, but my suspicion is that lots of people use companion with just a single Streamdeck, or at least a single type of Streamdeck.

[dnmeid]

Overlapping surfaces have been the standard ever since. When we started this workflow it was a little awkward to many people and still is today. Often beginners ask why both of their surfaces show the same buttons or how they can address the second surface individually. The programming style with a side-by side layout is only made possible recently with the larger grid size (not counting multiple streamdeck minis).
My assumption is, that a side-by-side layout will get more and mode popular with the current and coming GUI changes and at some point will be the prevalent layout style.
But overlapping surfaces will never die. I see a lot SD mini or legacy with Satellite for some limited remote access e.g. from a lectern or director. Overlap is what you want to have there so you can use same programming on different surfaces.

In my surrounding (professional events industry), I'd say 50% of the people are using 1x XL, 30% more than one XL, 20% mixed surfaces. The reason is that we like to add streamdecks to the genuine consoles. In the streamer and more budget-oriented world, the combination of XL and SD+ is very popular because it gives some rotaries and good amount of buttons. They usually replace the consoles completely with Streamdecks.

Speaking about my Mockup: What does confuse you? Keep in mind that this is only a quick Mockup, done over a year ago and does not 100% fit the context here. Have you read #2387 ?

[bryce-seifert]

That's interesting anecdotal evidence of what devices / how people are using them, but I would be interested in the raw data of the userbase as a whole.

Speaking about my Mockup: What does confuse you? Keep in mind that this is only a quick Mockup, done over a year ago and does not 100% fit the context here. Have you read #2387 ?

I read thru this and it makes slightly more sense, but I still conceptually have a hard time with the windows/surfaces concept as laid out there. I think I could grow to learn it, but trying to be an advocate for the most basic of user who just needs a simple setup with one or two devices. I feel like being as simple to go from install to programed surface is still important.Maybe, as you in that discussion, "once you see this in action it will be very easy." But having not seen anything like this implemented or drafted, and as such have a hard time breaking the current layout/ pages mentality. I do think getting more voices in the discussion would also be helpful.

All that said, this PR is likely not the best place for this whole discussion, so I'll defer to that discussion thread if I have more input.

[Julusian]

Do we have any telemetry on how many people actually use Companion with multiple surfaces / overlapping surface layouts?

There is some, here is a sampling of how many unique surfaces each online user has reported across the last week This will only include users who are online, an unknown proportion are offline.

Fun fact, one user of 2.2.0 has 87 surfaces!
There is a potential for data skew if users have copied around the full config folder including the machid file, as we use that file as the uniquely id of the machine.
This data doesn't include emulators either.

---

Maybe I’m the odd one out, but I don’t utilize surfaces in that way and would consider it more of an advanced use case.

Overlapping surfaces have been the standard ever since. When we started this workflow it was a little awkward to many people and still is today.

I think we are talking about two related but not quite the same thing.
I suspect that very few people have surfaces overlapping showing the same page. But many will have them overlap but on different pages.

So with what Dorian proposes in #2387, if we show every surface on every page, then yes it will be a confusing mess of overlaps.

Even though I implemented the larger grid, I haven't reconfigured my office setup to put everything on page 1 simply because I can't be bothered to reconfigure things. But if I were to add a new surface, then I would do it the new way.

---

But getting back to the question at hand, does this proposed view serve a good purpose or is it going down the wrong road?
Is the consensus that this should be done as a popover like the zoom, or is keeping it as this tab acceptable?

I still think it has value as a way of viewing the portion of the the grid being configured with a specific streamdeck in mind, while #2387 proposes something that will help with placing multiple within the grid. I don't think it has to be one or the other.
And maybe later on it could be turned into a new 'emulator' as well as this view or instead of.

Based on the stats above, I think the side by side view is probably not that useful, so while a nice idea probably won't be done.

And honestly, I am not how long it will be until the grid changes from #2387 get done. I so rarely find myself doing something complex enough to benefit from windows, that the times I do I would be satisfied with a better way to link/ref a button to another.
So I do think we should try to avoid doing things that won't work in the world proposed by #2387, but I don't think we should avoid doing things just because that proposes different/better/bigger-picture solutions.