Which WIFI router?

[jmvb-cl]

Hello
My setup is now 5 tally light

I have an ATEM 1 M/E production studio 4K and an ATEM 1 M/E broadcast panel. I don't use a computer for control
My first router was a TP-LINK TL-WR740N
Since I had range and drop issues, I thought changing it for powerfull one, the problems will gone.
I get a TP-LINK TL-WR940N, but the issues remain, every 30 or 40 camera change, got frozen the control panel for a few seconds. Sometimes all tally's lost connection, sometimes not.

Doing some research, I found that the control protocol uses UDP and is fairly intolerant to packet loss and network latency and as Strata pro app developers suggestion, now I'm trying an Apple Airport express.
In home test, at least the 40 changes freeze problem gone, but I concerned about the router range, since it lacks of external antenas. I hope do some in field test soon.

Which router are you using? do you have drops problem?

A guy from youtube recommend a D Link Dir 615, someone have use it?

[AronHetLam]

For our setup we use Ubiquiti gear, and usually it's stable from what I know. We don't have freezes. However, the Tallys are out of sight for the operator, as they are mounted on remote controlled PTZ cameras. So only the talent can see them, and usually they are more focused on the audience physically present, rather than the cameras.

You say your ATEM Freezes? In Skaarhojs ATEM libraries for Arduino there is a comment about the ATEMs freezing if they don't get an ack(nowledge) response before a certain amount of new packages are sent.

ATEM_tally_light_with_ESP8266/libraries/ATEMbase/ATEMbase.cpp

Line 205 in 2d6ff1e

} else if(_initPayloadSent && (headerBitmask & ATEM_headerCmd_RequestNextAfter) && _hasInitialized) { // ATEM is requesting a previously sent package which must have dropped out of the order. We return an empty one so the ATEM doesnt' crash (which some models will, if it doesn't get an answer before another 63 commands gets sent from the controller.)

This is not something I've run into, but it might indicate that you have a lot of dropping packages - especially from the ESPs to the ATEM. By nature the UDP protocol doesn't have assurance that your packages will arrive, and therefore the network gear is allowed to drop them. The TCP protocol does include assurance, but ATEM uses UDP, so it's ideal to have separate network for the tally system. Atem does build an assurance system into their custom protocol, but as it's built on top of UDP it is built on the assumption that some of the UDP packages arrive, It'll resend packages if it doesn't get an ack, but if all the ack's are dropped it's though luck, and the ATEM won't get the ack's it expects. And according to that comment I mentioned that might make the ATEM freeze.
The ESPs are not that powerfull, and if there is a lot of broadcast traffic on the network that might cause the ESP to drop packages as well, as it has to keep up with all of it, even though it doesn't need it all - It has to read a packages before it can determine if it is needed or not. I'm not a network expert, but this is some of what I've learned while working on this project.
There is an option in the Arduino IDE to make the ESP allocate more resources WiFi bandwith. That might help a little bit, but I don't think it'll solve the problem.

[jmvb-cl]

Thank Aron, you are very kind as usual.
Like I said, with both TP-LINK routers, after 30 or 40 camera changes, the Atem panel freeze for a second or two, and return to the main page if you are in other adjustments pages.
Now, with Apple one, the issue is gone, only the tally's reboot occasionally, even at close router range.

My network connection go from Atem to panel directly, as panel have 2 rj45 ports, the second go to the router.

How can I separate the tally system network?

[AronHetLam]

With "adjustment pages" you mean in the "ATEM setup" software? The only guess I have on the freezing is as described above.

Do you know if it actually reboots, or just looses WiFi connection and reconnects. If it actually reboots it would because of a chrash, which would need investigation. If you don't have the latest version of the software, make sure to get it installed.

Making a seperate network really depends on you network gear and setup. Consumer APs/routers typically let you configure 1 or 2 networks, and maybe a guest network, whereas eg the Ubiquity gear we use let you configure multiple networks within its configuration UI. There we have created a network just for AV and PA gear alone. Ideally you would have a network just for the ATEM and the tallys that doesn't even connect to the internet, but that of cause won't work if you use the ATEM to stream from.
The reason why is because you don't want unnecessary traffic between your devices. Eg if you have something like a Chromecast it continuously sends UDP broadcast packages to every device on the network, signaling that it's there. Essentially that's how devices are able to know about Chromecasts on a network. If you have a lot that kind of traffic I think it would quickly overload the ESP's ability to handle the data, as the ATEM also sends a lot of data to the ESPs.

But besides that I'm most likely not able to assist you any further, as I'm pretty confident that I've done what I can in the software, and the networking part really depends on your setup.

If it is because the ESPs doesn't have cpu power enough you could try with ESP 32 modules instead. They should have quite a bit more cpu power from what I read. I can verify/compile the code for them, but I don't have modules myself to test with, and don't plan on getting them anytime soon. So it's something you'd have to experiment with yourself. 😊

[jmvb-cl]

With "adjustment pages" I mean, meanwhile I'm in other pages different as main page in the broadcast panel.
I'll do a video to show it.
I don't use any computer in my setup.
The network is only for the tally's

[AronHetLam]

The only ATEM hardware I've worked with is an ATEM mini pro

[jmvb-cl]

I did 2 outdoor jobs, with apple modem, got mixed results.
Only 2 tally's setup
First day, no drops, only a few tally's reconnections.
Second day, mistakenly I set other wifi router (used for streaming) in the same wifi channel than apple modem (At the end of the day I realized of that.), and I had a lot of tally's connect - reconnect and respond delay issues.

[jmvb-cl]

Well, did 4 other jobs in a busy street in downtown, and was a disaster.
The tallys lost the signal after a few meter of both routers

I borrow a Mercusys AC12, 5 x 5dbi antennas and obtain better results.
Fixed cameras never lost the signal, and the mobile one, works ok 90% of time.

[jmvb-cl]

well Mercusys have issues too.
Last 2 job, in a indoor Basketball Court, first day, drops every 5 or 10 minutes.
Second day, I put the router in a mic stand in height, far from Atem, and perform better, only a couple of drop.

Just bought a Ubiquiti unifi UAP-LR and despite I only did home test, seems like I have a jackpot
No drops or freeze for at least 2 hour test

[jmvb-cl]

Ubiquiti unifi UAP-LR is the best so far.
I did several jobs and only a few drops per event.
The Atem freezes (if they occur) only last a blink (instead at least a second with the previous routers)

[AronHetLam]

Recently I tried updating our ATEM mini pro to the latest version, which resulted in it freezing sometimes in the middle of a show. We feed the video to OBS over usb, and assumed it was an issue with the usb connection in the latest firmware. We downgraded the ATEM again, and haven't had issues since.

It just struck me that it might be that the latest ATEM firmware is more sensitive to data loss/slow response or something like that, and that it was actually my tally lights causing the freezing.
I don't have time to look more into if that is actually what was causing the freezes for us, but I can look up what ATEM version I downgraded to if want to try it out.

[jmvb-cl]

Interesting...
My atem control software is 8.6.4... which one is yours?

[AronHetLam]

Our ATEM mini pro is running 8.2.1

[bfbones]

Be careful while downgrading those ATEMs, already bricked two of them trying to downgrade. Also no chance using recovery then.

I also experience drops since some weeks, I also upgraded my ATEM Extreme some time ago - maybe this is really a firmware problem?
I am using 4 Tallys.. maybe there are already too much connections?

[jmvb-cl]

Be careful while downgrading those ATEMs, already bricked two of them trying to downgrade. Also no chance using recovery then.

I also experience drops since some weeks, I also upgraded my ATEM Extreme some time ago - maybe this is really a firmware problem? I am using 4 Tallys.. maybe there are already too much connections?

Before upgrade, had you drop issues?

[AronHetLam]

Be careful while downgrading those ATEMs, already bricked two of them trying to downgrade. Also no chance using recovery then.

I also experience drops since some weeks, I also upgraded my ATEM Extreme some time ago - maybe this is really a firmware problem? I am using 4 Tallys.. maybe there are already too much connections?

Sounds like no fun! 🙁

Might sound like we're running into issues with the ATEM libraries by Skaarhøj not being updated as the ATEMs firmware is. If that's the case I sadly don't have the time or resources to keep it up to date.

[jmvb-cl]

Well, for me, the system which I build with 5 tallys and the Ubiquiti AP work very fine.
I can live with a couple of drops per night

Good work!

[jmvb-cl]

Any of you guys know if an "Isolate" network for the tally's would prevent the Atem connection drops?

[bfbones]

I downgraded my ATEM Mini Extreme and still get connections issues using a complete isolated wifi network, a dedicated Unifi Mesh-AP (because they are outdoor capable) and I don't know how to fix this.
Interestingly in my home test environment everything works without any issue for hours. But the moment I am at a production site everything begins to flicker, some tallys just disconnect, some of them are just running behind the current state and so on.
I think I have a general issue with my network or with the ATEM. When I try to connect to the ATEM using Metaswitcher on my Android phone I see all Tallys disconnect all at once. I have 4 Tallys in use.
@AronHetLam Do you think I should give it a try using the Tallyserver, or do you have another Idea?
Do you think the D1 Mini Pro with a external Wifi Antenna could help?

[AronHetLam]

What exactly do you mean by tally server? If you are referring to the second Arduino file, then that's only acting as a fake switcher for when I'm testing things at home. The server functionality is built in by default into the tally light, so they all act as a server as well as a tally. That way you can connect them to each other, instead of all to the switcher. I'm not sure if that helps though with your WiFi. Also I'm not able to do extensive testing, other than the small two cam setup I have inside, so I can't really say much about range. I've made the code able to run on an ESP32 as well, which should be able quite more powerful than the ESP8266, but again, I can't do extensive testing. But that said, with it working on the ESP32, it should be quite simple to use something like an m5stack ATOM https://shop.m5stack.com/products/atom-matrix-esp32-development-kit

[bfbones]

Just a small update now after two productions using the tally lights: after some system changes they're working perfectly.
I now use a ESP32 build in into my rack acting as a server for those 4 tallys. As mentioned before I have a Unifi Mesh-AP which is PoE powered and I can just throw that using 50m LAN near the main event area to have perfect coverage. My cam ops had no connection drops or delays on the tallys anymore.
This brings me to that the ATEM itself (Mini Extrem Pro) has some strange problems on it's firmware unable to serv 4 connections for the tallys. That single connections works.

[arkestlerdev]

@bfbones could you elaborate on how you did the “server” build please?