Standalone configuration-software to mimic other interfaces? aka "The big USB protocol discussion" ;)

[kripton]

Having a great interface (hardware wise) is the one thing. However, it needs to be supported by some DMX-data-generating software in order to be useful. With the current plan to use the Ja-Rule USB protocol, the device will only be usable "out of the box" using the OLA software. I'm not aware of any other software supporting this protocol. Of course, when using a larger number of universes, one might argue that having the dongle controlled by OLA and using some kind of network protocol (ArtNet or most probably sACN) from the data-generating-software to OLA is the best approach. I agree :)

The control software meant to switch the device to different "emulations" would of course need to detect the device in any case, no matter which VID:PID it's currently emulating. However, that might be pretty reliable when a pattern can be matched on the USB strings (Manufacturer name, product name and serial number)

Initial post split up per peternewman's suggestion: There is now one comment for each USB protocol I could think of. Feel free to add more ;)

[kripton]

Another way to tackle things:
Using tinyUSB, one could theoretically emulate a network device (RNDIS). Then, the host machine would see a network interface that gets its IP address (host side) from an emulated DHCP-server that seems to be running in the dongle. If that is all done, one could use any network DMX protocol from the host to the rp2040-dongle.

[kripton]

The code is already available: https://github.com/hathach/tinyusb/tree/master/examples/device/net_lwip_webserver
An integerated web server .... sounds like a plan :) Needs some investigations regarding protocol overhead and CPU time consumption

[kripton]

I'm not sure if any "network DMX protocol" supports RDM?

[peternewman]

I assume it would be link local ( https://en.wikipedia.org/wiki/Link-local_address#IPv4 )?

The big con of this is probably explaining to people why they have to have a PC in the way when this dongle seemingly has network support, plus the overheads of the protocols. Also that means if you want to output DMX and the incoming source is E1.31, you're potentially feeding it through OLA to just send it straight back out again as the same protocol.

It would also need some OLA work as currently each protocol only has one node, not one per machine network port.

Also OLA doesn't currently let you patch most network protocols between ports, as the port is set based on the OLA port. So you couldn't do E1.31 to E1.31 via OLA to the dongle.

Essentially just NATing (if that works for Multicast) and not using OLA would be "easier" for supported protocol.

I'm not sure if any "network DMX protocol" supports RDM?

Art-Net and E1.33 both do.

[kripton]

Regarding Link-Local addresses:

• While this seems reasonable at first, I don't think it's the way to go (at least with IPv4). According to the Wikipedia entry, link-local addresses can only be given by a host itself, not by a peer device. So while the dongle could have its own address right from the beginning, the host (= PC) would most probably try to reach a DHCP server and only after a timeout would assign itself a link-local IPv4. I would like to avoid that timeout by having the dongle reply to the DHCP-requests and assign the host one. That however raises the question WHICH IP should be used by the dongle(s) and which by the host. We could either derive it from the board's unique ID and hope to not collide, we could make it random and hope to not collide, ....
  With the IPv6 link-local addresses, this seems to be well feasible. Each interface has a link-local address anyways. lwIP (that's the one the tinyUSB-lib supports) can do IPv6. I just don't yet have that much experience with it. And it might be that some systems just don't have IPv6 enabled.

Regarding if it makes sense:

• You are right that making the dongle able to process ArtNet or E1.31 contradicts the need to use it with OLA a little. Of course, in most cases one could then use the dongle without needing to run OLA. However, OLA is still very handy since it could aggregate multiple interfaces, handle all the RDM logic, patch multiple inputs to a single universe (LTP/HTP-merge) and so. And if the possibility to use the interface without OLA is not promoted but rather left to "experts", why not?
• Yep, true, I hadn't thought about that OLA doesn't have multiple nodes per machine network port. That would need some more thinking about.
• I will do some more experimentation about the lwIP stack, IPv6 addressing and so. Even if it's just to have an integrated web server for dongle configuration and status, that might be useful by itself.

Oh great, I somewhere read something about the DMX network protocol being able to do ArtNet but it's good to have your confirmation about it :)

[peternewman]

That however raises the question WHICH IP should be used by the dongle(s) and which by the host. We could either derive it from the board's unique ID and hope to not collide, we could make it random and hope to not collide, ....

I think I didn't make my point very well. It would need to pick one of https://en.wikipedia.org/wiki/Private_network but if it picks say 192.168.1.1 and .1.2 then everything goes wrong (or at least gets a lot more complicated) if your PC has a wireless or wired connection in the same subnet. Even more so I guess if you've got multiple dongles on the same machine. I'd assume the link local stuff might be better in that regard as it's probably expected to be on a point to point, or at least standalone, network.

With the IPv6 link-local addresses, this seems to be well feasible. Each interface has a link-local address anyways. lwIP (that's the one the tinyUSB-lib supports) can do IPv6. I just don't yet have that much experience with it. And it might be that some systems just don't have IPv6 enabled.

From my very limited experience with IPv6, browsing to IPs seemed more painful in some software. Also I don't think there's Art-Net over IPv6. I think it might be possible for E1.31.

• You are right that making the dongle able to process ArtNet or E1.31 contradicts the need to use it with OLA a little. Of course, in most cases one could then use the dongle without needing to run OLA. However, OLA is still very handy since it could aggregate multiple interfaces, handle all the RDM logic, patch multiple inputs to a single universe (LTP/HTP-merge) and so. And if the possibility to use the interface without OLA is not promoted but rather left to "experts", why not?

I think most of the network protocols support merging in the node too. I think using the interface without OLA may be a challenge as you'd have to route and port forward stuff potentially.

• Yep, true, I hadn't thought about that OLA doesn't have multiple nodes per machine network port. That would need some more thinking about.

Shouldn't be too hard and would benefit everyone. The not being able to cross-patch universe numbers may be more fiddly to fix and have more impact.

• I will do some more experimentation about the lwIP stack, IPv6 addressing and so. Even if it's just to have an integrated web server for dongle configuration and status, that might be useful by itself.

Yeah I guess it would be a nicer status display if you've got the memory for some webpages.

Oh great, I somewhere read something about the DMX network protocol being able to do ArtNet but it's good to have your confirmation about it :)

You can see it now in action with two OLA instances, or OLA and DMX Workshop!

[kripton]

And another idea which is even more uncommon: A USB DMX interface with "Mass storage" as logical interface?
The Pico board can present itself as a "USB Flash Disk" to the OS, FAT formatted so it works on Linux, macOS and Windows. Fhe filesystem label could be the seriel number so multiple devices can be distinguished. There could be a config file in the root folder, maybe in INI format?

Each DMX port could be a folder with one file where DMX data is being written to (using a simple "file write" operarion on the host side) or read from.

RDM devices found via discovery could also be files or folders in a "RDM" folder, additional commands thr dongle could execute (such as entering the bootloader) could also be subfolders in a "CMD" folder. writing a file to that cmd subfolder executes the action? A bit like fhe sysfs interface on Linux ;)

That would probably eliminate a lot of permission problems, since most host OS can be convinced to let users mount, read and write "USB sticks".

The only problem could be that files written to a disk are not immediatelly writen but cached. This could introduce latency. Workarounds: "sync" mount option on Linux, flushing the file after ever, write?

Reference code: https://github.com/hathach/tinyusb/blob/master/examples/device/msc_dual_lun/src/msc_disk_dual.c

[peternewman]

Wouldn't it be hard to get performant read results, unless you're continually polling the file, especially with things like RDM commands.

[kripton]

With that you had me convinced to drop the idea for about two days. Today I found the solution that I already had in mind for RDM commands, but not yet for DMX IN. Here goes it goes:

• Think of it more like a "blocking read". We all know that not all storage behind a file system is fast. Think of CDs, DVDs, Floppies or even slow HDDs. So let's say you want to do a RDM discovery. To execute that, you don't write a file in the virtual filesystem. Instead, you read it. The initial read command triggers the command on the DMX line but the result is not returned to the host until the command is "done". That means the file read will block / stall for as long as the RDM discovery is running. As soon as that has finished (I don't know the typical timeouts for discoveries but I assume it's less than 2 secods), the "file read result" can be returned to the host.
• With DMX IN it's similar: As soon as a universe/port is patched as input, OLA can open the file and try to read it. This operation will block until new DMX data has arrived on the port. When this happens, the dongle completes the "read request" and the data can be processed by OLA. Immediately afterwards, OLA will again try to read the file but it will only actually get data from that file when a new DMX frame has arrived. This can lead to as many blocked reads as there are ports patched but since this is threaded on the host, I don't see much of a problem there
• With DMX OUTs, OLA will open the file, write to it and close it every time there is new DMX data to be sent. What I didn't think about while having the initial idea is that the host decides where on the virtual hard disk the data will be stored. It won't necessarily be the same offset every time. So that needs some tracking on which offset which file is written to by the host in order to actually work. That sounds like some effort but it's not impossible.

While this is not easy to do, has some overhead and we don't yet know if the performance is sufficient, I will keep on experimenting with it since the user can easily see all attached dongles without any additional software (since they will appear as removable disks with the serial number in the disk name) and dongle configuration and status can be done via "virtual files" as well.

[peternewman]

• That means the file read will block / stall for as long as the RDM discovery is running. As soon as that has finished (I don't know the typical timeouts for discoveries but I assume it's less than 2 secods), the "file read result" can be returned to the host.

A full RDM discovery would be longer I think, but I guess it would normally only be doing a single one and OLA doing all the hard work. It might still be just outside this, I'd need to double check the spec.

The rest of it certainly makes sense.

[kripton]

I could also think of situations where a direct connection without using OLA makes sense (dongles with lower universe count, inexperienced light operators, ...). Therefore it might make sense to have a configuration software (or DIP-switch on the device itself?) to configure the device to mimic other interfaces that are already supported by a range of software out there. The emulation of an existing dongle could either be permanent (stored in flash) or temporary (until the next USB bus reset).

[peternewman]

temporary (until the next USB bus reset).

That might want some thought, troubleshooting is going to be very confusing if it changes mode each time you unplug it!

[kripton]

Fully agree, that's not desirable. Instead, the dongle should have one "always on protocol" (such as the virtual network card so the web server can be accessed OR the virtual disk / file system) and remember which emulation / protocol mode it is in. And it should use that protocol / mode until it is told to do otherwise using the "always on protocol" or some hardware reset button.

[kripton]

Emulating the OpenDMX protocol / dongles: Serial-based, usually host (= PC)-based DMX512 timing, usually cheaper

• Could be done, probably only supports one universe, or we would need to have multiple CDC ACM USB interfaces. I'd say the practical limit would be 4 universes
• Pros:
  • Supported by a wide range of software
• Cons:
  • It's hard to reliably detect such a dongle since the common VID:PID pairs of USB-to-Serial adapters are being used
  • Only DMX output, no RDM, no DMX input
  • Since it's a virtual COM port and all data sent to it directly goes out the DMX line, only one universe can be supported

[peternewman]

• Only DMX output, no RDM, no DMX input

There is experimental RDM support on a PR for OLA.

I think there is Windows DMX input software for an Open.

[kripton]

Interesting, I was not aware.
Another negative point that I've just added above is that only one universe can be supported. As far as I know ;)

[peternewman]

Interesting, I was not aware.

From what I understand, finding the break is hardest, which isn't even needed for RDM controller, so adding transmit is possible.

Another negative point that I've just added above is that only one universe can be supported. As far as I know ;)

But you can buy dongles with multiple COM ports on, which people have extended our code to work with, so if your code could emulate multiple ports it could support multiple universes.

[kripton]

Emulating the Anyma uDMX and clones

• Technically possible
• Uses the USB control transfers, which seems a bit non-standard. As such, can only be operated using libusb (which is actually Pro and Con at the same time ;)
• Pros:
  • Also quite widely supported
  • Easy protocol
• Cons:
  • Since it uses control transfers and there is exactly ONE instance of the CONTROL endpoints, only one universe is possible without changing the current protocol in some way
  • Only DMX output, no RDM, no DMX input (without protocol modifications)
  • No 100%-reliable way to detect device via a VID:PID-pair. There are some commonly known pairs but technically, they are all "shared" and not guaranteed to not be used by other devices

[kripton]

Emulating the Digital Enlightenment / DMXControl Projects e.V. / Frank Sievertsen FX5 dongle

• Enumerates as a USB HID device. Technically it's not a HID-device in the classical sense but the choice is actually quite clever
• Pros:
  • Driver can make use of the OS' HID-driver (done so by QLC+ on Windows and Linux) or libusb (done so by OLA).
  • Protocol is very easy to re-implement and to extend to multiple universes. It's quite efficient as well.
  • There's a known, fixed set of VID:PID pairs to reliably detect such devices
• Cons:
  • Supports only one universe out-of-the-box (= without extending the drivers in the data-generating software). However, it might be possible to emulate multiple HID interfaces on one USB connection. Not yet tried, might be very interesting
  • Only DMX output, no RDM, no DMX input (without protocol modifications)

[peternewman]

• There's a known, fixed set of VID:PID pairs to reliably detect such devices

Isn't this a con in our case, as we either need to get another VID:PID pair added to other software, or be naughty and spoof their pair?

• Only DMX output, no RDM, no DMX input (without protocol modifications)

Nodle appears to have a PC In mode (from DMX):
http://docs.openlighting.org/ola/conf/ola-usbdmx.conf.html

[kripton]

True, either we'd need a separate pair and adapt all the software or we spoof, as the dongle's code currently in the main branch does. Detecting if it's an "original" or a "rp2040"-based device would then be done by the host software based on Serial number or Model name, just as I did in the OLA-branch for the dongle. Unaware software would then support the dongle in single-universe-mode, software aware of the additional commands could support multiple universes.

I don't know if spoofing another dongle would be naughty. I cannot find right now where I noted it down, but only one of the three VID:PID-pairs I know is officially registered. The other two are either "shared" or simply not registered at all. I would need to look that up again.

Yes, I'm aware of the different modes per dongle OR per port. But does OLA actually support DMX IN with a Nodle U1 device? I can't test since I don't have a device here but as far as I remember from the source code, DMX IN was not implemented. Or I skipped reading that part since I wanted DMX OUT to work first ;)

[kripton]

Okay, just found my notes:

QLC+ supports all 3 of them, on Windows and Linux

Digital Enlightenment USB DMX: 0x04B4 : 0x0F1F => Cypress Semiconductor Corp : UnknownDevice

Frank Sievertsen FX5         : 0x16C0 : 0x088B => Van Ooijen Technische Informatica : UnknownDevice

Nodle U1                     : 0x16D0 : 0x0830 => MCS : DMXControl Projects e.V., Nodle U1            << Only one supported by OLA

[kripton]

Implementing ENTTEC's USB PRO protocol

• Pretty complex but advanced serial-based protocol
• Most drivers are using the libftdi-library and as such work only with devices actually using a chip speaking the FTDI-protocol. Since I don't want to reverse-engineer that (hard to do and maybe legal risks), I would currently not assume that enabling support for that protocol is feasible. Nevertheless:
  • Pros:
    • Protocol is basically supported by any data-generating software out there
    • RDM and DMX input is already part of the protocol
  • Cons:
    • As I said, most probably an FTDI-chip would need to be emulated. Lots of work, legal risk
    • Number of ports supported by the protocol is probably limited (maximum 4 I would assume)

[peternewman]

* As I said, most probably an FTDI-chip would need to be emulated. Lots of work, legal risk

I assume you're not interested in putting a physical FTDI onboard/defeats the purpose?

* Number of ports supported by the protocol is probably limited (maximum 4 I would assume)

Enttec do dual port, with some complicated key stuff, unique per piece of software. DMXKing do dual port too (I don't think they do more).

[kripton]

Indeed, I would not like to use a real FTDI-chip since it defeats the purpose. One could also say that it would limit us in what we can do on USB but I would agree that compatibility weighs more.

I am aware of the complicated two-port mode of the PROs and that's exactly what I don't want. I don't see any use in artificially limits or similar here.

[HakanL]

FWIW the Enttec Pro protocol does not require FTDI, it can be used on any usb serial port. The new DMXking MAX devices uses a regular CDC-type USB serial port identifier and are working with several lighting software to make sure they don't filter out those devices. So that may give extra support for this protocol in various lighting software (like LightKey, QLC, QLab, etc). And yes it's limited to 4 universes with some DMXking-specific extensions.

[kripton]

Using Ja-Rule / Open Lighting Embedded protocol (the one I'm currently targeting)

• Uses BULK transfers on VENDOR-specific USB interfaces. As far as I know, only possible via libusb on the host side
• Pros:
  • Protocol is very well-designed and extensible.
  • Protocol is documented and firmware is open-source
  • One well-known VID:PID-pair to detect the devices
  • RDM controller and RDM responder already part of the control protocol
• Cons:
  • Fixed format for the serial number. World-Wide uniqueness of the serial numbers is not guaranteed (RP2040-based devices and Number1-devices both using the OLA vendor id might actually collide in the "unique device id" part of the serial number
  • Not support by software other than OLA
  • Imposes a limit of 15 ports per USB device (due to one pair of endpoints for each port, 16 pairs of endpoints possible but one pair RESERVED for CONTROL transfers). However, number or ports could be increased by introducing additional commands to the protocol

[peternewman]

• RDM controller and RDM responder already part of the control protocol

As per #1 (reply in thread) RDM responders are probably a non-issue unless we want to go really niche.

• Fixed format for the serial number. World-Wide uniqueness of the serial numbers is not guaranteed (RP2040-based devices and Number1-devices both using the OLA vendor id might actually collide in the "unique device id" part of the serial number

I think we've covered the Number 1 side of the clash in #7 but yes there is a (hopefully small) risk of clashes within RP2040 devices.

• Not support by software other than OLA

Although a number of bits of software support OLA as an output module, for the others you can go software->E1.31/Art-Net->OLA->Dongle.

• Imposes a limit of 15 ports per USB device

I find it rather crazy this is a con 😆 most of the commercial network attached hardware tops out at 8 ports!

[kripton]

• RDM controller and RDM responder already part of the control protocol

As per #1 (reply in thread) RDM responders are probably a non-issue unless we want to go really niche.

True, but it doesn't hurt if the protocol already supports it if we want to go that direction later.

• Not support by software other than OLA

Although a number of bits of software support OLA as an output module, for the others you can go software->E1.31/Art-Net->OLA->Dongle.

Yes but that would mean that the dongle only works with OLA. Of course I would prefer everyone to use OLA but I could also understand that some people are happy if the dongle works out-of-the-box with their light control software for simplicity reasons. That's why I listed this as a con here

• Imposes a limit of 15 ports per USB device

I find it rather crazy this is a con laughing most of the commercial network attached hardware tops out at 8 ports!

Yeah, think big :D. True, if the dongle should support multi-mode-ports (DMX IN and/or RDM), then the RP2040 will be limited by number of PIO SMs and number of GPIO PINs. However, I still like the idea of being able to drive 16 universes OUT-only from ONE USB port ;)

[peternewman]

Use a proper/full Ethernet stack 😄 :
https://www.raspberrypi.org/blog/how-to-add-ethernet-to-raspberry-pi-pico/

[kripton]

Indeed, not a bad idea. Since the hardware design (at least what I would propose) is already quite advanced (see #1 (reply in thread)) and pretty much all GPIO pins are in use (at least on the 4-slots-baseboard), this one currently won't fit in. But with its 9 IO pins, it might just make it for the 2-slots-baseboard ;). I'll assume I will add an expansion connector for the LAN8720 breakout board. Then the device could also be used as a standalone ArtNet/sACN -> DMX interface. Or vice versa. Or ArtNet/sACN -> Wireless DMX ;)

[kripton]

But I think I would rather go with one of those instead: https://eckstein-shop.de/ENC28J60-Ethernet-LAN-Netzwerk-Modul
Less pins, the baseboard is already using SPI for the nrf24L01 module and in contrast to the RMII-module, the Pico doesn't need to be clocked down to 50MHz