What can we realistically do while waiting for a reply from a node?

[AlCalzone]

@hanskroner
Let's take a look at improving network throughput. The basic communication (Send Data) with a node is shown in the following badly-drawn diagram:

The blue blocks are fixed because we need to wait for the REQ->, <-RES, <-REQ cycle to be completed before we send the next. I've learned this the hard way.
I've recently increased the default wait time for a node response from 1500ms to 10000ms to give slow nodes a chance, so the idle time (red) while communicating with non-responsive nodes is quite large.

Do we have to do sequential communication like this

or is it realistically possible to juggle multiple pending transactions at the same time

without creating a situations where node responses cannot reach us because we're saturating the network with requests?

[hanskroner]

The blue blocks are fixed because we need to wait for the REQ->, <-RES, <-REQ cycle to be completed before we send the next. I've learned this the hard way.

While I'm sorry to hear that, it sounds like lesson learned - that ACK->RES->REQ cycle is sacred.

I've recently increased the default wait time for a node response from 1500ms to 10000ms to give slow nodes a chance, so the idle time (red) while communicating with non-responsive nodes is quite large.

You can get a little more "scientific" than this, though it might be something better labeled as an improvement or optimization to come later. Since you know the length of the frame you're sending, you can use the timing in the ITU G.9959 spec. and assuming the worst case: 9.6kbps transmission, 4 routing attempts, 1 direct range attempt, and a best-effort guess for the Explorer Frame given the network size (if you're requesting Explorer Frames in the outgoing transmission attempts). That'll get you the upper limit of what a timeout should be - if you're interested in letting the protocol do all it can do to attempt to deliver your message.

The Z-Wave protocol is limited by what it can do running on a very constrained embedded system, but the serial host has the possibility of doing more. You can maintain a history of transmission times and the routes used to reach nodes. After a bit of data collection, you'll have a decent picture of what "typical" communication with a node looks like, and you'll be able to make an educated decision about decreasing the timeout wait.

If you think you need to compromise delivery for speed, you can start by reducing the protocol features - for example, don't try explorer frames. For extreme cases, you have the option of not requesting an ACK if you really don't care about delivery guarantees. You can then move on to timing out early (remember to abort the transmission). This will depend heavily on what the clients using the library tell you they prefer: upset users because transmissions take too long to timeout, or upset users because anything but quick communication results in transmission failures.

Do we have to do sequential communication like this

or is it realistically possible to juggle multiple pending transactions at the same time

There's unfortunately no straight-forward answer to this - the optimal solution depends on the topology and traffic of the network in question. If you're dealing with a large network, where there's a lot of routing, and there's a controller that's overzealously polling every possible feature of all device's Command Classes constantly, scenario # 1 being the most conservative approach might provide a better experience. For a network comprised mostly or exclusively of Z-Wave nodes that report their information back to a central controller via the Z-Wave Plus Lifeline or other associations, and where there's little need for frames to hop between repeaters, scenario # 2 is likely to provide the best experience.

Z-Wave as a protocol is completely stateless - there's no sessions or frame ids, and this is done on purpose. The fact that a GET results in a REPORT is a "happy coincidence" from the perspective of a controller - whether a REPORT is received as the expected reply to a GET, or unexpectedly and unsolicited, it must be processed the same way. zwave-js should already be setup to support scenario # 2, as it's entirely possible to receive an unsolicited frame from a different node while it's "waiting" (or rather, "expecting") for a reply to a GET - and that frame must get processed.

without creating a situations where node responses cannot reach us because we're saturating the network with requests?

It's way too easy to saturate a network, so you'll need to be judicious and test a lot when implementing this. Of course, if you send out 20 GETs, one after the other, requesting state from 20 different nodes you'll saturate the network - likely for long enough that one or more of the nodes will be unable to send out a reply and give up. It's very possible for zwave-js to build up a view of the network topology by asking the Z-Wave protocol for neighbor information on each node. That, coupled with the data I mentioned above will paint a picture of how much "traffic" is coming and going from the zwave-js controller - which in the majority of setups will be the brunt of it. You can use this information when selecting your transmission strategy, and adapting it when needed.