IS-12 & IS-14 Test Suites

[jonathan-r-thorpe]

• Initial commit
• Added IS-12 spec to Config
• Updated specs for IS-12-01 test suite

[jonathan-r-thorpe]

Carrying over the unresolved discussion around whether URL path of IS-12 specs should follow what other specs have done #800 (comment)

and comment about where "selector" and newly introduced "websocket" parameters are best located #800 (comment)

[cristian-recoseanu]

I just wanted to say that from a "product owner" point of view I have been using the test suite regularly and I am confident that it satisfies the testing plan the activity group set out. I want to congratulate and thank @jonathan-r-thorpe for all the hard work that went into this.

The two node implementations we have access to are passing the relevant tests succesfully.

[alabou]

The current IS-12 testing seems to assume that notifications received from a websocket are synchronous with the sending of commands which to my opinion is erroneous. Both paths to my understanding are totally asynchronous, requiring the use of the handle to synchronize responses with commands, possibly receiving notifications before and/or after command responses.

The notification cannot be retrieved only from def send_command(self, test, command_json) and this function must not assume any ordering between command responses and notifications.

For example if messages = self.ncp_websocket.get_messages() gets a notification before the command response, the send_command function will fail before it thinks there is no result.

[jonathan-r-thorpe]

Yes, you are correct that is making assumptions about the notifications being synchronous and that the handle should be used to associate a command with its command response.

I will modify to filter the messages on the command handle which should mitigate these problems.

[jonathan-r-thorpe]

For example if messages = self.ncp_websocket.get_messages() gets a notification before the command response, the send_command function will fail before it thinks there is no result.

Yes, there is a risk of this. This code should really loop until the corresponding response has been received, or a timeout is reached.

[alabou]

Here is what I used in the interim ... The expecting_notifications flag is set when we start using notifications (not a perfect solution). It works only if no more than one notification is expected after sending a command. It works with the current test suite but could fail with more elaborate use of notifications.

` def send_command(self, test, command_json):
"""Send command to Node under test. Returns [command response]. Raises NMOSTestException on error"""
# IS-12 Check message type, check handle numeric identifier
# https://specs.amwa.tv/is-12/branches/v1.0/docs/Protocol_messaging.html#command-message-type
# MS-05-02 All methods MUST return a datatype which inherits from NcMethodResult.
# When a method call encounters an error the return MUST be NcMethodResultError
# or a derived datatype.
# https://specs.amwa.tv/ms-05-02/branches/v1.0/docs/Framework.html#ncmethodresult

    # Assume single command
    command_handle = command_json['commands'][0]['handle'] if command_json.get('commands') else 0

    self.ncp_websocket.send(json.dumps(command_json))

    results = []
    start_time = time.time()
    while time.time() < start_time + WS_MESSAGE_TIMEOUT:
        if self.ncp_websocket.is_messages_received():
            messages = self.ncp_websocket.get_messages()

            # find the response to our request
            for message in messages:
                parsed_message = json.loads(message)

                if self.message_type_to_schema_name(parsed_message["messageType"]):
                    self.validate_is12_schema(
                        test,
                        parsed_message,
                        self.message_type_to_schema_name(parsed_message["messageType"]),
                        context=self.message_type_to_schema_name(parsed_message["messageType"]) + ": ")
                else:
                    raise NMOSTestException(test.FAIL("Unrecognised message type: " + parsed_message["messageType"],
                                                    "https://specs.amwa.tv/is-12/branches/{}"
                                                    "/docs/Protocol_messaging.html#command-message-type"
                                                    .format(self.spec_branch)))

                if parsed_message["messageType"] == MessageTypes.CommandResponse:
                    responses = parsed_message["responses"]
                    for response in responses:
                        if response["handle"] == command_handle:
                            if response["result"]["status"] != NcMethodStatus.OK:
                                raise NMOSTestException(test.FAIL(response["result"]))
                            results.append(response)
                if parsed_message["messageType"] == MessageTypes.SubscriptionResponse:
                    results.append(parsed_message["subscriptions"])
                if parsed_message["messageType"] == MessageTypes.Notification:
                    self.notifications += parsed_message["notifications"]
                if parsed_message["messageType"] == MessageTypes.Error:
                    raise NMOSTestException(test.FAIL(parsed_message, "https://specs.amwa.tv/is-12/branches/{}"
                                                    "/docs/Protocol_messaging.html#error-messages"
                                                    .format(self.spec_branch)))
                
            if not self.expecting_notifications and len(results) != 0:
                break
            if self.expecting_notifications and len(results) != 0 and len(self.notifications) != 0:
                break

        time.sleep(0.2)
                
    if len(results) == 0:
        raise NMOSTestException(test.FAIL("No Message Response received.",
                                          "https://specs.amwa.tv/is-12/branches/{}"
                                          "/docs/Protocol_messaging.html#command-message-type"
                                          .format(self.spec_branch)))

    if len(results) > 1:
        raise NMOSTestException(test.FAIL("Received multiple responses : " + len(responses)))

    return results[0]`

[jonathan-r-thorpe]

Yes, I think this is approaching a solution.

Although, I'm wondering whether we should somehow decouple the Notification handling from the Command handling. At the moment the assumption is that when the tests update the Device Model of the Node Under Test, any subscription notification will arrive at the same time as the CommandResponse and can therefore be polled using self.ncp_websocket.get_messages().

However, this may well be an unsafe assumption if the notification happens even a short time after the CommandResponse is returned.

Currently the incoming messages are cached in the WebsocketWorker until they are polled. However, it might be better to attach a handler to the websocket and validate/sort the messages as they arrive. Perhaps?

[jonathan-r-thorpe]

Although I guess that is the problem that the expecting_notifications flag is solving.

I will update the code in a separate PR according to your suggestion for your review.

[jonathan-r-thorpe]

@alabou I have now updated the code according to your suggestion.

[alabou]

Looks good to me for the actual test suite usage of notifications. If some day we need to test for notifications that are totally asynchronous with respect to commands then we'll need to revisit this. Thanks.

[alabou]

There is an issue when using notifications (IS-12 test_33) for objects self emitting notifications (i.e. monitors or others) ... test_33 subscribe to all objects and after a command, if expecting notifications, will return as soon as one notification comes back. This is bad because the first notification may not be the expected one. A partial fix is to indicate on start_logging_notifications the oid of the expected notification. This works fine for all but the objects self emitting notifications. The first notification form a self emitting object may not be the expected one. There is no easy fix for that unless we wait until WS_MESSAGE_TIMEOUT for a very specific notification (oid and property id)