CMock: More control on expectation order - proposal

[PVeps]

Use Case: SuT (software-under-test) calls several times some external function. This function sholud be mocked, the call arguments schould be checked. The call order isn't important,

The current CMock version (2.5.4) enforces the expectation registration order in test must be the same as the call order in SuT. An undesired dependency of the test on the SuT implementation arises.

Proposed feature:

• simplified: an option to relax the checking of the call order of the same function.
• more flexible: a possibility to define the call order (of cause inclusive "order doesn't matter").

Something similar is implemented by the Turtle mock library.

I realize that the C++ language gives more oppotunities that C. It might still be possible to implement a subset or something like the Turtle functionality.

[mvandervoord]

You're proposing a half-way feature to give more flexible control than just setting :enforce_strict_ordering: to false?

[PVeps]

Hello Mark. It only works for the different functions. But I need the call order control of the same mocked function.

[mvandervoord]

Ah. I understand what you're saying now. That's an interesting case that I haven't run into often. Typically people DO care about the order for a particular function.

In your scenario, let's say it's an ExpectAndReturn... would you expect the return order to be the order you specified, or would you expect it to be tied to the argument list and THEN specified order secondarily?

For example:

blah_ExpectAndReturn(1, 10);
blah_ExpectAndReturn(2, 20);
blah_ExpectAndReturn(1, 30);

If the release code actually looks like this:

int retval[3];
retval[0] = blah(2);
retval[1] = blah(1);
retval[2] = blah(1);

would you expect retval to be {10, 20, 30} or {20, 10, 30} ?

[mvandervoord]

Also, I assume you're talking about a complete set of arguments, so if you have this:

blah_Expect(1, 2, 3);
blah_Expect(4, 5, 6);

Then this is a failure:

blah(1, 2, 6);

Because, even though we don't care about the order of the argument sets, we DO care that it's a set and not just a member of an argument list?

[PVeps]

I would expect that each registered Expectation is a transaction and these transactions are checked according to the rule defined in the Test Case. If the rule says the order is not important, it would be checked if the expectation was fulfilled at an arbitrary execution point. If the rule states that the mocked functions must be called in a defined order, then the expectations would be checked in that order.

You can put it this way: The order is an overlapping expectation.
The return values belong to each registered expectation, just like the arguments - if it is not "_Ignore" variant of the expectation registration.

With the return values, i.e. the stub functionality (not the mock functionality), it is of course not so easy to implement an arbitrary call order. But if the developer defines such a behavior, as in your first example, then the call order is obviously important. Ignoring the order in this example is probably pointless from the point of view of SuT, not of the test framework.

P.S.: Sorry, I used here terminilogy, which is a little different from terminology in CMock. I meant the following:

• A stub is the function that replaces a function from the production code. The execution of a stub never has the goal to fail a test case. The stub allows linking and provides the return value if necessary.
• A mock, on the other hand, has the goal of direct testing of the SuT the behavior, and can therefore fail the test case if the behavior is not as expected. So "a mock" == "a stub + testing the behavior of SuT".

[PVeps]

Also, I assume you're talking about a complete set of arguments, so if you have this:

blah_Expect(1, 2, 3);
blah_Expect(4, 5, 6);

Then this is a failure:

blah(1, 2, 6);

Because, even though we don't care about the order of the argument sets, we DO care that it's a set and not just a member of an argument list?

Yes, of course.

[PVeps]

My example in the proposal is based on a real situation. The component sets two (abstract) signals in a database using a generic service. The order is unimportant, but the signal ids and the values of the signals (the service arguments) are. The database service is to be mocked.