pyterm: add native support

[OlegHahm]

Contribution description

I always found it annoying that native is lacking the convenience features of pyterm (e.g., history, concatenating of commands, logging etc). Of course, one could pipe stdin/stdout into something like socat and use pyterm's feature to connect to a TCP server (e.g., socat EXEC:${NATIVEBIN},end-close,stderr,pty TCP-LISTEN:4711,reuseaddr,fork & pyterm -ts 4711) but that's somewhat tedious and not well-aligned with the typical usage.

Hence, this PR will modify the term make target for native so that pyterm is called instead of calling the native binary directly. In turn, pyterm now has a new option which will spawn RIOT native processes as a subprocess (i.e., child process) and communicate to it via pipes.

Testing procedure

Call BOARD=native make term for arbitrary RIOT applications.

Caution: this is not well tested so far - expect the unexpected.

[miri64]

Setting the CI: run tests label to make sure they are run on CI execution (I can imagine this PR will make some trouble there...). Other than that it looks fine from my side.

[riot-ci]

Murdock results

✔️ PASSED

dd18461 pyterm: native: remove superfluous tap argument

Success	Failures	Total	Runtime
8101	0	8101	10m:09s

Artifacts

• Documentation preview

[OlegHahm]

There is an issue with riotctrl. I assume it is because the native process now becomes a child process of pyterm when called via make term but I was not quite able to debug this.

[benpicco]

Setting the CI: run tests label to make sure they are run on CI execution (I can imagine this PR will make some trouble there...). Other than that it looks fine from my side.

IIRC that is only for hardware based tests, all native tests should always run as part of CI

[OlegHahm]

Okay to squash and remove the WIP label?

[miri64]

Yepp, Murdock will take a while anyways, but tools-test now passes.

[OlegHahm]

I realized that this change breaks when passing arguments to native (like when using USE_ZEP=1). :-(

[miri64]

Mh... maybe, instead of passing parameter by parameter just have a “native parameters” parameter, to which you pass all parameters required for native. This way you can keep the old TERMFLAGS and just put them into the new “native parameters” parameter.

[miri64]

Of course that makes port somewhat unnecessary... not sure this is easily made optional

[OlegHahm]

Of course that makes port somewhat unnecessary... not sure this is easily made optional

Well, port seems to be a special case anyway.

[OlegHahm]

Anything left to be addressed?

[miri64]

What about #20172 (comment) and #20172 (comment)?

[OlegHahm]

Seemed to be unnecessary and would probably complicate the configuration.

[miri64]

Seemed to be unnecessary and would probably complicate the configuration.

Mh, but the comment says why socat was used: To not have extra characters in the stream. I can imagine that the tests depend on that. If you insist on pyterm at least set the pyterm flags that are used with the cleanterm target.

[miri64]

Just for that I set run tests to also test if the behavior non-native is preserved ;-).

[OlegHahm]

Mh, but the comment says why socat was used: To not have extra characters in the stream.

I'm under the impression that this comment is obsolete.

I can imagine that the tests depend on that. If you insist on pyterm at least set the pyterm flags that are used with the cleanterm target.

But as far as I understand the tests are executed using the cleanterm target anyway.

The tests run successful on my PC and in the CI.

From the test code I also see no reason why the tests should fail with pyterm.

[miri64]

Since Murdock does not seem to have a problem with that either, I am fine with that explanation.

[miri64]

Soft ACK from my side. @benpicco, since this touches also ZEP dispatch, can you please have another look too?

[benpicco]

This makes native so much nicer 😃

I also don't see how the tests would depend on pyterm - they use their own python based testing harness (e.g. RIOTCtrl).

examples/gnrc_border_router on native also still works as expected.

[benpicco]

Uh looks like tests/build_system/test_tools did not like this

[benpicco]

I also don't see how the tests would depend on pyterm

Turns out I was wrong - or rather they depend on the absence of pyterm. (As in they don't use pyterm on the target boards either, but now will always use it for native)

Now for some reason CI did not catch this, but an unrelated PR manages to shake them out.

[benpicco]

arg this also broke make debug on native