Utils on a remote machine: which approach?

[beraldoleal]

Hi all,

I'm opening this, because I believe we have a repeated pattern here. So I would like to open this discussion before I proceed to a more "drastic" change.

Some (if not all) of our utility methods needs to be executed on a remote host over SSH sometimes by some tests. And right now we are handling this in a ad-hoc manner. For instance network utils is doing a more OOP approach with some classes (RemoteHost and LocalHost()). But some we try to pass a session to each method (i.e distro and #4857).

I would love to see a more standard approach across our libraries, for instance using the same approach as in network example. And if you think this is a good way to go, we could think about using heritage and extending the base Host class or delegating to proper classes with properties.

IMO would be nice to have a common interface for local and remote calls, examples:

host = LocalHost()
host.arch
host.interfaces
host.packages
host.cpu.family

And if using a remote host, just a different object would need to be instatiated:

host = RemoteHost('192.168.1.1')
....

For sure I' m missing some important keys and to avoid frustration and regrets during the implementation, I would like to know what are you toughs on this.

[ana]

Hi @beraldoleal Thanks for opening this discussion. +1 to have a common interface for local and remote hosts.
I do wonder if this is something we need to implement fully or whether we would be checking for a library that does this first. But I have no strong feelings about the implementation.

[beraldoleal]

I don't see a need for a library for that. Is just a matter of better organization of the classes of some utility libraries.

If accepted, I would say that we could even use that as requirement (following this standard) before migrate the library to the new "utils" repository. (That is one of the reasons that I was thinking about a slow migration, starting with only network).

[willianrampazzo]

Looks like an interesting approach. I don't think I have a strong opinion now. The only thing I can think of is the benefits inside a test. With more spawners added, I don't see much benefit on the test side. On the other hand, I can see the benefit when using it as a utility outside of a test.

[beraldoleal]

Some tests they stress the network. For instance a test is running locally but is trying to bootstrap a remote machine and running commands on that machine. So test writters don't have to care about starting the ssh session here, since those classes are already doing this.

host = RemoteHost('192.168.1.1')
self.assertTrue(len(host.interfaces), 3)

[willianrampazzo]

Right, I understand the use case for the network utility. Still, I could not find a use case for other utilities, like the software manager, the asset, the cpu, and so on, when Avocado introduces spawners for remote machines.

I'm not against the change; I'm trying to find the use cases that make it worth the effort.

[beraldoleal]

We do have some requests to utility library change right now. For instance #4857

[cforno12-zz]

@willianrampazzo , I could give you an example of my use case.

I want to write an avocado test that runs a network preformance tool on both hosts (local and remote), but the suite requires some package dependencies, so I would like to check if the remote host has those dependencies before I attempt to run the preformance tool, and install those dependencies as well. I hope that gives you a clearer picture of what I'm attempting to accomplish

[willianrampazzo]

I understand that today, @cforno12, but with a remote spawner and using the requirements tags on the tests, that won't be needed anymore.

[pevogam]

As most of our tests are using some form of networking I could possibly share an opinion I have formed from the larger scale of application of all these approaches. We have some utilities that make use of session exactly as originally pointed out, a few that have inheriting classes specializing in handling the remote conversions, as well as some significant serialization.

Trying from all of these (and still keeping pretty much all of them around), I could say that the most extendable approach for us remained using the remote door (which is not fully contributed to and available from aexpect). The main benefit from using it in our case remains the fact that we don't have to create a separate remote class for each normal class we use which could quickly clutter our class hierarchies and is not possible if we would like to run some code from an external library remotely. Here as an example usage:

from aexpect import remote_door as door
# initialize aexpect session here
door.run_remote_util(session, 'extlib.subpackage',
                     'remote_method')

will run extlib.subpackage.remote_method() remotely. There are more advanced methods using persistent remote objects and more. There are of course also limitations about serialization but it works well for most basic data types. @beraldoleal I am not sure if you really mean your question for scaling up or just small enough application. I guess passing session or providing custom classes is good enough for small application but this is at least what we have judged as scaling better eventually in case you are interested to check it out.

[pevogam]

Hi @clebergnu, let me try to answer to the best of my knowledge without digging into the implementations we have, take this with a grain of salt just in case.

1. I noticed that in the serialization, arguments become strings. I imagine that this is what currently limits the APIs to "most basic data types", correct?

Not necessarily. This is the case with some of the simpler methods like remote utility calls what rely on a control file (inspired from the Autotest times) and thus do some serialization in direction of strings, but more advanced methods like using persistent remote objects could use custom serializers (e.g. pickle, serpent). All remote approaches we use are listed in:

https://github.com/intra2net/avocado-i2n/wiki/Step-3:-Coordinating-multiple-VMs#enhanced-remote-checks

and in particular the subsection

https://github.com/intra2net/avocado-i2n/wiki/Step-3:-Coordinating-multiple-VMs#remote-objects

uses much more elaborate serialization where the remote method could return a proxy object which is always directly serializable without writing any additional code. We utilize this shortcut for instance in our GUI test automation tool:

https://guibot.readthedocs.io/en/latest/source/guibot.guibot_proxy.html

(we avoid using custom or additional serializer and yet return python objects of greater complexity).

2\. Have you given any thought, or have you prototyped, something that lifts that limitation?

There are still some very special cases where the remote call could use something very wild like remote os.path module and very internal python modules but we have rarely had to go that far. We made use of a bit of serialization code for some custom classes eventually but we were not happy with the extensibility of this (just like I meant with the remote classes above) and an approach like using the remote door with its PyRO objects returning only PyRO objects (thus always serializable) worked our really well.

3\. Have you found any hard to solve situations?  One that comes to my mind is the "leaking" of output that can influence the result (say some method prints an unintended `RESULT`).

It seems that I am answering each question here while answering the previous one but a few hand picked issues were with very special python modules that have a lot of strong system dependencies. We haven't had any problems with leaking output that I can think of, aexpect manages output collection pretty well so far.

[beraldoleal]

will run extlib.subpackage.remote_method() remotely. There are more advanced methods using persistent remote objects and more. There are of course also limitations about serialization but it works well for most basic data types. @beraldoleal I am not sure if you really mean your question for scaling up or just small enough application. I guess passing session or providing custom classes is good enough for small application but this is at least what we have judged as scaling better eventually in case you are interested to check it out.

@pevogam my question it was more on the design aspect in terms of usability. I was not discussing the real method for sending a command or executing a RPC. Currently we are using this approach only with network utils that is using the remote_session.cmd(), but we could use the aexpect or any other RPC method behind the scene (if needed).

Currently multiple utility libraries are using a different method with different APIs with regarding the remote execution. I'm trying to discuss the API design itself, so we could hide implementation details from the users. IMO, users should not need to understand how aexpect works to get a cpu information in a remote host. I would prefer to run print(remote_host.cpu) and print(local_host.cpu) to have an uniform method of access.

[pevogam]

I am not sure I ever mentioned what part is visible to the user here and there are quick ways to wrap scalable logic like everything described above in a few classes or perhaps even a single class without making it visible to the user. But take it with a grain of salt and perhaps we could clarify more if/when there is a PoC of some kind.

[beraldoleal]

You can take as a PoC the utils/network/ package. I will take a deeper look inside aexpect and see how we could incorporate there (for instance inside common.py).

[pevogam]

Just a small reminder about aexpect, there is a ops utils PR (you had some some review already) that adds some useful utils based on the remotization approaches we discuss, e.g. taking a session extra argument.