[ For discussion ] New Features for Nornir

[ubaumann]

At AutoCon1 I had some great discussions about Nornir. I want to collect some ideas for new features (could be a minor change, major change, or plugins) and have an open discussion.

1. Async support Async architecture (inventory) #880, Async Architecture Topics #881 (by @ktbyers, @dgarros)
   1.1 async inventory
   1.2 async runner
2. Possibility to replace the data store strategy Inventory Data Plugin #889 (by @ubaumann)
3. Integrated task dispatching/abstraction for different OS/platforms (by @dgarros, @gbieli)
4. Session Log for connection plugins (by @gbieli)
5. Easy support/interface to use the internal nornir logging (by @gbieli)
6. Location for textFSM parsing templates (by @gbieli)
7. [plugin] Conditional runner (A condition could be not both PEs can be touched at the same time) (by @ubaumann)
8. Access Result in a MultiResult by job name and not index (by @gbieli, @ubaumann)
9. [brainstorming] Could we introduce a schema to Result.result to better know what kind of data a task returns? (by @ubaumann)
10. connection_options[conn_type] merge (more details: comment) (by @ktbyers)
11. Built-in support for encrypted data (in some way) with SimpleInventory or YAMLInventory (by @ktbyers)
12. Credential Helper/Object/Plugin. (like a key-value store to access secrets) (by @ubaumann)

I am happy to implement or coordinate some of these features. The discussion should aim to show whether these features would be of interest and whether there are more ideas out there.

[itheodoridis]

Hello Urs, quite a list!
I would like some more info on 6. I am usually running Netmiko in my Nornir tasks, I have no problem using textfsm there (with use_textfsm=True passing through), is it about defining custom location for those templates instead of env vars or something similar?

[ubaumann]

Hey Ioannis,
Yes, if I remember all the details correctly, it would be the idea to see how custom templates can be easier be managed in the way of "shipping" and consuming. Maybe the same solution could work for TTP or other template files.
To summarise: it should be possible to use custom templates, and after a pip install, it should work out of the box.
Does that make sense?

[itheodoridis]

Yes! Thank you, now I understand. Excellent idea!

[ktbyers]

Have connection_options[conn_type] be a dictionary merge and not a first match (i.e. merge where host overwrites group which overwrites default), but if a given key was not overwrote, then it would still be in use.

[ktbyers]

Async has two separate, but related items: async inventory, and async runner.

[ktbyers]

Built-in support for encrypted data (in some way) with SimpleInventory or YAMLInventory.

[ubaumann]

Built-in support for encrypted data (in some way) with SimpleInventory or YAMLInventory.

I like the support for encrypted data for the SimpleInventory or YAMLInventor. I just also added a new point for a Credential Key-Value store. I could imagine having an instance or part of it stored in the host object and it can be used to query secrets.

[ktbyers]

Yeah, I envision the SimpleInventory/YAMLInventory encryption as being very simple solution (and if you want something more sophisticated, then you would want some other solution).

[dbarrosop]

Nice work compiling the list. I'd suggest breaking it down into individual issues and detailing a bit more because a lot of things are very obscure for the people that weren't there.

Keep in mind I may be a bit on the fence about complex features that need to make it into the core, mostly because writing the feature is the easy/fun part, the work comes later having to maintain and live with the feature.

[aedwardstx]

In my organization, we ended up writing a similar library internally; largely because of our desire for a type safe solution. Here's my wish list:

• Strict type safety with mypy and/or pyright.
  • End to end type safety is not currently possible with Nornir which is very problematic for projects that enforce it.
  • A big part of this issue stems from MultiResult as it is not practical/possible to describe a variable length non-homogenous sequence with Python's type system. I ended up simply removing MultiResult leaving just AggregatedResult[Result] and couldn't be happier. This forces the user to include any important details in the Result.
    • MultiResult encourages not distilling results to a simpler form.
    • MultiResult objects contain the result of the task and sub_tasks executed for a host.
    • Results from sub_tasks should be distilled instead of returning all the sub_task results.
• Encourage or force the use of Pydantic models for task results. This helps ensure that job output can be serialized in API use cases.
• Encourage or force the use of immutable objects wherever possible.
• Importing Nornir was, the last time I checked, slow; ~350ms due to a pkg_resources import. Keep tabs on the import time since Nornir is frequently used in CLIs.
• Platform is a str vs. an Enum which limits the use of exhaustiveness checking offered by mypy/pyright.
• A much less magical plugin system. My preference would be to have no plugin system at all and use composable functions instead.

[ktbyers]

Yeah, all features should/will require testing and documentation i.e. without these it is not complete. And also probably a willingness to help with issues long term.

[ubaumann]

In my organization, we ended up writing a similar library internally; largely because of our desire for a type safe solution. Here's my wish list:

• Strict type safety with mypy and/or pyright.

  • End to end type safety is not currently possible with Nornir which is very problematic for projects that enforce it.

  • A big part of this issue stems from MultiResult as it is not practical/possible to describe a variable length non-homogenous sequence with Python's type system. I ended up simply removing MultiResult leaving just AggregatedResult[Result] and couldn't be happier. This forces the user to include any important details in the Result.

    • MultiResult encourages not distilling results to a simpler form.
    • MultiResult objects contain the result of the task and sub_tasks executed for a host.
    • Results from sub_tasks should be distilled instead of returning all the sub_task results.

• Encourage or force the use of Pydantic models for task results. This helps ensure that job output can be serialized in API use cases.

• Encourage or force the use of immutable objects wherever possible.

• Importing Nornir was, the last time I checked, slow; ~350ms due to a pkg_resources import. Keep tabs on the import time since Nornir is frequently used in CLIs.

• Platform is a str vs. an Enum which limits the use of exhaustiveness checking offered by mypy/pyright.

• A much less magical plugin system. My preference would be to have no plugin system at all and use composable functions instead.

Disclaimer: I am just a user like everyone else
We should try to keep the framework as simple as possible while providing the most value. I'm a fan of Pydantic and type hints, but we shouldn't make it harder for beginners to use. Personally, I support the plugin system and modular separation. For me, plugins are straightforward and easy to understand. There are parts we could optimize, but we should probably also not introduce too many breaking changes.

How are people using the platform string?

[ktbyers]

Yeah, I think we should keep the plugin system the same.

Platform string is passed to the plugin via the connection code and the plugin would decide its use/meaning (NAPALM, netmiko, scrapli, etc) so I don't really see how it would be enumerated (and don't think enumerating it is all that important).

[ubaumann]

@SimLi1333 took a swing at number 7. Conditional Runner

https://github.com/InfrastructureAsCode-ch/nornir_conditional_runner

Great work. The pull request to add it to the plugin list is right around the corner.