Continue processing on failed subtask

[Kani999]

Hello,

I have a task, which contains a loop where a lot of subtasks is executed.
If one of the subtasks fails, I want to continue on other subtasks.

Is there a way to continue on a loop of subtasks if some of them fail?

Currently, the whole task will end when the first exception occurs and it won't continue on the loop.

def test(task: Task):
   for IP in IPS:
       task.run(ip_update, ip=IP)

def ip_update(task: Task, ip, params, ifc_name):
    try:
        raise Exception # raise error
    except Exception as e:
        #  return the error and status failed
        return Result(host=task.host, result=f"Host: {task.host.name} - failed", severity_level=logging.ERROR, failed=True, exception=e)

Thank you

[ktbyers]

@Kani999

Issue #1, your task.run() call is invalid, your ip_update function requires both the params argument and the ifc_name arguments to be passed in and your task.run() call provides neither of those.

Issue #2, in your ip_update exception handler code, you are still setting failed=True which will cause a nornir.core.exceptions.NornirSubTaskError exception to be generated. So even though you handled the first exception, the way you return the Result object in the subtask is just causing a different exception.

There are different ways you can solve this depending on the broader context of what you are doing (i.e. not return failed=True and an exception from the ip_update subtask) or catch the exception at the test task level.

Note, your broader pattern of looping over IPs and calling task.run looks a bit odd to me...but I also don't have the broader context on what you are doing.

Regards, Kirk

[Kani999]

Issue 1 - your task.run() call is invalid ....

• Code I pasted was edited, so it's more like "pseudo-code" but good point

Issue 2, in your ip_update ...

• So if I return the failed=true, I can't continue looping over the IPS, no matter what.

---

My workflow is:

1. Nornir run over multiple hosts - task update_netbox_ips
   • gather data via pyats
   • convert data to a proper format
   • run inner sub-task process_ips
     • Loop over the IPS (gathered data):
       - based on some conditions run sub-sub-task create_ip or delete_ip
       • If creating IP was successful, return Result(changed=true) (everything went ok)
       • If create/delete fails (for any reason) return Result(failed=true) but I want to continue in looping over the rest of IPs. I just want to see, that some subtasks failed, but it does not mean I don't want to continue in processing the addresses.

---

I already changed the workflow to this process: (sub-tasks were removed)

1. Nornir run over multiple hosts - task update_netbox_ips
   • gather data via pyats
   • convert data to a proper format
   • run create/delete ips (as a method (not a task) which returns a list of actions and their response (changed or failed))
   • returns Result where changed or failed represents if everything went ok, or some of the action fails

---

Can you advise me what would be the correct workflow to process such a request via nornir? What should be the Task, sub-task, sub-sub-task, etc.

• Running Nornir.run over specified hosts:
  • get data from PyATS for the host
  • Convert data to a custom format for the host
  • Go through data and perform create/delete actions on data (updating netbox data)
  • Return Result with all actions/diffs that have been made within the update netbox data cycle

[dbarrosop]

So if I return the failed=true, I can't continue looping over the IPS, no matter what.

Yes you can, check the document I linked in another message.

In any case, this is just like any other piece of software, if you return an error you need to recover from it or terminate the application, you can't just continue as is. If an error can be ignored, just codify it in your code. My advice is that you don't return an error if you don't consider it an error or that your task recovers from the error if the subtask fails.

[Kani999]

@dbarrosop I checked the document, but I did not find anything useful for my case

I tried to catch the error when failed=true is set at

def test(task: Task):
   for IP in IPS:
       try:
           task.run(ip_update, ip=IP) # raises exception and return Result(failed=true)
       except Exception as e:
           continue # Even when the error is caught from the `task.run(ip_update, ip=IP)` it will terminate and won't continue in looping

but even when the error is caught, it will terminate the parent task.

But I already changed my workflow, so I do not execute the inner task now, so I solved my problem already.

[dbarrosop]

In addition to what Kirk said, read this https://nornir.readthedocs.io/en/latest/tutorial/failed_tasks.html