parallelizing _recursive_parse(...) function

[dpmilian]

The process where a client requests the custom data structures from a server (function load_data_type_definitions()) is quite slow, in my limited scenario it takes up to 10 seconds after a connection. I see that quite a lot of time is spent in the _recursive_parse() function in the asyncua.common.structures104 module, basically because all the recursive requests are performed synchronously (i.e. awaited within the loop). A quick modification to parallelize in part:

async def _recursive_parse(
    server, base_node, dtypes, parent_sdef=None, add_existing=True, level=0
):
    ch = await base_node.get_children_descriptions(refs=ua.ObjectIds.HasSubtype)

    requests = []
    for desc in ch:
        name = clean_name(desc.BrowseName.Name)
        requests.append(
            _read_data_type_definition(server, desc, read_existing=add_existing)
        )

    results = await asyncio.gather(*requests)

    requests = []

    for sdef, desc in zip(results, ch):
        name = clean_name(desc.BrowseName.Name)

        if sdef:
            if parent_sdef:
                for sfield in reversed(parent_sdef.Fields):
                    sdef.Fields.insert(0, sfield)
            dtypes.append(DataTypeSorter(desc.NodeId, name, desc, sdef))
            # print(f"\t [{parent_sdef}] here")
            requests.append(
                _recursive_parse(
                    server,
                    server.get_node(desc.NodeId),
                    dtypes,
                    parent_sdef=sdef,
                    add_existing=add_existing,
                    level=level + 1,
                )
            )
        else:
            requests.append(
                _recursive_parse(
                    server,
                    server.get_node(desc.NodeId),
                    dtypes,
                    parent_sdef,
                    add_existing=add_existing,
                    level=level + 1,
                )
            )

    await asyncio.gather(*requests)

provides a significant speed up (less than half the time). I have not run thorough tests yet but in my current application the results seem correct, are there any pitfalls I am not aware of?

[oroulet]

Good suggestion. I think parallelizing crawling is fine