[Subgraph] Enabling a feature to manage a graph in subgraph level.

[EunjuYang]

Abstract

To accommodate heterogeneous compute engines, it becomes necessary to separately administer distinct graph regions — referred to as subgraphs — each tailored to function according to specific compute engines. This approach differs from our current method of managing only one graph solely at the layer level; therefore, adding a new tier labeled 'subgraph' would allow us to structure our system hierarchically as follows: Graph > Subgraphs > Layers.

Requirements for Subgraphs

1. Each subgraph type should be able to define its own operations (e.g., forwarding, backwarding).
   • Currently, NNTrainer uses GraphCore to manage layer information within a graph, and the operations (e.g., forwarding, backwarding) in this graph are given from higher levels of the network.
   • These functions need to be modified so they can be managed at the subgraph level.
2. Subgraphs should be able to define their operations based on the compute engine type (e.g., CPU, NPU, GPU).
3. Both topological sorting at the subgraph level and at the layer level are needed.
4. Networks should be saved separately for each subgraph.
5. This feature should support backward compatibility.
6. Various subgraph type should be pluggable

Idea to implement

• In order to fulfill the requirements, I'm planning to implement base class for subgraph that inherits GraphNode. By doing so, we can manage subgraphs with NetworkCore, which supports topological sort and node management.
• As an interface to add a subgraph, I'm planning to add a new property used when createLayer is called, with a default value.

[EunjuYang]

In PR #2889, I added an API to create a SubGraph object (i.e., createSubGraph). Currently, this feature has been implemented for the nntrainer-dev API but not opened up for the ml::train API.
At present, PR #2889 includes only two properties: subgraph_name and engine, although I plan to expand these features further by adding new ones like input_subgraph and output_subgraph to facilitate connections between subgraphs. If you have any suggestions regarding additional SubGraph properties, feel free to leave a comment below.