[CPU] Enable memory reuse for nested graphs #27521

EgorDuplensky · 2024-11-12T10:53:06Z

Details:

All the nested graphs are now must be a part of a global memory reuse logic
The core logic of the memory reuse is untouched (a bit refactored)
Instead of solving memory reuse for every graph / subgraph, now all the edges and global execution indices are collected from the "virtually flatten" graph first and then memory reuse is solved once for a model.
All the nodes with nested graphs are updated, including:
1. LoRa
2. Composite
3. If
4. TensorIterator
5. Convolution + Sum fallback subgraph

Tickets:

ticket-id

EgorDuplensky · 2024-11-13T13:47:42Z

@maxnick Ready for review. Could you please take a look?

EgorDuplensky · 2024-11-14T10:51:34Z

Added a fix for Convolution + Sum fallback graph.
Now such graph is also a part of memory reuse.

src/plugins/intel_cpu/src/compiled_model.cpp

src/plugins/intel_cpu/src/edge.h

src/plugins/intel_cpu/src/graph.h

src/plugins/intel_cpu/src/node.h

src/plugins/intel_cpu/src/nodes/composite.cpp

src/plugins/intel_cpu/src/nodes/conv.cpp

src/plugins/intel_cpu/src/nodes/lora.cpp

src/plugins/intel_cpu/tests/functional/cmake/target_per_test.cmake

src/plugins/intel_cpu/src/graph.cpp

src/plugins/intel_cpu/src/nodes/lora.cpp

src/plugins/intel_cpu/src/graph.cpp

src/plugins/intel_cpu/src/nodes/conv.cpp

github-actions · 2024-12-26T00:23:08Z

This PR will be closed in a week because of 2 weeks of no activity.

github-actions · 2025-01-17T00:22:58Z

This PR will be closed in a week because of 2 weeks of no activity.

EgorDuplensky · 2025-01-21T19:08:01Z

src/plugins/intel_cpu/src/graph_context.cpp

+      m_auxiliaryNetworkMemoryControl(std::make_shared<NetworkMemoryControl>()),
+      m_memoryControl(m_auxiliaryNetworkMemoryControl->createMemoryControlUnit()) {


@dmitry-gorokhov I decided to not create a wrapper around m_auxiliaryNetworkMemoryControl and m_memoryControl.
Instead, it is redesigned, so m_memoryControl is just one memory control instance of m_auxiliaryNetworkMemoryControl

src/plugins/intel_cpu/src/node.h

src/plugins/intel_cpu/src/nodes/lora.cpp

src/plugins/intel_cpu/src/edge.h

src/plugins/intel_cpu/src/infer_request.cpp

EgorDuplensky · 2025-01-28T16:12:51Z

src/plugins/intel_cpu/src/graph.cpp

+    if (memoryControl->allocated()) {
+        return;  // memory is already allocated globally


The problem here is the way the legacy graph pipeline stages are implemented.
With a proper pipeline such situation will not be possible.

src/plugins/intel_cpu/src/graph.cpp

src/plugins/intel_cpu/src/compiled_model.h

src/plugins/intel_cpu/src/nodes/memory.cpp

src/plugins/intel_cpu/src/compiled_model.h

maxnick · 2025-01-31T15:43:04Z

src/plugins/intel_cpu/src/graph.cpp

+    if (memoryControl->allocated()) {
+        return;  // memory is already allocated globally


Are there any plans of implementing the proper pipeline? How it should look like?

src/plugins/intel_cpu/src/graph.cpp

src/plugins/intel_cpu/src/graph_context.h

src/plugins/intel_cpu/src/infer_request.cpp

src/plugins/intel_cpu/src/nodes/if.cpp

src/plugins/intel_cpu/src/nodes/input.cpp

src/plugins/intel_cpu/src/nodes/memory.cpp

src/plugins/intel_cpu/src/nodes/lora.cpp

src/plugins/intel_cpu/tests/unit/graph/merge_transpose_reorder_test.cpp

src/plugins/intel_cpu/tests/unit/graph/memory_state.cpp

EgorDuplensky · 2025-02-04T10:53:55Z

@maxnick Could you please check one more time?

maxnick · 2025-02-05T09:04:11Z

@dmitry-gorokhov , could you please make the final review?

EgorDuplensky · 2025-02-05T20:45:22Z

Rebased to fix merge conflict

Sync node indexes must be registered to global allocation context in order.

EgorDuplensky requested review from a team as code owners November 12, 2024 10:53

github-actions bot added category: inference OpenVINO Runtime library - Inference category: CPU OpenVINO CPU plugin category: build OpenVINO cmake script / infra labels Nov 12, 2024

EgorDuplensky changed the title ~~Enable memory reuse for nested graps~~ Enable memory reuse for nested graphs Nov 12, 2024

EgorDuplensky changed the title ~~Enable memory reuse for nested graphs~~ [CPU] Enable memory reuse for nested graphs Nov 12, 2024

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch from da1d1a0 to 18563de Compare November 12, 2024 15:27

EgorDuplensky mentioned this pull request Nov 12, 2024

[CPU] Refactor memory control and allocation #27259

Closed

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch 2 times, most recently from 4734e61 to e140018 Compare November 13, 2024 13:32

github-actions bot removed the category: inference OpenVINO Runtime library - Inference label Nov 13, 2024

EgorDuplensky assigned maxnick Nov 13, 2024

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch from e140018 to d5bda07 Compare November 13, 2024 13:51

maxnick requested changes Nov 22, 2024

View reviewed changes

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch 2 times, most recently from 408e748 to 11a5115 Compare November 25, 2024 15:28

maxnick reviewed Dec 3, 2024

View reviewed changes

src/plugins/intel_cpu/src/graph.cpp Outdated Show resolved Hide resolved

src/plugins/intel_cpu/src/graph.cpp Outdated Show resolved Hide resolved

src/plugins/intel_cpu/src/graph.cpp Outdated Show resolved Hide resolved

maxnick reviewed Dec 4, 2024

View reviewed changes

src/plugins/intel_cpu/src/nodes/lora.cpp Outdated Show resolved Hide resolved

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch 2 times, most recently from 7f3fadf to 9e170ae Compare December 10, 2024 17:56

maxnick reviewed Dec 11, 2024

View reviewed changes

src/plugins/intel_cpu/src/graph.cpp Show resolved Hide resolved

src/plugins/intel_cpu/src/nodes/conv.cpp Show resolved Hide resolved

github-actions bot added the Stale label Dec 26, 2024

mg-intel removed the Stale label Jan 2, 2025

github-actions bot added the Stale label Jan 17, 2025

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch 3 times, most recently from ed7ca84 to 467ef3b Compare January 28, 2025 17:32

EgorDuplensky commented Jan 28, 2025

View reviewed changes

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch 3 times, most recently from 9ac1226 to 2497eb3 Compare January 30, 2025 12:19

maxnick reviewed Jan 31, 2025

View reviewed changes

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch from c1d5f75 to 482a180 Compare February 3, 2025 18:10

maxnick added this to the 2025.1 milestone Feb 4, 2025

maxnick approved these changes Feb 4, 2025

View reviewed changes

maxnick assigned dmitry-gorokhov Feb 5, 2025

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch from 5f112c1 to aa3a98b Compare February 5, 2025 20:45

dmitry-gorokhov approved these changes Feb 6, 2025

View reviewed changes

EgorDuplensky added 6 commits February 6, 2025 13:44

[CPU] Enable memory reuse for nested graphs

c828723

Fix e2e issue

fdecfe8

Sync node indexes must be registered to global allocation context in order.

Apply review comments

7765ac1

Fix CI issue

ec4aa58

Apply review comments 2

c75bcec

Apply review comments 3

3c5620c

EgorDuplensky force-pushed the enable_memory_reuse_for_nested_graphs branch from aa3a98b to 3c5620c Compare February 6, 2025 12:44

maxnick added this pull request to the merge queue Feb 6, 2025

Merged via the queue into openvinotoolkit:master with commit be3edd1 Feb 6, 2025
181 checks passed

maxnick deleted the enable_memory_reuse_for_nested_graphs branch February 6, 2025 17:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CPU] Enable memory reuse for nested graphs #27521

[CPU] Enable memory reuse for nested graphs #27521

EgorDuplensky commented Nov 12, 2024 •

edited

Loading

EgorDuplensky commented Nov 13, 2024

EgorDuplensky commented Nov 14, 2024

github-actions bot commented Dec 26, 2024

github-actions bot commented Jan 17, 2025

EgorDuplensky Jan 21, 2025

EgorDuplensky Jan 28, 2025

maxnick Jan 31, 2025

EgorDuplensky commented Feb 4, 2025

maxnick commented Feb 5, 2025

EgorDuplensky commented Feb 5, 2025

		m_auxiliaryNetworkMemoryControl(std::make_shared<NetworkMemoryControl>()),
		m_memoryControl(m_auxiliaryNetworkMemoryControl->createMemoryControlUnit()) {

		if (memoryControl->allocated()) {
		return; // memory is already allocated globally

[CPU] Enable memory reuse for nested graphs #27521

[CPU] Enable memory reuse for nested graphs #27521

Conversation

EgorDuplensky commented Nov 12, 2024 • edited Loading

Details:

Tickets:

EgorDuplensky commented Nov 13, 2024

EgorDuplensky commented Nov 14, 2024

github-actions bot commented Dec 26, 2024

github-actions bot commented Jan 17, 2025

EgorDuplensky Jan 21, 2025

Choose a reason for hiding this comment

EgorDuplensky Jan 28, 2025

Choose a reason for hiding this comment

maxnick Jan 31, 2025

Choose a reason for hiding this comment

EgorDuplensky commented Feb 4, 2025

maxnick commented Feb 5, 2025

EgorDuplensky commented Feb 5, 2025

EgorDuplensky commented Nov 12, 2024 •

edited

Loading