Add idle model unload for mediapipe LLM graphs by atobiszei · Pull Request #4486 · openvinotoolkit/model_server

atobiszei · 2026-08-28T12:02:58Z

Cherry-picked from idle-models branch (65f2bab).

Cherry-picked from idle-models branch (65f2bab). Adds UNLOADED state machine, per-graph idle unload, ActiveInferenceGuard, config/schema/metrics additions, and related tests.

Cherry-picked from idle-models branch (ceb9f4d). Adds ModelGroupManager, group_name config field, CLI idle_unload_timeout_seconds, status codes, and model_group_manager tests.

atobiszei · 2026-08-31T11:22:45Z

+    // Idle unload: timestamp (nanoseconds from steady_clock epoch) of the last
+    // inference activity. Updated in create() on every inference acquisition and
+    // when an in-flight inference finishes (via ActiveInferenceGuard destructor).
+    // Held as shared_ptr so executors can safely write to it even after the
+    // definition is retired/destroyed — the atomic outlives the definition.
+    std::shared_ptr<std::atomic<int64_t>> lastActivityTimeNs;


This possibly could be moved to servable definition?

atobiszei · 2026-09-01T08:51:34Z

 // Expect old stream to continue responding until closure
 // Expect new stream to be rejected
-TEST_F(MediapipeStreamFlowAddTest, InferOnUnloadedGraph) {
+TEST_F(MediapipeStreamFlowAddTest, InferOnSleepingGraph) {


Wrong rename

atobiszei and others added 11 commits August 28, 2026 13:43

Add idle model unload for mediapipe LLM graphs

67dda0f

Cherry-picked from idle-models branch (65f2bab). Adds UNLOADED state machine, per-graph idle unload, ActiveInferenceGuard, config/schema/metrics additions, and related tests.

Extend idle model management to groups

e844d84

Cherry-picked from idle-models branch (ceb9f4d). Adds ModelGroupManager, group_name config field, CLI idle_unload_timeout_seconds, status codes, and model_group_manager tests.

spelling

722d6ff

stype

9d98be5

style

479feab

fix

542d3ff

fix

58a7096

style

920681e

build fix

e78a175

initialization fix

f9f6dc7

Make model group management use servable loading queue

7df197b

atobiszei commented Aug 28, 2026

View reviewed changes

Comment thread src/model_management/modelmanager.cpp

atobiszei commented Aug 28, 2026

View reviewed changes

Comment thread src/model_management/modelmanager.hpp Outdated

atobiszei commented Aug 28, 2026

View reviewed changes

Comment thread src/model_management/modelmanager.hpp Outdated

atobiszei commented Aug 28, 2026

View reviewed changes

Comment thread src/model_management/modelmanager.hpp Outdated

Update

50bc626