Repository navigation
Minicpm ocr - #876
Closed
michyu-amd wants to merge 19 commits into
Closed
Minicpm ocr#876michyu-amd wants to merge 19 commits into
michyu-amd wants to merge 19 commits into
Conversation
Wires the forked minicpm_ocr_npu engine into FastFlowLM: MiniCPM_OCR wrapper,
automodel.hpp include, the four all_models.hpp edits, the backend registration,
the CMake link, a model_list.json entry (name MiniCPM-OCR-1.0-NPU2, family
minicpm-ocr), the vendored .so, the nine xclbins and a test harness.
The xclbins are MiniCPM-V-4.7's, byte-identical: this model reuses Qwen3.5-0.8B's
text kernels and Qwen3.8-27B's vision kernels, so the port needed no new kernel
and no new MODEL_TYPE.
One real behaviour fix in modeling_minicpm_ocr_image.cpp. find_best_resize()
hardcoded `patch * 4`, which is the "16x" case:
image_processing_minicpmv4_6.py:
# "4x" merges one 2x2 block, "16x" merges twice
merge_factor = 2 if downsample_mode == "4x" else 4
MiniCPM-OCR is "4x" and merges once, so it needs `patch * 2`. Left as it was,
every image would be resized to a resolution the HF processor never picks, and
the token counts would drift with no error anywhere. `merge` is now derived from
MINICPM_OCR_DOWNSAMPLE_FACTOR and threaded through both call sites.
Verified on the NPU. The three-phase test passes end to end -- prompt, cached
follow-up, clear_context, thinking (17*23 = 391), and an image. Decode ~45-48
tok/s, prefill 131-580 tok/s. A 960x720 photo costs 1287 prompt tokens and a
783x646 text page reads back correctly, LaTeX included.
Image token counts match HF exactly (1260 for both images tested). The 4-token
prompt difference against HF is the empty <think>\n\n</think>\n\n block that
HF's template always inserts and FLM lets the model generate -- a chat-format
choice, not a mismatch.
The harness Makefile also needed common/prefix_slot_cache.cpp and -lwebp, which
automodel.cpp and image_reader.cpp reference unconditionally. No other test
harness lists either, so they share this gap.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every call site reaches fail_inference() from a `catch (...)`, so the engine's actual error -- an XRT failure, a bad shape, a throw from the vision tower -- was discarded and the user was told only "Inference failed; the current conversation was cleared". That cost us a diagnosis today: the very first run of the harness failed during decode at context length 28 with no further detail, and has not reproduced in 19 subsequent runs (10 short, 5 full four-turn, 4 ad hoc). With nothing to go on, the one occurrence is unrecoverable. std::current_exception() is still live inside a catch block, so a bare `throw;` recovers the message without changing control flow, and the guard keeps it safe if fail_inference() is ever reached outside a handler. Both paths verified in isolation: the rethrow recovers the original what(), and the no-exception case reports rather than terminating. The four-turn test still passes unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… set FLM_MINICPM_OCR_DUMP writes `tokens`, which insert() has already shortened by think_trim, so it ends at "<|im_start|>assistant\n" and is 2 tokens (thinking on) or 4 (off) shorter than the context the model actually sees -- decode() feeds the <think> tokens back so the KV checkpoint can land on a boundary the next turn reuses. Compared against HF's apply_chat_template() output, that reads as a prompt mismatch when there is none. It cost a long detour today: the rendered template is byte-identical to HF in both modes, which was only settled by rendering the same .jinja through minja directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The harness ran four fixed turns, so prefill could only be measured at the one
length its image prompt happened to produce. --prompt-file runs a single turn on
a file's contents and exits, which is what made the GDN/conv prefill comparison
measurable across 1k to 24k tokens (models_simple/test_input/{1k,2k,8k}.txt,
concatenated for longer).
That mattered: at 1k the array looked 82% slower, at 24k only 10%. One length
would have given a badly wrong answer in either direction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With --prompt-file, --follow-up <file> sends a second turn the way flm serve sends one (whole conversation, restore_allowed), so the prompt cache prefills only from its saved head. That is the path where one prefill call can take a different GDN backend from the one before it, and it is what tested the host <-> array state handoff in both directions. Note the cache only splits a turn into head + tail when the head is at least min_boundary = 512 tokens; a shorter first turn is re-prefilled whole. Co-Authored-By: Claude <noreply@anthropic.com>
…gh the prefix slots Three conversations, each a first turn plus a follow-up: A a ~2k-token document (turn 1 runs the GDN on the NPU, the follow-up on the host), B a short prompt, C an image (skipped with --image ""). Phase 1 runs each one alone with the slots off (--rounds times, to see the engine repeat itself); phase 2 runs A1 B1 C1 A2 B2 C2 with the slots on and requires every follow-up to resume from its own snapshot, at the same point as the reference, with every reply byte identical; phase 3 repeats the interleaved order with the slots off to show what they save. Greedy, penalties off. First run (--Length 96): PASS. Follow-up prefill, slots on vs off when interleaved: A 336 vs 783 ms, B 149 vs 185 ms, C 313 vs 2015 ms. Co-Authored-By: Claude <noreply@anthropic.com>
With the prefix slots on, --follow-up's CLI-style turn one saves no snapshot, so turn two prefills the whole conversation in two calls (head, then tail). --resume sends turn one with restore_allowed as well, so turn two resumes from its snapshot and prefills only the reply and the new turn, like a cached follow-up in `flm serve`. The --follow-up comment said that already; it is corrected to say what --follow-up really does. Note: "Prefill time" in the statistics block accumulates across turns; turn two's own time is the difference. Co-Authored-By: Claude <noreply@anthropic.com>
The merge of main brought b4c2b90, which adds `continue_final_message` to AutoModel::apply_chat_template and updates every wrapper; MiniCPM_OCR only exists on this branch, so it no longer built ("marked override, but does not override"). Ported MiniCPM-V's changes from that commit by hand, as MiniCPM_OCR's wrapper is a fork of it: - the new apply_chat_template signature, rendering through _render_chat_template(); supports_continue_final_message() -> true - continue mode keeps every prompt token (think_trim 0) and feeds no think preamble in decode() - token_history drops a token the engine never ran (EOS not forwarded, or a turn cut short by max_tokens / cancel / the context cap) decode() and apply_chat_template() are now identical to MiniCPM-V's but for the class name. Only MiniCPM-OCR's files change. Checked on the NPU: the default 4-turn test gives the same replies as before, --multi-slot passes. Co-Authored-By: Claude <noreply@anthropic.com>
Replace the locally built libminicpm_ocr_npu.so and GateDeltaNet_prefill.xclbin with the IRON CI artifacts from AMD-ROCm-Internal/FastFlowLM_IRON PR #117 (run 40974217259, head 028476f9c), and add the missing XRT Windows dll/lib and the HRX so/dll/lib. Register minicpm_ocr_npu.dll in the inno and wix installers. The other MiniCPM-OCR xclbins (attn, conv, layer, lm_head, mm, vision_*) are byte-identical to the ones already shipped for Qwen3.5-0.8B / MiniCPM-V-4.7. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Engine (lib/xrt/libminicpm_ocr_npu.so) and new xclbins in xclbins/MiniCPM-OCR-1.0-NPU2/ from FastFlowLM_IRON branch minicpm_ocr: - vision_overlay.xclbin: vision attention + matmul A in one xclbin. - layer_with_lm_head.xclbin, layer_with_lm_head_kvb.xclbin: decode layers and lm_head in one design (no xclbin switch per token); the _kvb build reads finished K/V chunks as host-packed bfp16 and skips the y rescale when no row max moved (preferred when present). - attn_bfp16.xclbin: prefill attention on host-made bfp16 q / K / V^T tiles. - mm_bfp16_glu.xclbin: prefill matmul on host-made bfp16 x tiles, with the GLU computed on the array. - Engine: host prefill work under the array runs (row chunks, fused norms, range syncs), GDN host/array switch point 1536 -> 5120 tokens, a bfp16 KV copy for decode. Each new path has an env switch back to the old one. Speed (Strix Halo, balanced profile): prefill 124 tok 122 -> ~84 ms, 1K 383 -> ~230, 8K 2,218 -> ~1,380, 24K 7,878 -> ~4,400; decode 47.5 -> ~66.7 tok/s short, 26.7 -> ~46.4 at 24K. Golden replies changed only at exact or near ties; checked against a CPU fp32 reference. Local build, not the CI one. Co-Authored-By: Claude <noreply@anthropic.com>
Brings in the remote branch (main merge, CI-built Windows / HRX engine libs, CI GateDeltaNet_prefill.xclbin, model list, installer entries). Conflict in src/lib/xrt/libminicpm_ocr_npu.so resolved to the local engine (built from FastFlowLM_IRON minicpm_ocr, which also contains the remote IRON branch; rebuilding it from the merged IRON tree gives the same bytes). The CI GateDeltaNet_prefill.xclbin gives bit-identical prefill logits (512..24K). The Windows / HRX libs stay the CI builds of the earlier engine source. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…(CI run 42411780863) Linux/Windows libs for XRT and HRX and the 13 xclbins of minicpm-ocr:1.0, exactly as built by CI (IRON 796e61d52). conv.xclbin is unchanged. Checked on Strix Halo: 12/12 golden replies and prefill logits identical to the local build, multi-slot prompt cache passes, speed unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
…ngine) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
minicpm-ocr:1b(MiniCPM-OCR-1.0-NPU2): text and image input, thinking on or off, prefix slots.What's in it
MiniCPM_OCRAutoModel wrapper, registration, and amodel_list.jsonentry: 32K default context,vlm: true.minicpm_ocr_state_api, including image hashing.continue_final_messagechange (b4c2b90): MiniCPM-OCR supports it the same way as MiniCPM-V. Itsapply_chat_template()anddecode()are identical to MiniCPM-V's apart from the class name.src/test/minicpm_ocr_npu/:--prompt-filefor timed single turns;--follow-upand--resumefor cached second turns;--multi-slotfor interleaved conversations through the prefix slots.