Repository navigation
h264enc pipelining; espdl releases the GIL and gains HumanFaceDetect - #325
Conversation
…ash; P4X_EYE gets a 9 MB app slot
A plain Encoder runs the PPA's RGB565-to-YUV420 conversion and the H.264 encode one after the other, so a 1920x1072 frame costs about 40 + 38 ms (12 fps). The two are separate hardware engines. With pipeline=True, encode() converts frame N on the caller's thread while a worker task encodes frame N-1, and returns frame N-1's access unit (b"" for the first); flush() and close() return the last one. A frame then costs about the longer of the two. stats() gains wall_us, how long encode() kept its caller. The default stays the plain encoder, since the one-frame latency changes what encode() returns. The C API (castif's cast task) is unchanged. Built for ESP32_GENERIC_P4 P4X_EYE with h264enc, ppa, cameraif, vision; the hardware test now also checks the pipelined stream against the plain one byte for byte. Not yet run on a board.
espdl's detect(), classify() and predict() held the GIL for the whole model run (70 to 80 ms for a 240x240 frame on the ESP32-P4), stopping every other thread. The model run now happens without it, so detection on a _thread leaves the main loop running; examples/espdl_thread.py measures the main loop's longest pause during detection. espdl.HumanFaceDetect(msr_path, mnp_path) runs ESP-DL's two-stage human_face_detect models (MIT; release 0.5.0, the one for the pinned esp-dl 3.3.6) and returns (x, y, w, h, score, keypoints) with ten keypoint coordinates; the whole detection runs without the GIL. examples/espdl_face_landmarks.py marks faces in a photo. esp-vision's py_espdl.cpp stays as pinned: the two changes are patch files in modules/vision/esp-vision-patches, applied to a copy in the build tree. Built for ESP32_GENERIC_P4 P4X_EYE; not yet run on a board.
|
Run on the ESP32-P4X-EYE (OV2710 at 1920x1080, frames borrowed with
At 720p the pipeline halves the caller's wait (34 to 18 ms) and the sensor's 25 fps is the ceiling. At 1080p the PPA conversion is the limit and gets slower when it overlaps the encode (41 to 67 ms), so the pipeline buys one frame a second there. The README's "PPA is the limit" note is right; a YUV420 frame straight from the ISP is the next step for 1080p. Internal RAM: with the board's display (DMA
|
…d apps is 8.5 MB)
Three changes for camera apps on the ESP32-P4.
h264enc.Encoder(..., pipeline=True)overlaps the PPA's colour conversion of one frame with the H.264 encode of the frame before. A plain encoder runs them back to back, so a 1920x1072 camera frame costs about 40 ms to convert plus 38 ms to encode, 12 fps; pipelined, a frame should cost about the longer of the two.encode()then returns the previous frame's access unit (b""first) andflush()orclose()the last one. It is off by default, because the one-frame latency changes whatencode()returns.stats()gainswall_us. The C API that castif uses is unchanged. The README notes that the PPA does one operation at a time, so a preview's ownppa.srm()each frame adds to the conversion time.espdlnow releases the GIL while a model runs, so detection on a_threadno longer stops the main loop (and LVGL) for 70 to 80 ms per frame.espdl.HumanFaceDetect(msr_path, mnp_path)adds ESP-DL's two-stage face detector, returning(x, y, w, h, score, keypoints)with five landmark points; its model files are MIT, from ESP-DL'shuman_face_detect0.5.0, and are copied to the board like the other espdl models. esp-vision'spy_espdl.cppstays as pinned: both changes are patch files inmodules/vision/esp-vision-patches, applied to a copy in the build tree.Checked: the firmware builds for
ESP32_GENERIC_P4P4X_EYEwithh264enc,ppa,cameraif,vision,pydevices, the patched copy carries the six GIL releases and the new class, and the code was read against esp_h264 1.4.1 and esp-dl 3.3.6. Not verified yet: nothing here has run on a board. The pipelined frame rate, the byte-for-byte match the hardware test checks, the main loop's pause inespdl_thread.py, and the face detector's output all wait on a board run, and the two new examples have noTested on:line until then. This branch sits on the P4X_EYE variant branch, so the diff against main includes those commits.