h264enc pipelining; espdl releases the GIL and gains HumanFaceDetect

build_mp.py

@@ -1293,6 +1293,19 @@ def num(v):
         cursor += num(r[4])
     after = rows[idx + 1:]
     moved = (", ".join(f"'{r[0]}'" for r in after) + " moved to follow it. ") if after else ""
+    # A table that names its filesystem gives it a size, and growing the app
+    # pushes that partition past the end of the flash. The filesystem is the
+    # part that can shrink: cut the last data partition to what is left.
+    flash_bytes = int(flash.group(1)) * 1024 * 1024 if flash else 0
+    if after and flash_bytes and cursor > flash_bytes:
+        last = after[-1]
+        if last[1] == "data" and last[2] in ("fat", "littlefs", "spiffs"):
+            room = flash_bytes - num(last[3])
+            if room < 0x10000:
+                die(f"autosize: the app grew to {hex(size)} and leaves no room for '{last[0]}' in {flash.group(1)} MB")
+            last[4] = hex(room)
+            cursor = flash_bytes
+            moved += f"'{last[0]}' cut to {hex(room)} to fit the flash. "
     if any(r[1] == "data" and r[2] in ("fat", "littlefs", "spiffs") for r in after):
         fs = ["# The filesystem's partition moved with them."]
     else:

docs/guides/use-esp-vision-modules.md

@@ -78,6 +78,38 @@ These ran on an ESP32-P4 board, each with an example. The arena is what
 
 Each example's docstring says where to download its model.
 
+## Detect in a thread
+
+`espdl` lets go of the interpreter while a model runs, so detection on a
+`_thread` leaves the main thread free: a display, a UI or a recording loop
+keeps running through each `detect()`, `classify()` or `predict()`. Only the
+image preparation before the model and the results after it hold the
+interpreter. Use each model object from one thread at a time, and don't call
+its `deinit()` while it is detecting.
+[`espdl_thread.py`](https://github.com/PyDevices/micropython-pydevices/blob/main/modules/vision/examples/espdl_thread.py)
+detects on a thread and measures the main loop's longest pause meanwhile.
+
+## Faces with landmarks
+
+`espdl.HumanFaceDetect(msr_path, mnp_path, *, score=0.5, nms=0.5, mnp_score=0.5, mnp_nms=0.5)`
+runs ESP-DL's two-stage face detector: MSR finds candidates in the whole
+picture, then MNP looks at each one closely and places five key points.
+`detect(img)` takes an RGB565 `image.Image` and returns a list of
+`(x, y, w, h, score, keypoints)`, where `keypoints` is ten ints, five (x, y)
+pairs in ESP-DL's order: left eye, left mouth corner, nose, right eye, right
+mouth corner. `set_thresholds()` takes the same four keywords as the
+constructor, and the whole detection runs without the interpreter's lock.
+
+The two model files come from ESP-DL's
+[`models/human_face_detect`](https://github.com/espressif/esp-dl/tree/master/models/human_face_detect),
+in its `models/p4` or `models/s3` folder: `human_face_detect_msr_s8_v1.espdl`
+and `human_face_detect_mnp_s8_v1.espdl`. They are MIT-licensed, by Espressif;
+the firmware doesn't carry them, so copy them to the board's filesystem and pass
+their paths. Release 0.5.0 of the `espressif/human_face_detect` component is
+the one for the ESP-DL this firmware is built with.
+[`espdl_face_landmarks.py`](https://github.com/PyDevices/micropython-pydevices/blob/main/modules/vision/examples/espdl_face_landmarks.py)
+marks the faces in a photo.
+
 ## Run long scripts as `main.py`
 
 An mpftp `exec` that outlasts mpftp's wait resets the board over USB

docs/how-it-works/esp-vision.md

@@ -14,7 +14,7 @@ The leading dot keeps it out of `--modules` and `all`, because esp-vision's own
 [`modules/tflite`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/tflite) and [`modules/vision`](https://github.com/PyDevices/micropython-pydevices/tree/main/modules/vision) are our glue around esp-vision's files:
 the ESP-IDF components esp-vision uses, at its versions; the include paths and
 definitions MicroPython's QSTR scan needs (it sees only a module's own paths,
-never a linked component's); and four pieces of our own.
+never a linked component's); and five pieces of our own.
 
 - **imlib compiles inside the module**, not as an ESP-IDF component. As a
   component it would wait for MicroPython's generated headers while being one of
@@ -32,6 +32,12 @@ never a linked component's); and four pieces of our own.
   those models use is in esp-vision's list already.
 - **`src/vision_preview_none.c`** stands in for esp-vision's preview, which
   streams frames to its IDE over its own USB multiplexer.
+- **`esp-vision-patches/`** holds two small changes to esp-vision's
+  `py_espdl.cpp`: `espdl` releases the GIL while a model runs, and its module
+  table lists `HumanFaceDetect` from **`src/vision_espdl_face.cpp`**. The
+  build applies them, in order, to a copy of the file in the build tree, so
+  the pinned checkout stays as it is, and a pin move that breaks one stops the
+  build with the patch's name.
 - **`src/vision_h264.c`** is `h264`'s encoder, in place of esp-vision's
   `platform/h264.c`. It is that file with one addition: the encoder in a P4
   before revision 3 reads only packed YUV420 and refuses RGB565, so firmware

modules/.esp-vision/ESP_VISION.lock

@@ -1,5 +1,7 @@
 # esp-vision (espressif/esp-vision, Apache-2.0): the source of our tflite
 # module and, later, its image stack. Pinned by full commit; fetch.sh puts it
 # at modules/.esp-vision/src. Its files are compiled as they are: our glue
-# lives in the module directories that use it (modules/tflite).
+# lives in the module directories that use it (modules/tflite, modules/vision),
+# and modules/vision/esp-vision-patches changes py_espdl.cpp in a build-tree
+# copy, never here.
 esp-vision  https://github.com/espressif/esp-vision.git  ecda8004b95e1df275ac2a1f9cdf487d03defc63

modules/h264enc/README.md

@@ -26,7 +26,7 @@ take it too (`--interpreter circuitpython --modules h264enc,ppa`), and
 
 ## API
 
-### `h264enc.Encoder(width, height, fps=30, gop=30, bitrate=3_000_000, qp_min=10, qp_max=45, *, canvas_w=0, canvas_h=0, out_size=0)`
+### `h264enc.Encoder(width, height, fps=30, gop=30, bitrate=3_000_000, qp_min=10, qp_max=45, *, canvas_w=0, canvas_h=0, out_size=0, pipeline=False)`
 
 `width` and `height` are multiples of 16, from 80x80 to 1920x2032. `gop` is
 the frames from one IDR (key) frame to the next. With `canvas_w` / `canvas_h`
@@ -35,6 +35,8 @@ puts a 720x720 panel on a 1280x720 TV. `out_size` caps one access unit. The
 default holds a keyframe of the whole canvas at its raw size (1.4 MB at
 1280x720, in PSRAM), because a busy picture such as a camera's can encode
 bigger than any fixed guess, and a frame that doesn't fit stops the encoder.
+`pipeline=True` overlaps the colour conversion of one frame with the encode
+of the one before; see [Pipelined encoding](#pipelined-encoding).
 
 The P4 has one H.264 encoder, so one `Encoder` (or one cast) at a time: a
 second raises `OSError` saying it is busy, and the first carries on. `close()`
@@ -44,14 +46,60 @@ gives the encoder back; so does the object being collected, and `with` works.
 
 One frame of native-order RGB565 (what displaydev framebuffers hold), at least
 `width * height * 2` bytes. Returns one access unit: Annex B NAL units, SPS
-and PPS included before each IDR.
+and PPS included before each IDR. A pipelined encoder returns the previous
+frame's access unit instead, and `b""` for the first frame.
+
+### `flush() -> bytes`
+
+A pipelined encoder's last access unit, the one `encode()` hasn't returned
+yet, or `b""` when there is none. A plain encoder always returns `b""`.
 
 ### `keyframe`, `force_idr()`, `set_bitrate(bps)`, `stats()`
 
-`keyframe` says whether the last `encode()` made an IDR frame. `force_idr()`
-makes the next one an IDR (a viewer joining late needs one). `set_bitrate()`
-takes effect from the next frame. `stats()` is a dict: `frames`, `keyframes`,
-`bytes`, `errors`, and the last frame's `ppa_us` and `enc_us`.
+`keyframe` says whether the access unit `encode()` (or `flush()`) last
+returned is an IDR frame. `force_idr()` makes the next frame the encoder
+starts an IDR (a viewer joining late needs one). `set_bitrate()` takes effect
+from the next frame. `stats()` is a dict: `frames` and `keyframes` returned so
+far, `bytes`, `errors`, the last conversion's `ppa_us` and the last encode's
+`enc_us`, and `wall_us`, how long the last `encode()` kept its caller.
+`pipeline` says which kind of encoder this is. `close()` returns what
+`flush()` would have, or `None` for a plain encoder; leaving a `with` block
+discards it, so flush first.
+
+## Pipelined encoding
+
+The colour conversion runs on the PPA and the encode on the H.264 engine, two
+separate pieces of hardware. A plain encoder runs them one after the other, so
+a frame costs their sum: at 1920x1072 about 40 ms to convert and 38 ms to
+encode, 12 frames a second. A pipelined encoder converts frame N on the PPA
+while frame N-1 is still being encoded, so a frame costs about the longer of
+the two.
+
+```python
+enc = h264enc.Encoder(1920, 1072, fps=25, gop=25, pipeline=True)
+with open("clip.h264", "wb") as f:
+    for frame in frames:
+        f.write(enc.encode(frame))     # the previous frame's access unit
+    f.write(enc.flush())               # and the last one
+enc.close()
+```
+
+The price is one frame of latency: each `encode()` returns the frame before.
+`encode()` has finished reading `frame` when it returns, so you can capture the
+next picture into the same buffer straight away. The encoder keeps two YUV
+canvases in PSRAM rather than one (3 MB each at 1920x1088) and starts one
+small task the first time a pipelined encoder opens, and keeps it.
+
+It's off by default because the latency changes what `encode()` returns, and
+code that pairs each frame with its own access unit (writing timestamps, or
+checking `keyframe` straight after) would quietly be off by one.
+
+The PPA does one operation at a time. Any other `ppa.srm()` call, such as a
+preview cropping and scaling each camera frame for a screen, waits while the
+encoder's conversion runs, and the conversion waits for it. A preview's PPA time
+adds to the conversion's, so with a preview running at 1080p the PPA, not
+the encoder, sets the frame rate. `stats()["ppa_us"]` and your own preview's
+timing show how close the two together come to the frame period.
 
 ## Colour