Firmware sets: each board keeps its filesystem from one release to the next

.github/workflows/release-firmware.yml

@@ -289,9 +289,19 @@ jobs:
           name: firmware
           path: firmware
 
+      # The notes name every board whose filesystem this set moves, against the
+      # newest set people could download (a published release, pre-releases too).
       - name: The tag may go on a draft, and the Release's notes
         run: |
-          python3 scripts/release_firmware.py notes firmware/firmware.json --out "$RUNNER_TEMP/notes.md"
+          prev=$(gh api "repos/$GITHUB_REPOSITORY/releases?per_page=100" \
+            --jq '[.[] | select(.draft | not) | select(any(.assets[]; .name == "firmware.json"))][0].tag_name // empty')
+          previous=()
+          if [[ -n "$prev" ]]; then
+            gh release download "$prev" -p firmware.json -D "$RUNNER_TEMP/previous"
+            previous=(--previous "$RUNNER_TEMP/previous/firmware.json")
+            echo "comparing with $prev"
+          fi
+          python3 scripts/release_firmware.py notes firmware/firmware.json "${previous[@]}" --out "$RUNNER_TEMP/notes.md"
           echo "TAG=$(python3 -c 'import json; print(json.load(open("firmware/firmware.json"))["tag"])')" >> "$GITHUB_ENV"
           echo "PRERELEASE=$(python3 -c 'import json; print("--prerelease" if json.load(open("firmware/firmware.json"))["preview"] else "")')" >> "$GITHUB_ENV"
 

CONTRIBUTING.md

@@ -93,6 +93,20 @@ inside the flash image, UF2 blocks, the vector table of a GIGA's main image) and
 every file's sha256 and size, the flash offset, and the pins the set was built
 from.
 
+### Each board keeps its filesystem
+
+Every MicroPython esp32 and rp2 cell names the filesystem its boards keep from
+one release to the next (`fs`), and builds with `--keep-fs` at that size. An
+image that outgrows the flash left beside it fails its build, which says what
+size would fit, instead of shipping a set that leaves an updated board with an
+empty filesystem. `assemble` checks each filesystem is where its cell says and
+records it in `firmware.json`.
+
+When an image no longer fits, build fewer modules into it, or change the cell's
+`fs` on purpose. That moves the filesystem on that board, and the draft's notes
+say so: they compare the set with the newest published one and name each board
+whose filesystem moved.
+
 ### Build a set
 
 Actions > Release firmware > Run workflow. A pull request that touches the

build_mp.py

@@ -1520,6 +1520,102 @@ def fs_size(text):
     return n if n and n % 4096 == 0 else None
 
 
+def keep_fs_refused(keep_fs, fits):
+    """What a build with --keep-fs says when the firmware doesn't fit beside it."""
+    say(f"\nThe firmware doesn't fit beside a {keep_fs // 1024} KB filesystem."
+        + (f" A {fits // 1024} KB one would" if fits else ""))
+    say(("  fit it, but that" if fits else "  A smaller one") + " moves the filesystem and a board flashed with it comes up")
+    say("  with an empty one. (--keep-fs, so nothing was rebuilt: build fewer modules,")
+    say("  or copy the board's files off and build without --keep-fs.)")
+
+
+# A filesystem partition, by its subtype. MicroPython mounts the one named vfs
+# (or ffat); with neither, it makes one of the flash after the last partition.
+ESP32_FS_SUBTYPES = ("fat", "littlefs", "spiffs")
+
+
+def esp32_table_rows(src):
+    rows = []
+    for raw in Path(src).read_text().splitlines():
+        line = raw.strip()
+        if line and not line.startswith("#"):
+            fields = [f.strip() for f in line.split(",")]
+            rows.append((fields + [""] * 6)[:6])
+    return rows
+
+
+def esp32_bytes(v):
+    v = v.strip().lower()
+    mult = 1024 if v.endswith("k") else 1024 * 1024 if v.endswith("m") else 1
+    return int(v.rstrip("km"), 0) * mult
+
+
+def esp32_used_table(build, port_dir):
+    """The partition table a configured build in `build` used, or None."""
+    try:
+        sdk = (build / "sdkconfig").read_text()
+    except OSError:
+        return None
+    used = re.search(r'^CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="(.*)"$', sdk, re.M)
+    if not used:
+        return None
+    src = Path(used.group(1))
+    return src if src.is_absolute() else port_dir / src
+
+
+def esp32_keep_fs_layout(rows, flash_bytes, fs_bytes):
+    """(app row, named filesystem row or None, app size) for a table whose
+    filesystem keeps fs_bytes at the end of the flash. The app partition takes
+    what the partitions after it leave."""
+    apps = [i for i, r in enumerate(rows) if r[1] == "app"]
+    if len(apps) != 1:
+        die(f"--keep-fs sizes a table with one app partition; this one has {len(apps)}")
+    idx = apps[0]
+    fs = [i for i, r in enumerate(rows) if r[1] == "data" and r[2] in ESP32_FS_SUBTYPES]
+    if fs and (fs != [len(rows) - 1] or fs[0] < idx):
+        die("--keep-fs needs the filesystem partition, if the table names one, last and after the app")
+    named = fs[0] if fs else None
+    between = sum(esp32_bytes(r[4]) for i, r in enumerate(rows) if i > idx and i != named)
+    return idx, named, flash_bytes - fs_bytes - esp32_bytes(rows[idx][3]) - between
+
+
+def esp32_keep_fs(src, flash_bytes, fs_bytes, dest):
+    """The table at src with its app partition sized so the filesystem is the
+    last fs_bytes of the flash, where a board flashed earlier with the same
+    size has it."""
+    rows = esp32_table_rows(src)
+    idx, named, size = esp32_keep_fs_layout(rows, flash_bytes, fs_bytes)
+    if size < 0x10000:
+        die(f"--keep-fs: a {fs_bytes // 1024} KB filesystem leaves no room for the app in {flash_bytes // 1024 // 1024} MB")
+    rows[idx][4] = hex(size)
+    cursor = esp32_bytes(rows[idx][3]) + size
+    for i in range(idx + 1, len(rows)):
+        rows[i][3] = hex(cursor)
+        if i != named:
+            cursor += esp32_bytes(rows[i][4])
+    if named is not None:
+        rows[named][4] = hex(fs_bytes)
+    where = "" if named is not None else " (the flash after the last partition)"
+    body = ["# Name, Type, SubType, Offset, Size, Flags",
+            f"# Generated by build_mp.py from {src} for --keep-fs: '{rows[idx][0]}' sized to {hex(size)}",
+            f"# so the filesystem{where} is {fs_bytes // 1024} KB at {hex(flash_bytes - fs_bytes)}."]
+    body += [", ".join(r).rstrip(", ") for r in rows]
+    dest.write_text("\n".join(body) + "\n")
+    return dest
+
+
+def esp32_keep_fs_fits(log_text, table, flash_bytes):
+    """The largest filesystem, in whole 64 KB, that leaves room for an app the
+    size the log says it is: 0 if none does, None if the log isn't an app
+    partition overflow."""
+    m = re.search(r"app partition is too small for binary \S+ size (0x[0-9a-fA-F]+)", log_text)
+    if not m:
+        return None
+    _, _, room = esp32_keep_fs_layout(esp32_table_rows(table), flash_bytes, 0)
+    room -= (int(m.group(1), 16) + 0xFFFF) & ~0xFFFF
+    return room & ~0xFFFF if room >= 0x10000 else 0
+
+
 def rp2_autosize(log_text):
     """MICROPY_HW_FLASH_STORAGE_BYTES that leaves the firmware room to fit, from
     the linker's memory table; None if the log isn't a FLASH overflow."""
@@ -2101,7 +2197,7 @@ def main():
     ap.add_argument("--flash", help="esp32 flash size, e.g. 16MB")
     ap.add_argument("--no-autosize", action="store_true", help="refuse instead of growing the firmware's flash: esp32's app partition, rp2's firmware over its "
                     "filesystem, the Arduino GIGA's over its M4 core's flash")
-    ap.add_argument("--keep-fs", metavar="SIZE", help="rp2: build beside the filesystem a board already has, its size in bytes or KB "
+    ap.add_argument("--keep-fs", metavar="SIZE", help="rp2 and esp32: build beside the filesystem a board already has, its size in bytes or KB "
                     "(576K), and refuse instead of moving it when the firmware doesn't fit")
     ap.add_argument("--coredump", action="store_true",
                     help="esp32: a troubleshooting build that writes a core dump to flash on a crash (standard builds don't)")
@@ -2170,14 +2266,16 @@ def main():
         die("--flash is for esp32")
     if args.coredump and port != "esp32":
         die("--coredump is for esp32")
-    if args.keep_fs and port != "rp2":
-        die("--keep-fs is for rp2")
+    if args.keep_fs and port not in ("rp2", "esp32"):
+        die("--keep-fs is for rp2 and esp32")
     keep_fs = fs_size(args.keep_fs) if args.keep_fs else None
     if args.keep_fs and not keep_fs:
         die(f"--keep-fs takes a size in whole 4 KB sectors, in bytes or KB (576K), not '{args.keep_fs}'")
     if args.flash and args.flash.upper() not in FLASH_SIZES:
         die(f"--flash takes one of {', '.join(FLASH_SIZES)}")
     flash = args.flash.upper() if args.flash else None
+    if keep_fs and port == "esp32" and not flash:
+        die("--keep-fs on esp32 needs --flash: the filesystem is the end of the flash, so its place depends on the flash size")
     if port == "esp32" and flash is None and not args.port:
         # Upstream's generic tables assume 4 MB, which most module sets outgrow.
         flash = choose("Flash size (none keeps the board's own):", list(FLASH_SIZES), allow_none=True)
@@ -2310,6 +2408,10 @@ def shell(cmd, log=None):
                 die("--coredump: this board's table has no coredump partition yet, and the dump needs one. "
                     "The ESP32-S3 boards' table has it.")
             tinyusb = tinyusb_component(mp, chip)
+            flash_bytes = int(flash[:-2]) * 1024 * 1024 if flash else 0
+            kept = build / "partitions-keep-fs.csv"
+            if keep_fs and table:
+                table = esp32_keep_fs(table, flash_bytes, keep_fs, kept)
             for attempt in (1, 2):
                 text, used = esp32_fragment(chip, module_dirs, board, variant, flash, table, args.coredump)
                 frag.write_text(text)
@@ -2324,6 +2426,22 @@ def shell(cmd, log=None):
                         say(f"  fragment: {u}")
                     shell(make_esp + ["submodules"])
                 rc, out = shell(make_esp, log=True)
+                if keep_fs and table != kept:
+                    # The board's own table, found only once a build has
+                    # configured: size it, and build against that.
+                    src = esp32_used_table(build, port_dir)
+                    if not src and rc:
+                        break
+                    if not src:
+                        die("--keep-fs: this build didn't say which partition table it used, so there is none to size")
+                    table = esp32_keep_fs(src, flash_bytes, keep_fs, kept)
+                    say(f"\n--keep-fs: building once more against {table}, sized from the board's {src.name}.\n")
+                    continue
+                if keep_fs:
+                    fits = esp32_keep_fs_fits(out, table, flash_bytes) if rc else None
+                    if fits is not None:
+                        keep_fs_refused(keep_fs, fits)
+                    break
                 if rc == 0 or attempt == 2:
                     break
                 grown = esp32_autosize(out, build, port_dir, build / "partitions.csv")
@@ -2377,10 +2495,7 @@ def shell(cmd, log=None):
                 if not storage:
                     break
                 if keep_fs:
-                    say(f"\nThe firmware doesn't fit beside a {keep_fs // 1024} KB filesystem. A {storage // 1024} KB one would")
-                    say("  fit it, but that moves the filesystem and a board flashed with it comes up")
-                    say("  with an empty one. (--keep-fs, so nothing was rebuilt: build fewer modules,")
-                    say("  or copy the board's files off and build without --keep-fs.)")
+                    keep_fs_refused(keep_fs, storage)
                     break
                 if args.no_autosize:
                     say(f"\nThe firmware doesn't fit. MICROPY_HW_FLASH_STORAGE_BYTES={storage} would fit it.")

docs/guides/add-a-board-variant.md

@@ -40,7 +40,8 @@ The partition table is found the same way, the most specific `partitions.csv`
 winning ([`variants/esp32/partitions.esp32s3.csv`](https://github.com/PyDevices/micropython-pydevices/blob/main/variants/esp32/partitions.esp32s3.csv)
 adds a coredump partition on the S3, which a standard build leaves unused).
 Without one, the board's own is used. Either way, autosize grows the app
-partition when the image doesn't fit.
+partition when the image doesn't fit, unless the build keeps a board's
+filesystem with `--keep-fs`.
 
 Run the CPU, flash and PSRAM at the fastest clocks the board's chips are
 rated for. The CPU's rating is the chip's: 240 MHz for an ESP32, S2, S3 or C5,

docs/guides/fit-firmware-on-a-small-flash.md

@@ -16,28 +16,42 @@ To refuse the growth and stop instead, add `--no-autosize`.
 
 ## Keep a board's files through an update
 
-On rp2, autosize works the filesystem's size out again in each new build
-folder, so a newer image that is a few KB bigger can come out with a smaller
-filesystem than the one already on your board, and flashing it empties the
-board. To keep the board's filesystem where it is, ask the board its size:
+Autosize works the firmware's share out again in each new build folder, so a
+newer image that is a few KB bigger can come out with a smaller filesystem
+than the one already on your board, and flashing it empties the board. To keep
+the board's filesystem where it is, ask the board its size. On rp2:
 
 ```python
 import rp2
 f = rp2.Flash()
 print(f.ioctl(4, 0) * f.ioctl(5, 0))  # 589824 is 576 KB
 ```
 
+On esp32:
+
+```python
+import esp32
+print(esp32.Partition.find(esp32.Partition.TYPE_DATA, label="vfs")[0].info()[3])  # 2752512 is 2688 KB
+```
+
 and build with that size:
 
 ```bash
 ./build_mp.py --port rp2 --board RPI_PICO --modules displayif,lvgl,pydevices --keep-fs 576K
+./build_mp.py --port esp32 --board ESP32_GENERIC_S3 --variant LCD_7 --flash 8MB --modules all --keep-fs 2688K
 ```
 
 If the firmware fits beside it, the image keeps that filesystem and your files
 are there after the flash. If it doesn't, the build stops and says what size
-would fit, rather than moving it. A rebuild in the same build folder starts
-from the size that folder used last, so it keeps the filesystem without
-`--keep-fs` unless the image has outgrown it.
+would fit, rather than moving it. On esp32 the filesystem is the end of the
+flash, so `--keep-fs` needs `--flash`, and the app partition takes the flash
+before it. On rp2, a rebuild in the same build folder starts from the size that
+folder used last, so it keeps the filesystem without `--keep-fs` unless the
+image has outgrown it.
+
+The firmware page's MicroPython images are built this way, so updating a board
+from one release to the next keeps its files. A release that has to move a
+board's filesystem says so at the top of its notes.
 
 On esp32, autosize works only if the flash size is right: the partition table
 tool refuses a table larger than the configured flash, so give `--flash`.
@@ -60,7 +74,7 @@ compile runs twice.
 ## When nothing will fit
 
 How much fits depends on the flash. On a Pico 2 (4 MB), `all` builds and fits,
-using 93 % of its 2560 KB firmware region and leaving a 1536 KB filesystem. On
+using 94 % of its 3392 KB firmware region and leaving a 704 KB filesystem. On
 a Pico (2 MB) it can't: the image is bigger than the whole flash, so the build
 stops at the first link with the overflow error. Name what your project uses instead:
 

docs/reference/build-command.md

@@ -3,7 +3,7 @@
 ```text
 ./build_mp.py [--interpreter {micropython,circuitpython}] [--port PORT]
               [--board BOARD] [--variant VARIANT] [--modules MODULES]
-              [--flash FLASH] [--no-autosize] [--coredump] [--clean]
+              [--flash FLASH] [--no-autosize] [--keep-fs SIZE] [--coredump] [--clean]
               [make arguments]
 ```
 
@@ -22,6 +22,7 @@ optional. Leave out `--modules` and it lists [`modules/`](https://github.com/PyD
 | `--modules` | A comma list of short names, full paths, or `all`. See [Modules](modules.md). |
 | `--flash` | esp32 flash size, such as `16MB`. |
 | `--no-autosize` | esp32, rp2 and the Arduino GIGA: refuse instead of growing the firmware's share of flash. |
+| `--keep-fs` | esp32 and rp2: build beside the filesystem a board already has, its size in bytes or KB (`576K`), and stop instead of moving it. See [Keep a board's files through an update](../guides/fit-firmware-on-a-small-flash.md#keep-a-boards-files-through-an-update). |
 | `--coredump` | esp32: a troubleshooting build that writes a core dump to flash on a crash. Standard builds don't. |
 | `--clean` | Delete this target's build directory first. |