WIP: Ffmpeg decoder, encoders and filters. - #10990
Draft
lgirdwood wants to merge 79 commits into
Draft
Conversation
Add a loadable (LLEXT) audio module that wraps FFmpeg. Two build modes: - decoder (default): a compressed elementary stream (FLAC/AAC/Opus, selected via Kconfig) is parsed and decoded to PCM (process_raw_data); - filter (CONFIG_FFMPEG_DEC_FILTER_MODE): a PCM source/sink effect that runs an FFmpeg audio filter graph (afftdn noise reduction). FFmpeg is a west-pinned source (see west.yml) cross-built by ffmpeg.cmake as a CMake ExternalProject, enabling only the Kconfig-selected decoders and filters. The module supplies the libc surface FFmpeg needs that the SOF core does not export to LLEXT (ffmpeg_dec-shims.c), a SOF-heap-backed malloc (ffmpeg_dec-alloc.c), fast single-precision float math (fastmathf.c), and routes av_log() into the Zephyr log. The avfilter graph backend and the PCM effect ops are in ffmpeg_dec-filter.c. Also adds the rimage module manifest (ffmpeg_dec.toml + per-platform includes), the topology widget (ffmpeg_dec.conf), host test tooling (ffmpeg_dec_prepare.sh) and docs (README/TESTING). Verified building and load-ready (all symbols resolved, signed) for ace30 (ptl) with the Zephyr SDK: FLAC and FLAC+AAC decoders, and the afftdn filter effect. Runtime decode/denoise verification is pending hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the ffmpeg_dec component to the host->component->host benchmark harness so a topology instantiating the module can be built and driven (testbench or target). Adds the per-format bench configs (generated with bench_comp_generate.sh), registers the widget include and the ffmpeg_dec32 BENCH_CONFIG in cavs-benchmark-hda.conf, and adds ffmpeg_dec to the s32 component list in tplg-targets-bench.cmake (the filter effect path processes S32). Also drop the codec-setup bytes control from the ffmpeg_dec widget class so it is a plain 1-in/1-out effect; the decoder's STREAMINFO control is added at the instance level in a decoder topology instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CONFIG_FFMPEG_DEC_MP3: enables the libavcodec mp3 decoder and mpegaudio parser in the cross-build, the FFMPEG_DEC_CODEC_MP3 codec id mapping, and the float math layer (the mp3 decoder uses it). Verified building for ace30 (ptl) with decoders [flac,mp3]. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FFmpeg has no native MP3 encoder, so add MP3 encode through libshine, a small fixed-point encoder (good fit for a DSP - no float dependency). libshine is a west-pinned project; ffmpeg.cmake cross-builds it (its lib needs no config.h and its autotools CLI/shared link cannot work bare-metal, so the objects are compiled and archived directly), generates a shine.pc, and enables --enable-libshine --enable-encoder=libshine in FFmpeg. FFmpeg's require_pkg_config link test for -lshine pulls newlib malloc and hence Zephyr-runtime symbols (z_errno_wrap, ...) that only exist at module load; a configure-test-only stub is passed via --extra-ldflags so the test links (--extra-ldflags does not affect the static-archive build). Gated by CONFIG_FFMPEG_ENC_MP3. Verified building for ace30 (ptl): libshine.a cross-builds and libavcodec.a contains the libshine encoder. A module encode path (PCM->MP3) is a separate build mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add CONFIG_FFMPEG_DEC_ENCODE_MODE: build the module as an encoder using the modern .process_raw_data path in reverse of the decoder - avcodec_send_frame/avcodec_receive_packet. ffmpeg_dec-encode.c opens the libshine MP3 encoder, converts SOF interleaved S32 to the encoder sample format per frame, and emits the compressed elementary stream. The interface ladder in ffmpeg_dec.c selects encoder / filter / decoder ops by Kconfig. libshine is linked into the module (after libavcodec, which references it) from its own cross-build install dir. ffmpeg.cmake now allows an encoder-only build (no decoder) and makes the decoder configure flags conditional. The encoder path pulls a few more libc symbols, added to the local surface: calloc/posix_memalign (alloc), modf/localtime_r/iconv*/__xpg_strerror_r (shims). fastmathf is now always built for the real backend (its sofm_log2f backs the shims' log10()). Verified building and load-ready (no unresolved externals, signed) for ace30 (ptl) as an encoder-only module. Runtime encode + real-time framing need hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add HIFI.md: analysis of the hot audio-processing paths in the module (decode/ filter/encode) and the linked FFmpeg DSP, and where Xtensa HiFi intrinsics would help. FFmpeg is built --disable-asm (scalar C on Xtensa) and has no _xtensa DSP init, so the generic C kernels run. Documents: the module's own PCM conversion loops (easy HiFi wins we own), FFmpeg's DSP dispatch contexts (float_dsp, flacdsp, mpegaudiodsp, tx) as fork-patch targets mirroring the other arch inits, fastmathf vectorisation, and a cost/benefit priority order. References SOF's existing HiFi kernels (src/math/*_hifi*) as the template. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the webrtc_vad SOF module wrapping libfvad — a standalone C port of the WebRTC Gaussian mixture model VAD (identical to the algorithm shipping in Chrome/WebRTC since 2011). Module characteristics: - Single-source, single-sink PCM pass-through - Accumulates 10/20/30 ms frames (configurable via Kconfig) - Broadcasts NOTIFIER_ID_VAD event on every frame classification - Supports S16_LE and S32_LE at 8/16/32/48 kHz, mono or stereo - Fixed-point; no FPU required - Two backends: real (libfvad) and pass-through stub for CI/LLEXT Build: - libfvad is cross-compiled by src/audio/webrtc_vad/webrtc_vad.cmake - Source is fetched via west (modules/audio/libfvad, pinned SHA 532ab666c20d) - LLEXT packaging supported via llext/ subdirectory Add NOTIFIER_ID_VAD to the notifier_id enum so downstream components (e.g. webrtc_ns2 / RNNoise) can share the same VAD event channel. UUID: c790b11d-5d14-e54e-be36ba4ad732cc14 Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the webrtc_ns SOF module wrapping the classic WebRTC Noise Suppression algorithm (spectral Wiener filter, fixed-point) from the webrtc-audio-processing 0.3.1 library. Module characteristics: - Single-source, single-sink PCM effect - Fixed-point implementation; no FPU required - Operates at 8 or 16 kHz on 10 ms frames (160/80 samples) - Supports S16_LE and S32_LE, mono or stereo (per-channel instances) - Four suppression levels: Mild/Medium/Aggressive/VeryAggressive (configurable at build time or via IPC4 set_configuration) - Two backends: real (WebRTC NS) and pass-through stub for CI/LLEXT - Designed as a build-time alternative to webrtc_ns2 (RNNoise): lower CPU, no 48 kHz constraint, fully fixed-point Build: - WebRTC NS subset is cross-compiled by src/audio/webrtc_ns/webrtc_ns.cmake (6 C files from modules/audio/webrtc-apm) - Source fetched via west (modules/audio/webrtc-apm, tag v0.3.1) - LLEXT packaging supported UUID: 0fc8faef-945f-004b-8d5f315047f1136a Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the webrtc_aec SOF module wrapping WebRTC AECm (Acoustic Echo Canceller Mobile) — the fixed-point Q15 echo canceller designed for mobile and embedded devices. Module characteristics: - Dual-source (2 input pins): pin 0 = microphone capture, pin 1 = playback echo reference - Single-sink: echo-cancelled microphone output - Fixed-point Q15; no FPU required - Operates at 8 or 16 kHz on 10 ms frames - Supports S16_LE and S32_LE, up to WEBRTC_AEC_CHANNELS_MAX channels (one AecmCore handle per channel) - Pipeline-ID heuristic (matching google_rtc_audio_processing) used to distinguish mic vs. reference at prepare() time - Two backends: real (AECm) and pass-through stub for CI/LLEXT Pin binding follows the same pattern as google-rtc-aec: both DAI capture copiers are connected as sources; the firmware resolves mic-vs-ref from pipeline membership. Reference topology: tools/topology/topology2/development/ cavs-nocodec-webrtc-aec.conf (dual SSP: SSP0=mic, SSP2=ref) UUID: e5da7b5b-133a-ba46-b517651d6300bb83 Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the webrtc_ns2 SOF module wrapping xiph/rnnoise — a recurrent neural network (GRU-based) noise suppressor originally developed by Jean-Marc Valin at Mozilla. Module characteristics: - Single-source, single-sink PCM effect - Floating-point; internal float math, hardware FPU beneficial - Hard requirement: pipeline sample rate MUST be 48000 Hz - Processes 480-sample (10 ms) frames; partial periods accumulated internally - Supports S16_LE and S32_LE, mono or stereo (one DenoiseState per channel; up to WEBRTC_NS2_CHANNELS_MAX channels) - Float scale bridging: pipeline samples normalised to ±1.0 are scaled to RNNoise's native ±32768 range and back - VAD dual-use: rnnoise_process_frame() returns per-frame speech probability [0.0, 1.0]; this fires NOTIFIER_ID_VAD events at a configurable threshold (WEBRTC_NS2_VAD_THRESHOLD_PCT) - Two backends: real (RNNoise) and pass-through stub for CI/LLEXT Design notes: - rnnoise_init() used at prepare/reset instead of rnnoise_create() to avoid per-init heap allocation (embedded-friendly path) - Model weights (rnn_data.c) are const float[] placed in .rodata (Flash/ROM); ~85-340 KB depending on model variant - Peak stack ~12-16 KB per rnnoise_process_frame() call - No expf/tanhf in hot path: replaced by 201-entry tansig LUT Build: - RNNoise cross-compiled by src/audio/webrtc_ns2/webrtc_ns2.cmake (6 C files: denoise.c rnn.c rnn_data.c pitch.c celt_lpc.c kiss_fft.c) - Source fetched via west (modules/audio/rnnoise, SHA 70f1d256) - LLEXT packaging supported UUID: eacfacdc-2a87-c942-97e894c917a740db Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Register all four WebRTC audio processing modules in the SOF build
system:
Kconfig (src/audio/Kconfig):
- rsource the four module Kconfig files (alphabetical order)
CMakeLists (src/audio/CMakeLists.txt):
- add_subdirectory() guards for COMP_WEBRTC_{VAD,NS,AEC,NS2}
UUID registry (uuid-registry.txt):
- c790b11d-5d14-e54e-be36ba4ad732cc14 webrtc_vad
- 0fc8faef-945f-004b-8d5f315047f1136a webrtc_ns
- e5da7b5b-133a-ba46-b517651d6300bb83 webrtc_aec
- eacfacdc-2a87-c942-97e894c917a740db webrtc_ns2
west.yml:
- Add remotes: libfvad (dpirch), webrtc-apm (freedesktop.org),
rnnoise (xiph)
- Add project entries with pinned revisions:
libfvad @ 532ab666c20d → modules/audio/libfvad
webrtc-apm v0.3.1 → modules/audio/webrtc-apm
rnnoise @ 70f1d256 → modules/audio/rnnoise
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add Topology 2.0 component widgets, capture pipeline templates, and
standalone development/test topologies for all four WebRTC modules.
Component widgets (include/components/):
webrtc-vad.conf — pass-through VAD; S16/S32; any rate; 1→1
webrtc-ns.conf — fixed-point spectral NS; 8/16 kHz; S16/S32; 1→1
webrtc-aec.conf — AECm echo canceller; 10ms IBS/OBS; 2 input pins
(pin 0 = mic, pin 1 = echo ref); S16/S32; 2→1
webrtc-ns2.conf — RNNoise GRU NS; 48 kHz only; S16/S32; 1→1
Capture pipeline templates (include/pipelines/cavs/):
webrtc-ns-capture.conf — DAI-copier → VAD → NS → module-copier
webrtc-aec-capture.conf — copier → AEC(2-pin) → copier
webrtc-ns2-capture.conf — RNNoise → module-copier (48 kHz locked)
Development/test topologies (development/):
cavs-nocodec-webrtc-ns.conf — single SSP loopback; VAD+NS
cavs-nocodec-webrtc-aec.conf — dual SSP (SSP0=mic, SSP2=ref); AEC
mirrors cavs-nocodec-rtcaec.conf
cavs-nocodec-webrtc-ns2.conf — single 48 kHz SSP; RNNoise
All three development topologies registered in tplg-targets.cmake
targeting TGL nocodec with NHLT preprocessing.
Compile example:
alsatplg -c development/cavs-nocodec-webrtc-aec.conf -D PLATFORM=tgl -o sof-tgl-nocodec-webrtc-aec.tplg
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the root Kconfig that exposes the ffmpeg_dec module configuration tree (codec selection, filter mode, cold-split experimental option) and the standalone ffmpeg.cmake cross-build helper for out-of-tree builds. These files were omitted from the earlier ffmpeg_dec commits and are grouped here to keep all ffmpeg_dec build system pieces together. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…and webrtc modules Create and update README.md documentation for the FFmpeg decoder/encoder/filter module and all four WebRTC audio modules: - ffmpeg_dec/README.md: Added Mermaid architecture diagram, detailed decoder/encoder/filter configurations, build choices, and topology guides. - webrtc_vad/README.md: Included Mermaid data flow, GMM classification details, ring-buffer accumulation behavior, and notification topology mappings. - webrtc_ns/README.md: Documented fixed-point spectral Wiener filter complexity, channel separation, and Kconfig rules. - webrtc_aec/README.md: Described dual-input AECm echo cancellation routing, pipeline-ID heuristics, and layout designs. - webrtc_ns2/README.md: Documented deep-learning/RNNoise GRU topology, 48 kHz lock rules, VAD probability thresholds, and DSP scaling metrics. Each document contains a tailored Mermaid diagram mapping data/control flows. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Document the per-frame AAC-LC decode hot spots for Xtensa HiFi work: IMDCT (libavutil/tx), the AVFloatDSPContext vector kernels (vector_fmul_window and friends), x^(4/3) inverse quant, and TNS, plus SBR/PS for HE-AAC. Includes a toolchain-requirement note: the HiFi intrinsic headers (xt_hifi*.h) and an ace30 core-isa.h with XCHAL_HAVE_HIFI4=1 are only available via the LLVM/xt-clang Xtensa toolchain, not the Zephyr-SDK GCC currently used by ffmpeg.cmake, so the ff_*_init_xtensa kernels need an intrinsic path plus a scalar C fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…loat_dsp
Activate the Xtensa HiFi AVFloatDSPContext kernels (lgirdwood/FFmpeg sof-hifi,
ff_float_dsp_init_xtensa) in the ffmpeg_dec module build:
- ffmpeg.cmake: stop passing --disable-asm to FFmpeg configure. FFmpeg treats
every per-arch optimisation dir -- including our C-intrinsic libavutil/xtensa/
-- as "asm"; --disable-asm forces arch=c and drops them (ARCH_XTENSA=0). There
is no Xtensa assembly, so leaving asm enabled is safe and is what makes
ARCH_XTENSA=1 and builds ff_float_dsp_init_xtensa.
- west.yml: bump ffmpeg to sof-hifi @ a600a78acd (adds libavutil/xtensa HiFi
float_dsp on top of the av_tx table cap).
Verified: pristine ptl (intel_adsp/ace30/ptl) AAC build (FLAC+AAC) reconfigures
FFmpeg with ARCH_XTENSA=1, compiles libavutil/xtensa/float_dsp_init.o, links it
into ffmpeg_dec.llext with no errors or dangerous relocations. Under the
Zephyr-SDK GCC the kernels run as the scalar C fallback (GCC lacks xt_hifi4.h);
the xtfloatx2 SIMD path activates with an xt-clang/XCC build for the ace30 core.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pick up the __XCC__ gate on the Xtensa float_dsp SIMD path so the ffmpeg_dec module builds cleanly with the LLVM Xtensa clang (was: intrinsic branch entered via the xt_hifi4.h wrapper and failed on undeclared xtfloatx2). GCC behaviour is unchanged (scalar fallback either way). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a clang branch to the FFmpeg cross-build so the module can be built with the
LLVM Xtensa clang (needed for any HiFi intrinsics -- the Zephyr-SDK GCC ships no
xt_hifi/ae_* intrinsics header at all). clang is a generic driver, so unlike the
target-specific GCC it needs the target/core/sysroot spelled out, uses the
Zephyr-SDK GNU binutils (ar/as/nm/objcopy), and cannot link the bare-metal
configure *test* executables (no default crt), so those link via the GNU gcc
(--ld). All values are derived from CMAKE_AR (GNU binutils even under clang):
cross-prefix, sysroot, and -mcpu=<core> from the triple. Also disables auto/SLP
vectorisation: the LLVM Xtensa backend cannot lower the v2i32 bswap FFmpeg
byteswap code gets SLP-vectorised into ("Cannot select: v2i32 = bswap"), and
it buys nothing (no packed float SIMD codegen on Xtensa anyway).
The GCC path is unchanged (guarded on CMAKE_C_COMPILER_ID) and re-verified green:
pristine ptl AAC build -> ARCH_XTENSA=1, float_dsp_init.o compiled, ffmpeg_dec.llext
linked. The clang recipe is validated at the FFmpeg-configure/build level: a
standalone decoder-only cross-build (avcodec+avutil+swresample, FLAC) with
~/work/llvm-project clang links clean and produces elf32-xtensa-le objects. A full
firmware clang build additionally requires the Zephyr LLVM toolchain variant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Setting the bytes control max inside the host-gateway-tdfb-drc-capture pipeline class had no effect: alsatplg leaves the widget control at the default 1024, so pushing a larger TDFB blob with sof-ctl failed. Move the max to the widget instantiation in sdw-dmic-generic.conf where it is honored, and keep the original 16384 headroom so the setting also covers larger beamformer blobs that may be added later. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Hook a tdfb blob validator into the model handler so a corrupted or mismatching run-time configuration update is rejected before it can replace the working blob. Capture then continues with the previously set filters instead of being interrupted by bad re-configuration. The TDFB blob is variable size and the per-filter walk in tdfb_init_coef() was not bounded against the IPC payload, so a bad length field could push tdfb_filter_seek() past the buffer. The new validator walks every FIR section and the trailing arrays with byte-bounded steps, requires the layout to exactly match config->size, and rejects blobs whose channels or filter_index values do not fit the running stream. The stream channel counts are cached in tdfb_comp_data so the validator can consult them, and the validator is re-run at prepare time once those counts are known. A malformed initial blob is discarded so the runtime path cannot dereference it. With ingress fully validated, the redundant sanity checks inside tdfb_init_coef() are dropped. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
This patch hooks a drc blob validator into the model handler so a corrupted run-time configuration update is rejected before it can replace the working blob. Playback or capture then continues with the previously set parameters instead of being interrupted by a bad IPC. The DRC configuration is a fixed-size struct sof_drc_config, so the validator requires the IPC payload size to match exactly and the self-declared config->size to agree with it. It is installed in drc_init() when the model handler is created, so every blob swap is covered - including any received while the component is in READY - and the redundant size check that drc_prepare() used to run on the initial blob is dropped. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add native_sim board configuration and support in the build script. This allows building and running tests on the host using Zephyr's native_sim target. native_sim leverages the POSIX architecture, but the libfuzzer support specifically requires CONFIG_ARCH_POSIX_LIBFUZZER to be set. Therefore, this wraps fuzzer-specific code in ipc.c and the build of fuzz.c behind this config to allow clean compilation on the standard native_sim board. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add native_sim board target to the sof-qemu-run scripts, and add an option to additionally run it under valgrind. The default build directory is set to ../build-native_sim Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
When building the firmware for native_sim, debugging allocations with host machine tools like Valgrind is constrained due to Zephyr's internal minimal libc tracking the heap manually via static pools. By bypassing Zephyr's memory interception on native_sim using nsi_host_malloc, dynamically tracked memory can surface appropriately to Valgrind memory checkers without causing a libc heap pool panic. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Keep spinning in case user needs to inspect status via monitor. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
When building the native_sim fuzzer, the host allocator does not possess the strict bounds of the internal Zephyr memory pools. If the fuzzer generates a malformed payload requesting an excessively large size (e.g. 4GB), it passes directly to the host ASAN allocator which aborts due to OOM or protection limits. Adding a 16MB cap allows these to fail gracefully. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Only run sof_run_boot_tests() at boot for QEMU or native_sim (STANDALONE) targets. Other platforms trigger via IPC.
The feed-forward and feedback paths copy frames * channels samples into fixed intermediate buffers but only checked the frame count. Bound the total sample count against the buffer capacity so an unexpected channel count cannot overflow the buffers. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
The remap routines used host-provided channel map entries to index the source frame, guarding only against the -1 unmapped sentinel. Skip any entry outside [0, source channels) so a crafted map cannot read past the source frame. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
The fragmented calibration get path advanced a read offset by a host-controlled index without checking it against the calibration data size, allowing reads past the buffer. Reject offsets at or beyond the data size and fragments that would extend past it. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Export ZEPHYR_BASE in platf_build_environ so that post-build west commands (such as west sign on PTL/ACE30) can discover Zephyr's west extension commands when invoked from arbitrary working directories during CMake build. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…lter mode
The filter-mode process path converts interleaved S32 PCM to planar float on
the way into the FFmpeg graph and back on the way out. On HiFi4/HiFi5 VFPU
cores our LLVM Xtensa clang now provides packed conversion intrinsics, so do
the conversions two lanes at a time:
- S32 -> float: float.sx2(x, 31) == (float)x * 2^-31 (fuses the /2^31)
- float -> S32: trunc.sx2(f, 31) == (int32_t)(f * 2^31) with int32
saturation (fuses the *2^31 and the manual clamp)
Packed values are moved through the AE register file with AE_L32X2/AE_S32X2
(the backend has no generic 64-bit-vector memory op); the strided interleaved
side gathers/scatters via an 8-byte aligned temp while the contiguous planar
side loads/stores packed directly. The scalar C loop is kept as a fallback for
GCC and non-VFPU cores, guarded by FFMPEG_AF_VFPU_CONV.
Emission verified with the built clang for intel_ace30_ptl: the vectorized
bodies contain exactly one float.sx2 / trunc.sx2 with AE loads/stores, scalar
mul.s only in the <=1-element remainder tails.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d path When the SOF build uses the LLVM Xtensa clang, ffmpeg.cmake configures the FFmpeg ExternalProject with clang driven by --target/-mcpu/--sysroot only. clang's --sysroot does not add GCC's internal fixed-include directory (lib/gcc/<triple>/<ver>/include) -- the GNU target driver adds it implicitly, clang does not. That directory is where <xtensa/config/core-isa.h> lives. Without it, __has_include(<xtensa/config/core-isa.h>) fails in libavutil/xtensa/float_dsp_init.c, so XCHAL_HAVE_HIFI*_VFPU stay undefined, the VFPU gate is not taken, and the file silently compiles to scalar C -- i.e. the HiFi packed-float SIMD is dropped from the ace40/ace30 build with no diagnostic. Ask the matching GNU gcc for that directory (-print-file-name=include) and add it to the FFmpeg compile flags, guarded on the xtensa/config subdir actually being present so it is a no-op for other toolchains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…odec Link libavcodec/libswresample/libavutil statically into the base image for the =y (non-LLEXT) build instead of the =m loadable module. The =m LLEXT path is blocked on LLVM by the ET_REL relocation issue; LLD resolves everything at link time for =y, sidestepping runtime relocation entirely. CMakeLists.txt (=y else branch): cross-build the FFmpeg archive via ffmpeg.cmake, expose its headers, and link avcodec (before its deps) into the SOF interface library, ordered after ffmpeg_ext. No -lm/-lc: the base image is bare-metal and libav* libc refs resolve from Zephyrs minimal libc. A new
Flesh out the decoder module: pull compressed input from the source ring into a
padded linear bounce buffer, strip the FLAC container header once so the raw
parser sees a bare frame elementary stream, drive libavcodec send-packet /
receive-frame, and stage decoded PCM into a linear buffer drained into the sink
across as many DP cycles as the sink free space needs. STREAMINFO arrives as a
bytes-control blob via set_configuration and is stored as avctx->extradata.
Two fixes make it actually decode and drain on the target:
* ffmpeg_dec-ffmpeg.c: on this cut-down static libavcodec build
avcodec_alloc_context3() does not apply the AVOption defaults, so
avctx->max_samples is left zero and ff_get_buffer() rejects every audio
frame ("samples per frame N exceeds max_samples 0", -EINVAL). Restore the
upstream INT_MAX default before avcodec_open2(). Also route av_log() into
the Zephyr log with identical-message rate limiting so a stalled stream
cannot flood the 64 KiB mtrace ring.
* ffmpeg_dec.c: implement compress drain. On a DRAIN trigger the kernel sets
pipeline->expect_eos and blocks in compress_drain() until the module reports
completion. Pre-build the COMPR-magic module notification (the same template
the Cadence decoder uses, which sof_ipc4_compr_drain_done matches on), and
once the last PCM is flushed - input drained or the decoder yields no more
output under expect_eos - send it once and mark the sink EOS so the state
propagates to the DAI. Without this cplay's compress_drain() timed out.
Verified end-to-end on Panther Lake (ace30, LX7+HiFi4): a 136 KB FLAC clip
decodes to PCM and cplay returns cleanly ("Close Normally") instead of the
prior 25 s drain timeout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…bled get_codec_info() reports the DSP's compress codec capabilities to the driver. Add SND_AUDIOCODEC_FLAC (playback) to the list when CONFIG_FFMPEG_DEC_FLAC is set, so the compress PCM enumerates FLAC as a supported decode format now that the ffmpeg_dec (libavcodec) module provides it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the ffmpeg_dec widget into a compress pipeline. The stock compress path
instantiates the Cadence decoder; ffmpeg_dec needs its own pipeline + platform
instance, gated on a new COMPR_FFMPEG define so the stock compr.conf (COMPRESSED)
is unaffected. They share the COMPR_* pcm/pipeline ids, so enable one or the
other.
* new compr-playback-ffmpeg.conf: clone of compr-playback with the decoder
widget replaced by ffmpeg_dec.
* new platform/intel/compr-ffmpeg.conf: the platform instance, with the FLAC
STREAMINFO bytes control attached at instance level.
* new components/ffmpeg_dec/flac_streaminfo.conf: the STREAMINFO setup blob.
* ffmpeg_dec.conf: change the widget DAPM type "effect" -> "decoder". The
compress kernel path (sof_ipc4_compr_get_module) only finds the codec module
when its DAPM id is decoder/encoder, so a plain effect widget is never
configured on a compress pipeline.
* cavs-mixin-mixout-hda.conf / sof-hda-generic.conf / compr-default.conf:
include the new pipeline and the COMPR_FFMPEG IncludeByKey.
* tplg-targets.cmake: add sof-hda-generic-ffmpeg-compr build target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…alls Replace the return-0 double-libm stubs in the =y build with real newlib math, so lossy decoders (AAC/Opus/Vorbis) get correct math instead of garbage. FLAC is integer-only and never calls libm at runtime, but the references now resolve to real kernels. The Zephyr SDK's pre-built newlib math cannot be linked as-is: the image is compiled -mlongcalls and linked --no-relax, but the SDK objects use direct call8 (there is no -mlongcalls multilib), so an out-of-range inter-object call (e.g. -> memset) fails at link time with "dangerous relocation: call8: call target out of range". So compile newlib's math from source with -mlongcalls -mtext-section-literals (the compiler then emits callx8, unlimited range), which links cleanly under --no-relax. The newlib libm math/ + common/ sources are vendored under libm/ (the exact member set the SDK's libc.a selects, so there are no duplicate definitions; see libm/README for provenance and regeneration). ld/, complex/ and fenv/ are omitted: FFmpeg audio never calls long-double or complex math, and the math/+common/ symbol closure is self-contained (external tail = compiler soft-float builtins + memset, already in the image, + __errno). ffmpeg.cmake compiles libm/newlib_math_sources.cmake into libnewlib_m.a (new newlib_m target); CMakeLists.txt links it last as an archive so only the math objects a decoder actually references are pulled in. ffmpeg_dec-builtin-libc.c drops the fake libm stubs and bridges __errno to Zephyr's per-thread errno (z_impl_z_errno) for the math error paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ace30/40 core-isa reports XCHAL_HAVE_FP=1 (a single-precision scalar FP0 unit), so LLVM lowers plain-C float math in libav* to hardware FP0 ops (mul.s, mov.s, ...). But SOF firmware is built CONFIG_FPU=n: the scalar FP coprocessor is never enabled for any thread (SOF's own float code uses the separate always-available HiFi vector VFPU). An FP0 instruction therefore faults at runtime (EXCCAUSE 0, illegal instruction) the first time a codec runs scalar float math -- e.g. ff_mpadsp_init building the MP3 synthesis window. Integer codecs (FLAC) never emit one, which is why they worked. Disable the LLVM "fp" target feature for FFmpeg's own --cc so it emits soft-float libcalls (__mulsf3, __truncdfsf2, ...) instead. This is scoped to _ff_cc_ff and must NOT reach the newlib libm build (_ff_cc): libm's <fenv.h> selects its FP variant on XCHAL_HAVE_FP and its rur.fcr/rur.fsr inline asm needs the "fp" feature to name the FPU control regs. libm double math is soft-float regardless, and its unreferenced single-precision FP0 variants get GC'd, so libm stays safe on the plain _ff_cc. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SOF runs module init and prepare (including the backend configure step) on the EDF work-queue thread. The ffmpeg_dec lossy decoders (MP3/AAC/Opus) build large cos/pow tables in avcodec_open2(), whose call chain is ~13 KiB deep and overflows the 8 KiB CONFIG_STACK_SIZE_EDF default -- spilling past the guard into the adjacent thread's stack and faulting on its next context switch (EXCCAUSE 9, load/store alignment in xtensa_restore_high_regs). This is not the DP thread's stack, so bumping the topology stack_bytes_requirement has no effect. FFMPEG_DEC_FLOAT_MATH is auto-selected by exactly those decoders, so default the EDF stack to 32 KiB when it is set. The integer FLAC decoder does not need it and keeps the 8 KiB default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the runtime pieces to decode an MP3 elementary stream end-to-end: - Pad the decode packet to AV_INPUT_BUFFER_PADDING_SIZE. FFmpeg's bitstream reader over-reads its word cache past the packet payload; the mpegaudio parser can hand back a packet pointing into the unpadded host input, so copy the payload into the padded scratch and zero the tail before decode. - Skip a leading ID3v2 tag in ffmpeg_dec_skip_container(). Real-world MP3 files prepend an ID3v2 tag that the mpegaudio parser would mis-sync on; skip the syncsafe-encoded tag length so the parser sees frames. The FLAC "fLaC" container skip is generalised alongside it. - Advertise SND_AUDIOCODEC_MP3 to the kernel from get_codec_info() when CONFIG_FFMPEG_DEC_MP3 is built (guarded so it is not advertised twice when the Cadence MP3 decoder is also present). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Parametrize the ffmpeg compress topology on COMPR_FFMPEG_CODEC (default flac). FLAC carries a STREAMINFO codec-setup bytes control (applied as libavcodec extradata before avcodec_open2); MP3 is a self-describing elementary stream and needs no extradata, so the control is wrapped in an IncludeByKey and omitted for mp3. The default resolves to flac, so the existing sof-hda-generic-ffmpeg-compr target is byte-for-byte unchanged. Add a sof-hda-generic-ffmpeg-mp3-compr target that overrides COMPR_FFMPEG_CODEC=mp3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The HiFi FIR cores issue 64-bit AE_L32X2/AE_S32X2 circular accesses into the delay line. buffer_start is offset from the delay_lines base by a multiple of 8 bytes, so the base itself must be 8-byte aligned or those paired accesses fault (EXCCAUSE 9). mod_alloc() with alignment 0 may return only a 4-byte-aligned pointer; request the 8-byte alignment explicitly with mod_alloc_align() instead of relying on the allocator default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ffmpeg_dec_emit_frame() copied the decoder's native PCM straight into the sink buffer, assuming the decoder sample format matched the sink. That is false: the SOF IPC4 pipeline propagates the PCM (sink) bit depth back onto this widget's output pin, so the fixed-point mp3 decoder (S16) and 16-bit FLAC (S16) feed an S32 sink. The old memcpy packed two 16-bit samples into each 32-bit slot and sized the frame from the source bytes-per-sample, so the sink received half the bytes it expected filled with mis-strided data -- audible as constant harmonic distortion and half-rate playback. Convert every source sample through a signed Q1.31 intermediate and re-quantise to the negotiated sink frame format (S16/S24_4LE/S32/FLOAT), de-planarising planar layouts in the same pass. Integer source formats cover the fixed-point decoders built today (mp3, FLAC); the float paths cover the planar-float decoders (AAC, Opus) added later. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ffmpeg_dec data-processing module decodes a whole codec frame (up to ~24 ms of audio for MP3) per process() call, but never advertised an output block size. The DP bind path sizes the ring buffer between this module and the downstream LL chain to 3 x mpd.out_buff_size (ipc4 helper.c), so with out_buff_size left at 0 the ring collapsed to ~3 LL periods - far smaller than one decoded frame. As a result process() could only drain a fraction of each decoded frame into the sink per LL period, then dribbled the remainder out at the downstream real-time rate before decoding the next frame. Decode (~17 ms/frame) and the real-time drain (~24 ms/frame) ran serially instead of overlapping, so a 48 kHz MP3 played at ~0.62x real time (measured 16.2 s of wall for 10 s of audio, gap 21.7 ms/frame). Advertise mpd.out_buff_size in init, sized to hold ~2 decoded frames (worst case stereo S32, capped so the 3x ring stays bounded, floored at one frame for large blocks). The ring now holds several frames, letting decode run ahead and overlap the real-time drain. Measured wall for the same clip dropped to 9.9 s - real time - decode cost unchanged (gap 6.6 ms/frame). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
struct idc_payload lives in coherent (SHARED_DATA) memory and is shared across cores, but it is only a byte array with no explicit alignment, so the linker was free to place it (and every per-core slot) on an odd address. Handlers that read the payload as a wider type -- e.g. the cross-core get_attribute path, which reads a uint32 type field -- then take an EXCCAUSE 9 (load/store alignment) fault when a component runs on a secondary core (ptl/ace30). Align the payload to the cache line. That is the granularity the payload is already built around (IDC_MAX_PAYLOAD_SIZE == DCACHE_LINE_SIZE * 2) and it guarantees every per-core slot starts word-aligned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The remote (cross-core) get_attribute handler passed the caller coherent (uncached) IDC buffer straight to comp_get_attribute(), which memcpy()s a whole ipc4_base_module_cfg into it. On Xtensa the compiler lowers that struct copy to wide accesses (AE_L32X2 / 64-bit) that the uncached window does not support, faulting EXCCAUSE 9 -- so binding a component that lives on a secondary core (e.g. ffmpeg_dec offloaded to core 1 on ptl/ace30) crashed. Have the handler fill a local cacheable copy and publish it to the coherent buffer with aligned 32-bit word stores (ipc4_coherent_copy32); do the same for the read-back on the initiator side. Only COMP_ATTR_BASE_CONFIG is supported remotely, so the fixed-size local copy is always correct. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A remote component prepare() can be expensive: a DP decoder one-time codec open builds large trig tables -- measured ~580 ms for mp3 on a secondary core. Waiting for it stalls the initiator core IPC/EDF thread for that whole time, and with the default CONFIG_IDC_TIMEOUT_US the blocking k_p4wq_wait() instead times out mid-prepare (-EAGAIN) and races pipeline teardown into a crash. Send the prepare IDC non-blocking. Ordering is preserved without the wait: each core drains its IDC work through a single in-order p4wq worker, so the prepare completes before the cross-core trigger (START) queued behind it. For a compress stream the START IPC also arrives only after the app first write, giving the remote prepare time to finish off the critical path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ffmpeg_dec_prepare() opened the backend decoder (avcodec_open2), but prepare() is dispatched to the component core over IDC and runs on that core single IDC worker thread. avcodec_open2 builds large trig tables and can take hundreds of ms (~580 ms for mp3), monopolising the IDC worker so a cross-core prepare/trigger on the initiator core stalls or times out. Defer the open to the first process() call, which runs on the module own DP thread (deep 64 KiB stack, off the IDC critical path). The open is gated on cd->configured; reset() flushes but keeps the decoder open, so it runs only once per fresh prepare(). The first process() returns early after opening and decodes on the next cycle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the ffmpeg_dec widget to core 1 to free core-0 headroom, which also runs the DRC/IIR/SRC of the mix path; sharing a core with the decoder left too little margin and caused occasional steady-state glitches. The cross-core placement exercises the IDC get_attribute and prepare/trigger paths, which are fixed in firmware (idc_payload alignment, ipc4_coherent_copy32, non-blocking comp_prepare_remote, deferred avcodec_open2). Requires that firmware. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
comp_reset_remote() sent the cross-core IDC_MSG_RESET with IDC_BLOCKING, which only waits CONFIG_IDC_TIMEOUT_US (15 ms) before returning -EAGAIN. A data-processing (DP) module reset runs schedule_task_cancel() -> k_thread_join() on the target core and cannot return until the DP thread finishes its in-flight process() call. That call is bounded but can be long during a codec cold-start (a deferred avcodec_open2 table build, up to ~1.4 s for aac, or the first decode after it), far exceeding the IDC timeout. When it timed out the component was left un-reset (not COMP_STATE_READY), so pipeline_reset() failed and the host-side free was then refused with an invalid-state error and the remote core could not be disabled - every teardown wedged until reboot. The caller (pipeline_reset -> host) depends on the reset actually being finished before it frees the component and powers down the remote core, so the reset must stay synchronous. Add IDC_BLOCKING_NO_TIMEOUT, which is IDC_BLOCKING but waits K_FOREVER for the handler to complete, and use it for comp_reset_remote(). A normal end-of-stream teardown joins an idle DP thread and returns immediately, so this only waits when it must. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
process() treated "decoder produced no PCM this cycle" as end-of-stream whenever pipeline->expect_eos was set. The host sets expect_eos as soon as it has written the whole (buffer-resident) file, long before the DSP has decoded it, so expect_eos is true for essentially the entire stream. A compressed chunk can legitimately end mid-frame -- variable-size ADTS AAC frames routinely do -- leaving the parser holding a partial frame it completes from the next chunk. That cycle consumes input but yields no PCM, which is normal, not EOS. The spurious EOS truncated AAC playback to its first few frames (~100 ms) and returned the drain immediately. Drop the produced==0 EOS trigger. The real drain still completes on a later cycle once the input is genuinely exhausted (avail==0 with expect_eos, handled at the top of process()). Fixed-point MP3/FLAC were unaffected only because every chunk yielded a full frame. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add AAC-LC as the second lossy codec decoded by the built-in (=y) libavcodec module, after MP3. AAC-LC is the first floating-point decoder exercised, so it pulls in libavutil tx (MDCT) and float_dsp. - base_fw: advertise SND_AUDIOCODEC_AAC to the host when CONFIG_FFMPEG_DEC_AAC is built (guarded against the Cadence AAC decoder), so the compress device accepts the codec id. - CMakeLists: compile the FLT/FLTP -> Q1.31 sample-conversion glue with soft-float (-target-feature -fp). core-isa.h reports XCHAL_HAVE_FP=1, so LLVM would otherwise lower the conversion to hardware FP0 ops that fault on this CONFIG_FPU=n image. The property must target the modules_sof scope (created in a different directory) via TARGET_DIRECTORY. Also build the =y heap-routing shim. - ffmpeg.cmake / ffmpeg_dec-builtin-alloc.c: --wrap malloc/free onto the SOF DSP heap so av_malloc() draws from .heap_mem (AAC-LC needs a single ~541 KiB allocation the common-libc arena cannot hold); define SOF_AAC_LC_ONLY so the decoder drops the LD/ELD-only transforms. - compr-ffmpeg: the DP thread now also runs the deferred avcodec_open2; AAC-LC float table build + per-frame decode is the deepest path and needs the 256 KiB stack. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add sof-hda-generic-ffmpeg-aac-compr, mirroring the MP3 variant, so the compress pipeline is built with COMPR_FFMPEG_CODEC=aac. AAC (like MP3) is self-describing and needs no STREAMINFO setup control, so the codec IncludeByKey in compr-ffmpeg.conf already omits it for any non-flac key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…count The HDA compress gateway free-runs its write pointer circularly and re-presents already-played data, so host_handle_eos() never sees dma_stat.pending_length reach 0 and the source buffer is never flagged END_OF_STREAM. A compress drain therefore never completes. On DRAIN the kernel now hands the host copier the total committed gateway byte count via a new SET_LARGE_CONFIG param (IPC4_COPIER_MODULE_CFG_PARAM_DRAIN_BYTES = 100), stored in host_data.drain_bytes. Once the copier has delivered that many bytes (total_data_processed >= drain_bytes), host_handle_eos() sets the source buffer to END_OF_STREAM and stops the copy, giving the pipeline a correct end of input. drain_bytes is cleared on reset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consume the source END_OF_STREAM state set by the host copier at the committed drain boundary. The decoder finishes the drain in two cases: - avail == 0 with the source at END_OF_STREAM: the input is fully drained. - a produce cycle yields nothing while at END_OF_STREAM: the copier only flags END_OF_STREAM after delivering the whole committed file, so every real frame has already decoded; the free-running HDA gateway then only re-presents stale junk that yields no PCM, so the first empty cycle means the stream is exhausted. Signal EOS (idempotent via cd->eos_sent) to release the host's compress_drain(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The planar-float output path multiplied every sample by 2^31 as a scalar float, which is soft-float on this DSP (no scalar HW-FP) and dominated the AAC decode (~65-82% of each frame). Multiplying an IEEE-754 float by 2^31 only bumps the exponent, and the cast truncates toward zero, so the same Q1.31 value is obtained by shifting the 24-bit significand: pure integer, constant-time, and bit-exact with the old float path. |f| >= 1.0 saturates; denormals round to 0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… topology
Bring up the libshine fixed-point MP3 encoder end-to-end in the ffmpeg_dec
module in encode mode (CONFIG_FFMPEG_DEC_ENCODE_MODE / CONFIG_FFMPEG_ENC_MP3),
built =y into zephyr.elf, plus the compress-capture topology that feeds it PCM
from the onboard HDA analog capture BE and delivers MP3 to the host.
Firmware:
- ffmpeg_dec-encode.c: encode path driven through the DP source/sink
interface. The module exposes .process + .is_ready_to_process instead of
.process_raw_data; is_ready_to_process gates DP re-invocation on
source_get_data_available(sources[0]) > 0. The DP task opens the encoder
lazily (deferred avcodec_open2) on first process(), accumulates a full
1152-sample stereo frame (need = 9216 bytes) from the source ring, encodes,
and writes the variable-size MP3 frame to the sink.
- ffmpeg_dec.c/.h: wire the new source/sink process + is_ready_to_process ops.
- base_fw.c: advertise SND_AUDIOCODEC_MP3 for capture, guarded so it is not
duplicated when the Cadence MP3 encoder is present.
- ffmpeg.cmake / CMakeLists.txt: build libshine in encode mode.
Topology (tools/topology/topology2):
- include/components/ffmpeg_enc.conf (new): encoder widget class, same module
UUID as ffmpeg_dec, DAPM type "encoder" (the compress kernel path accepts an
encoder-type module for capture). MP3 is self-describing, no setup control.
- include/pipelines/cavs/compr-capture-ffmpeg.conf (new): compress-capture
pipeline; in-pipeline LL head copier -> ffmpeg_enc (DP) -> aif_out host-copier.
- platform/intel/compr-capture-ffmpeg.conf (new): compress-capture PCM (id 51,
direction capture, MP3) and routes tapping the shared HDA analog capture BE.
- cavs-mixin-mixout-hda.conf: turn the HDA analog capture module-copier into a
proper 2-pin fan-out (num_output_pins/num_output_audio_formats 2 +
Object.Base.output_pin_binding by sink name; requires the
common/output_pin_binding.conf include) so the compress FE can bind a second
consumer on its own pin instead of colliding on pin 0.
- compr-default.conf / sof-hda-generic.conf / tplg-targets.cmake: COMPR_ENC_*
defines, include wiring, and the sof-hda-generic-ffmpeg-mp3-enc-compr target.
Status: the MP3 encoder core is proven at runtime on aphid (PTL) -- deferred
avcodec_open2 succeeds ("libshine MP3 open OK, rate 48000 ch 2 frame_size 1152
need 9216"), the self-test emits a valid MP3 frame, firmware boots with no
fault. The live-capture data path is not yet flowing: the compress-capture FE
receives 0 bytes from the shared analog capture BE, and starting the compress FE
disrupts a concurrently running normal capture. The two topology fixes above
(fan-out pin-binding + in-pipeline head copier) are real and correct but did not
change the symptom; the remaining work is at the firmware/kernel DPCM level
(compress-capture trigger/hw_params from a shared analog BE) and is left as a
documented follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The libshine MP3 encoder now produces valid MP3 on PTL/aphid silicon (avcodec_open2 OK, MPEG-1 Layer III 128 kbps/48 kHz/stereo, no faults) once the shine clang/Xtensa backend miscompile is worked around. Two sizing fixes are required to reach the first encoded frame without overrunning the capture DAI. process() runs on the DP thread and blocks for the full duration of a single libshine frame encode while the LL side keeps producing real-time capture PCM. The default (zero) in_buff_size gives too shallow an input ring, which overflows mid-encode, backs up the shared HDA-capture fan-out copier, and overruns the capture DAI gateway before the first MP3 byte exists. Set in_buff_size = FFMPEG_ENC_PCM_STAGE (16 KiB) so the ring (sized ~3x this by bind) absorbs one whole worst-case encode. Bump the compress-capture module heap_bytes_requirement to 131072 to cover the deepened input ring plus pcm_in and mp3_out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The compress-capture (encode) FE fed its DP ffmpeg_enc module from a module-copier second output pin (module-copier.4.2 pin 1). A copier output pin does not deliver data across a pipeline boundary, so the encoder input ring stayed empty and the DP module starved - crecord returned 0 bytes. Rebuild the HDA analog capture BE around the proven mixer fan-out when COMPR_FFMPEG_ENC is set: the BE (dai-copier -> gain -> mixin.4.1) fans a single mixin out to two mixouts - mixout.3.1 for the normal analog capture FE (pcm0c) and mixout.91.2 for the encoder FE (pcm51) - exactly as the playback mix path distributes a stream. mixin->mixout delivers across the pipeline boundary; a copier pin does not. Because a DP module breaks the LL copy walk, the encoder tap mixout cannot share the DP encoder pipeline (it would sit on the far side of the break from that pipeline sink scheduler and never run). Isolate it in its own timer-scheduled mixout-capture pipeline (COMPR_ENC_MIXOUT_PIPELINE_ID=93): standalone it self-schedules on the timer, drains the mixin, and its output crosses the boundary into the DP encoder input ring - mirroring how the decode pipeline LL host-copier feeds ffmpeg_dec. Runtime graph: mixin.4.1 -> mixout.93.1 -> ffmpeg_enc.91 -> host-copier. Verified on PTL (aphid): PCM->MP3 end to end, host-readable MPEG-1 layer III 128 kbps 48 kHz stereo (0xFFFB frame sync), avcodec_open2 OK, no faults. The normal analog capture FE (arecord pcm0c) runs unaffected on the shared BE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP build support for llext based and runtime linked ffmpeg audio encoding, decoding and filters as a west module. Builds and links today. Next steps are integration with topology and kernel then optimization.