UDP push to the U cell loses packets: the socket buffer overflows on key-frame bursts #251

Problem

An RTP push to the U cell (udp://@:5600) loses packets on the receiving box although nothing is lost on the network. It was found while testing pusher (our RTP sender for tests) in the lab (no VPN, everything on one LAN):

• Sender: camera ssc377d-imx335 (.113), H.265 1080p, Majestic outgoing (1200-byte packets) or pusher (1400-byte packets).
• Receiver: Aceline box (.145), Decoder 2.18.2, Android 10.
• The drops counter of the :5600 socket in /proc/net/udp6 on the box keeps growing: Majestic ≈47 packets/s, pusher without pacing ≈13 packets/s (33/s for H.265 1080p30 at 6.5 Mbit/s). The sender has no errors (SndbufErrors 0).
• The log says NAL unit dropped: packet loss inside a fragmented NAL unit, and the picture is damaged after key frames.
• The box has rmem_default 229376 and rmem_max 262144. RtpListener asks for SO_RCVBUF 2 MiB, but an app can't go past rmem_max, so the kernel gives about 512 KiB of buffer. That's only ~200 datagrams, about the size of one 1080p key frame.
• With pacing on the sender (pusher 0.18.x, 32 Mbit/s) there are no drops. The box starts dropping between 48 and 64 Mbit/s of instantaneous rate.

So the socket buffer overflows on the burst of a key frame. Any short stall of the thread that reads the socket turns into lost packets.

Second finding: stream checks of other slots

In a 20-minute run with pacing, the only loss episode (50 packets) happened exactly when Decoder started a background check of slot 1 (Probe-1: playing over TCP, RTSP to another camera), and Quad-8: UDP pt 97: 50 packets lost followed.

Stream checks (#100) run in the U view while the debug page is in use, and /stats.json counts as in use. A tool that polls /stats.json (pusher's test harness, monitoring) therefore makes Decoder open RTSP sessions to the other cameras of its slots while the U cell receives a push.

Plan

1. RtpListener: a dedicated receive thread with a raised priority that only reads datagrams into a ring of buffers. Reordering, NAL assembly and the decoder run on the existing thread and never hold up receive().
2. Log the effective SO_RCVBUF (getReceiveBufferSize()) when the socket opens.
3. /stats.json: for the U cell, the socket's kernel drop count (from /proc/net/udp{,6} when the app can read it) next to the RTP sequence gaps, to tell overflow on the box from loss on the network.
4. Stream checks: no checks of other slots while the U cell is receiving a stream.

The sender side (pacing) is handled in pusher separately.