[moi15moi]

This PR partially fix #71

This new render type draws ASS subtitles with OpenGL directly onto each decoded video frame, inside media3's video effects pipeline. It is much faster then the other render type because, first of all, it use the gpu to blend libass bitmap. Secondly, instead of uploading each libass bitmap to the gpu, it create an atlas to send has many bitmap as possible to the gpu.

It is much more complex, but it helps a lot to improve the performance. From what I could see, with my phone, I can now play really heavy .ass subtitle and we have similar performance to mpv.

I tested it with these 3 samples on my phone which are heavy (thanks to @witchymary for providing them):

• https://files.catbox.moe/q4eab5.mkv
• https://files.catbox.moe/ioxho8.mkv
• https://files.catbox.moe/t4xd9a.mkv

To be sure it is frame accurate, I used this sample (frame_accurate.zip). Each frame goes from black to white to black to white. On each black frame, there is a .ass white draw. Like that, when playing the video, you should only see white frame. If you see any black frame, there is a problem with the accuracy.

I added many unit tests for this new render type to cover has many case as possible.
You can run those tests on a non-android device (the CI runs those tests).

Note that I did use Claude to create this PR.

[moi15moi]

I just realised that the crop option in the demo app doesnt work properly with this new renderer.
I will look about it.

Edit: From what I can see, AssRenderType.EFFECTS_OPEN_GL as the same problem, so it might be ok?

[moi15moi]

I forgot to add that it render to the video resolution (not the screen size).

[moi15moi]

@skoruppa I just saw that NuvioTV switched from libass-android to your implementation in NuvioMedia/NuvioTV#4165

I find it kinda funny that both PR (mine and yours) use the same approach (an atlas) and almost have been made in same time.

I was wondering if you tested this PR? Do you know if yours perform better (I haven't check the detail, but I was just wondering).?

[skoruppa]

Hey! Yeah, both landed on an atlas approach around the same time ;)

The main difference is the rendering path: ours draws on a separate SurfaceView overlay (pinned to the video frame's vsync via eglPresentationTimeANDROID), while yours draws directly onto the video frame through setVideoEffects/GlShaderProgram. Both use atlas packing, but yours has the native C packing (AssSkyline) with PBO uploads on GLES3, which is neat.

Your EFFECTS_ATLAS approach is actually really interesting for me - we have a subtitle timing issue on some Android TVs where the overlay SurfaceView goes through a shorter display pipeline than the video surface (TV image processing adds latency to the video path only), causing subtitles to appear ~7 frames early. Compositing directly onto the video frame like your PR does would eliminate that entirely. I need to check it.

Performance-wise, our overlay SurfaceView path works well on both phones and TVs (the rendering is off the main thread, speculative render-ahead for heavy typesetting, etc), but your in-place draw avoids the second surface entirely which is cleaner. The tradeoff is HDR/DV support - our overlay path handles that since it's a separate layer, but for SDR content maybe yours would be the better approach? Will do some testing ;)

[moi15moi]

while yours draws directly onto the video frame through setVideoEffects/GlShaderProgram

Yes, that's a big advantage of setVideoEffects. You are sure that the subtitle will be rendered/composited on the right frame. With typesetting, it is something that is needed or the subs will look weird.
I believe it's the only way to be sure that the subtitle will be frame accurate.

the rendering is off the main thread, speculative render-ahead for heavy typesetting

That sounds really interesting. I should think about doing that (but probably in another PR). The main problem I see with that is that we would need to do a copy of libass bitmap to "cache" them and it would take more memory usage.
I know mpv calls ass_render_frame on the fly (so no prerendering) and, overall, it perform great, so it might be overkill to do that but I need to think about it.

The tradeoff is HDR/DV support.

There is probably a way to support HDR/DV with my approach, but I don't know well how HDR/DV works and I don't have the material to test it, so that's why I didn't really tried to implement it.

Will do some testing ;)

Thanks!

[skoruppa]

I tested your EFFECTS_ATLAS approach on my TCL TV and it works great - subtitle timing is perfect, exactly as you described

One issue I ran into: on heavy typesetting (karaoke with many quads), the synchronous rendering on the GL thread was causing video frame drops on this TV. Since ass_render_frame + atlas pack + upload all happen inside queueInputFrame, a slow libass render (30-80ms) would exceed the 42ms frame budget and stall the video pipeline.

I solved this by running the libass render + atlas GL drawing on a dedicated thread with a shared EGL context. The render thread writes to a shared RGBA texture, and the video effect's GL thread blends it onto each video frame with a single fullscreen quad. This way libass never blocks video delivery - same model as the overlay SurfaceView but composited onto the frame. I used glFenceSync/glClientWaitSync for cross-context texture visibility.

On the speculative render-ahead: we ended up not needing it for this approach since the render thread runs independently - if libass is slow, the video thread just redraws the last cached texture. Subtitle animation may drop to a lower framerate but video stays smooth

Would you be OK with me shipping a version in NuvioTV that builds on your EFFECTS_ATLAS PR? I would credit you our libass lib. The core atlas + GL code from your PR is really well done and I rather build on it than reimplement from scratch ;)

[peerless2012]

Yes, I think the render shoul work on a thread and set a timeout, just like other render type do. or it will block the video frame.

[skoruppa]

I investigated a bit and the problem with video effects approach is that HDR just can't work - using them will always tone map video to SDR.

Because of that I think I will leave both methods in Nuvio ;) and let users decide what is more important for them, frame perfect ass subs or hdr with ass subs but with suboptimal sync

[peerless2012]

I investigated a bit and the problem with video effects approach is that HDR just can't work - using them will always tone map video to SDR.

Because of that I think I will leave both methods in Nuvio ;) and let users decide what is more important for them, frame perfects ass subs or hdr with ass subs but with suboptimal sync

We can add OverLaySurface to render ass in a SurfaceView, that maybe useful.

[skoruppa]

Update after more testing: the video effects path has more limitations than just HDR:

• setVideoEffects forces DefaultVideoFrameProcessor which drops frames on 60fps content. Even a total passthrough effect (zero GL work, just forwarding the frame) still drops frames - the overhead is in DVFP itself (SurfaceTexture latch, GL context management, shader chain), not in our effect code.
• Dolby Vision content through DVFP produces a green/corrupted image. DV requires the native hardware compositor path

Because of these issues the default mode in Nuvio will be Overlay. Video Effect is available as an opt-in for users who need frame-accurate ASS timing on 24fps SDR content and accept the tradeoffs. The player only activates DVFP when an ASS track is actually selected (lazy activation), so non-ASS content never pays the overhead even with Video Effect mode enabled

[moi15moi]

Thanks for testing all of this!

The hardware you're testing on seems REALLY weak. If turning on DefaultVideoFrameProcessor with a passthrough effect is enough to drop frames on 60fps video, the GPU can't even manage the bare minimum DVFP does at 60fps: sample the decoded frame, draw it into a texture, then draw it again onto the output surface. Was that 4K60 or 1080p60?

I checked why setVideoEffects with no effects seems to affect your device so much.
This is my conclusion.

Without setVideoEffects
MediaCodec is configured with the SurfaceView's surface (getSurfaceForCodec() returns displaySurface). The decoder writes each frame straight into that surface's buffers. When the frame is due, ExoPlayer calls releaseOutputBuffer(index, releaseTimeNs) and the buffer goes directly to the display.

In brief, the app does no GPU work and makes no copies of the frame.

With setVideoEffects

1. The codec decodes into DefaultVideoFrameProcessor's SurfaceTexture instead of the SurfaceView (getSurfaceForCodec() returns videoSink.getInputSurface()).
2. The GL thread picks up the buffer. onFrameAvailable (L146) leads to maybeQueueFrameToExternalShaderProgram (L400), which latches the frame with updateTexImage().
3. GPU pass 1. DefaultShaderProgram.createWithExternalSampler (L279) converts the frame from YUV to RGBA into an intermediate texture. This is also where Dolby Vision breaks because the GPU only sees the pixels, not the Dolby Vision metadata needed to interpret them.
4. The effects passed to setVideoEffects run. With a passthrough effect, nothing happens here.
5. GPU pass 2. FinalShaderProgramWrapper.renderFrameToOutputSurface (L507) redraws the frame onto the SurfaceView and calls eglPresentationTimeANDROID + eglSwapBuffers.

In brief, the GPU draws every full frame twice and the display receives an RGBA buffer made by the GPU instead of the decoder's YUV buffer.

Have you tried mpv on your TV?
I believe it should have the same problem as this PR.
Also, when you tested this PR, did you select the "release" variant type in android studio? From what i can see, there is a pretty big difference between the debug and the release performance.

I am currently trying to find an alternative solution, but I don't think there is a solution that would allow us to be frame accurate (which, for me, is extremely important). I don't care if the video lags due to the subtitle. I prefer subtitle accuracy over lag.

[skoruppa]

The test was 1080p60 anime on a TCL MediaTek TV (API 34, Mali-G52). Not a flagship but more capable than most TV boxes our users run (Amlogic S905X, Rockchip, older Fire TV sticks). If DVFP can't hold 1080p60 passthrough on this hardware, it definitely won't work on weaker devices. It was on a release build.

The implementation I got runs better than mpv on my device which is good :D

[moi15moi]

The implementation I got runs better than mpv on my device which is good :D

Are you talking about this PR or what you did in NuvioMedia/nuvio-libass#1 ?

[peerless2012]

I think the async render and not block or block with a timeout is a better way. Maybe we just move render and copy the pieces memory to page on a thread, upload tex and render still work on player thread. As the most heavy work is libass render, not the gl render.

[moi15moi]

I think the async render and not block or block with a timeout is a better way.

I don't agree.
Even if for one frame we take more time than the frame duration (ex: we took 50 ms instead of less then ~42 ms for a 23.976 fps video), it won't make the video lag because queueInputFrame is async, so if the next frame take ~34 ms, it will be ok. Due to that, it's not possible to know what is a "good" timeout value.

I believe it's better to lag then to skip subtitles.

As the most heavy work is libass render, not the gl render.

The gl can take a signicative amount of time when there is a lot of bitmap.
I did see 10 to 15 ms with some sample I shared.

[peerless2012]

Can you test this file ass test?