support using `decord` library for video input

[zijielou]

Problem/Use Case

Is the input for scenedetect limited to video file paths? Does it support inputting tensors, such as videos read using decord? Additionally, can scenedetect detect whether a video contains multiple scenes without segmenting the video?

[Breakthrough]

Is the input for scenedetect limited to video file paths?

PySceneDetect also supports PyAV which allows input from any file-like object, as well as MoviePy which should work with streams. Most backends already support network streams and image sequences.

Does it support inputting tensors, such as videos read using decord?

Could you clarify what you mean by inputting tensors? Do you mean a large list of videos?

decord looks pretty cool, thanks for the link - I'll look into what's required for integrating that as a backend for PySceneDetect in addition to OpenCV/PyAV/MoviePy. I'm curious if it will be faster than those, especially with GPU decoding.

Additionally, can scenedetect detect whether a video contains multiple scenes without segmenting the video?

Yes, by default it doesn't cut videos. You need to explicitly state if any output should be created.

[zijielou]

Thanks for your reply! After reading a video using the following command:

import decord
video_path = 'path_to_video.mp4
vr = decord.VideoReader(video_path)

Can PySceneDetect be used directly for scene detection? For example:

from scenedetect import detect, ContentDetector
scene_list = detect(vr, ContentDetector())

[Breakthrough]

Thanks for your reply! After reading a video using the following command:

import decord video_path = 'path_to_video.mp4 vr = decord.VideoReader(video_path)

Can PySceneDetect be used directly for scene detection? For example:

from scenedetect import detect, ContentDetector scene_list = detect(vr, ContentDetector())

It doesn't seem to be a fully-compliant cv2.VideoCapture drop-in replacement as purported, but judging by the API examples, we should be able to integrate it into PySceneDetect as a backend. I can consider this for a future release though.

If you need this sooner, you could try implementing the VideoStream interface for decord's VideoReader. That would allow using it with PySceneDetect's SceneManager API.

Note that GPU support is out of scope for the time being, but I am open to adding support for it in the future. Right now there are no libraries on PyPI that come with GPU-enabled binaries out of the box (decord also falls into this category), so I would prefer if that problem could be solved first before we tackle GPU support.

[moi15moi]

decord isn't maintained anymore. Is it really a good idea to use it?

[Breakthrough]

We'll need to reconsider this if decord isn't being maintained. When we get to this, if it's not too difficult to implement the VideoStream interface for decord we might still want to add this just for the sake of compatibility.

It does seem to get quite a bit of downloads still, so even if it's not maintained, we still might want to at least support it for users who have it installed.

[awkrail]

Decord is developed for deep learning. It focuses on sampling frames effectively because recent video AIs, including video LLMs, sample K (K=8, 16, 32) frames uniformly for their inputs. On the other hand, video shot detection requires all of the frames, so I don't think that decord is useful for our case. In addition, it is not maintained, so I don't think that it is necessary to integrate discord into PySceneDetect.

[Breakthrough]

I'm curious if the performance claims made by decord are still accurate, in which case it could be worth exploring a side project to create a more performant replacement.