Simple and robust algorithm for video thumbnail generation

The simplest algorithm that comes to mind for generating a thumbnail for a video is to randomly pick a frame in the video (even simpler could be to just pick the first frame of the video). While the algorithm is very intuitive and looks good at first glance, it has one problem – it can pick a monotonous frame, like an all black frame, which obviously is not a good thumbnail because it doesn’t convey any information regarding the video.

A good heuristic algorithm (referred from this stackoverflow post) to tackle this problem is to select a few random frames (let say, 5) in the video and pick the frame that has largest file size as video thumbnail – the idea being that the JPEG of a monotonous frame will compress into much smaller file than that of an interesting frame (one with lots of objects and colors).

The approach mentioned in the algorithm looks neat and hence I thought of evaluating it on few videos and see for myself if it really solves the problem of picking monotonous frames as video thumbnails.

I evaluated it on 10 videos – 2 random videos each (720p at 24 FPS) from Friend’s (American sitcom) season 3 to 7. I divided the video frames into 3 categories – bad, interesting, and transition. Monotonous frames are bad frames. Interesting frames are those that contain lots of objects and colors. The frames which lead to transition from bad to interesting frames, or vice versa, are transition frames. For each video, I picked 5 frames randomly out of the first 120 frames (initial 5s of the video) which consists of 2 bad frames, 1 transition frame, and 2 interesting frames.

Here are the results:

[Test Video 1] Friends S3 E3

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

114_interesting
Interesting Frame 1 – 164 KB File Size

[Test Video 2] Friends S3 E16

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

54_transition
Transition Frame – 170 KB File Size

[Test Video 3] Friends S4 E5

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

104_interesting
Interesting Frame 1 – 250 KB File Size

[Test Video 4] Friends S4 E13

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

99_interesting
Interesting Frame 2 – 150 KB File Size

[Test Video 5] Friends S5 E7

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

93_interesting
Interesting Frame 2 – 234 KB File Size

[Test Video 6] Friends S5 E22

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

116_interesting
Interesting Frame 2 – 150 KB File Size

[Test Video 7] Friends S6 E10

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

98_interesting
Interesting Frame 1 – 238 KB File Size

[Test Video 8] Friends S6 E24

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

80_interesting
Interesting Frame 1 – 228 KB File Size

[Test Video 9] Friends S7 E1

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

89_interesting
Interesting Frame 1 – 223 KB File Size

[Test Video 10] Friends S7 E19

This slideshow requires JavaScript.

The above slideshow is displaying the 5 randomly picked frames along with their respective types and JPEG file sizes. The thumbnail for this video is –

101_interesting
Interesting Frame 2 – 246 KB File Size

Out of the 10 thumbnails selected by the algorithm, 9 are interesting frames and 1 is transition frame. The algorithm did a very good job of not picking monotonous frames as video thumbnails.

P.S. This algorithm is not a perfect solution but it certainly does a very good job of generating a good thumbnail for a video.

Advertisements

4 thoughts on “Simple and robust algorithm for video thumbnail generation”

  1. Probably not a good idea to link to illegal media downloads, Harish.

    You can also see some of the places where the approach doesn’t get you a good result, so I’d love to see some more analysis and options if possible.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s