OT: odd thought about variable frame rate.

Chienworks wrote on 6/24/2004, 12:02 PM
I was just sitting here pondering a bit and wondering if this topic had ever been addressed ... not necessarily in this forum, but as a practical application. One of the problems with video is that the file size is huge because a full frame picture is presented many times per second. There are spacial compression schemes out there that reduce the amount of information needed for each frame and also temporal compression schemes that re-use information from previous frames to cut down the amount of information necessary even more. Variable bit rate encoding enhances this by cutting down on bits in low motion portions of the video and saving them for high motion portions. However, they all require a new picture every X amount of time no matter what is happening in the video.

Sometimes to make smaller files, or to increase the quality of a file without increasing the size, i'll encode at a lower frame rate. Maybe i can get away with 10fps for this video and then each frame can have 3 times as many bits for the same final file size. Often this can work well when there isn't much motion. The problem is that occasionally a fast action scene will come along and the result looks very jerky. For that matter, picture slide shows could usually work well at 1fps or less ... except when there is a transition. Then the full frame rate is needed to make the transition smooth. Unfortunately this means that there must be that many frames per second during the stills too.

So, why isn't there a video format that allows the frame rate to vary? When there is heavy action the video can run at 29.97fps. When the action is very slow it can drop to, say, 12fps for a while, or even 1fps when it's a still shot. Each frame would have to include it's duration, and probably a timecode stamp as well to ensure proper playback sync. It doesn't seem like this would be difficult to achieve.

Encoders could be written to automatically detect content and adjust the frame rate as necessary the same way that VBR encoders currently detect which scenes need more or less bits. Perhaps a program like Vegas would be able to include a manual FPS envelope so that the editor could specify the frame rate changes.

Anyway, that's my weird thought for the day.

Comments

mark2929 wrote on 6/24/2004, 12:51 PM
Perhaps You could have an encoder that Automaticly detects the changes in motion... and adjusts itself accordingly... I think thats a Brilliant line of thought and would be very usefull on the Internet..
JohnnyRoy wrote on 6/24/2004, 12:53 PM
I suggest you patent the idea. Since you have just placed it in the public domain by posting, you have given up your international rights but in the US you still have one year to file a patent from the time it is first introduced to the public. Sounds like a great idea to me. ;-)

~jr
Chienworks wrote on 6/24/2004, 2:08 PM
Unfortunately the patent office will require a working model demonstrating the idea. I'm not sure my programming skills are up to that sort of task. I write order entry & billing software for a living, not multimedia stuff. Hmmmmm. Then again ... i suppose i could try.
Jsnkc wrote on 6/24/2004, 2:32 PM
I had a idea a couple years back to have computers connected to a huge music database, then when you walk in a record store you could just go up to a computer, select the tracks you wanted, then it would burn and print and then pop out a CD for you.

Dang Starbucks must have read the post I made on a messageboard about it !
Former user wrote on 6/24/2004, 2:36 PM
A potential gotcha is determining how much of a change in a frame is required before a new frame is drawn.

A mouth moving, a head move or a scene change. This could get real tricky. Compression looks at each pixel and redraws a new frame depending on what pixels have changed and which haven't.

In the case of even a talking head, the mouth would change each frame.

Dave T2
farss wrote on 6/24/2004, 3:23 PM
This is a more complex problem than you probably realise. To some extent changing the fps is not what you want to do. You need to consider both sides of the issue, how the encoder is going to work and then how it will be decoded. Potentially the idea could be feasible for non realtime compression. Having to makeup new frames to that the images can be shown at 35 or 30 fps is going to require a lot of silicon horsepower.
Pretty much what you're talking about though is already done with VBR mpeg-2 encoding and mpeg-4 pushes the concept even further. WMV also employs much the same same and again at a quite advanced level.
The biggest problem is defining movement. Even a locked down camera shooting a still scene may have movement, changes in air temp etc can cause the image to shift, you don't see it but the encoders do. Mpeg-4 permits you to define a screen area as being static, in other words effectively say a CK background to save bandwidth.
Where variable fps is already being discussed is for digital cinema projection. Scenes with low motion can be shot / projected at say 18 fps but fast motion could be run at 60 fps to improve temporal resolution. Kind of blows the whole 24fps looks more natural argument out of the water.
I'd also mention that with a bit of hard work you can achieve what you want using mpeg-2 encoders. Encode say the titles at a low bandwidth, your slow scenes at medium and the fast motion at high and then stitch it all back together. Sounds tedious but I believe that's how Hollywood does it.
johnmeyer wrote on 6/24/2004, 5:17 PM
The "I" frames in MPEG2 already do pretty much what you describe. If the current frame is identical to the previous frame (a video of a still photo, for instance), the "I" frame is trivially small -- pretty much the same thing as changing the frame rate to 1 fps or less. Even with slight motion, like a person sitting still in a chair, the intermediate frame information is very small. However, you still get the subtle motion at 29.97 fps, which is really important because any motion, no matter how subtle, looks lousy at less than 15 fps (roughly the original silent film speed).