How to quantize (frame-for-frame) many events?

NickHope wrote on 2/2/2016, 9:32 AM
My Samsung Galaxy Note 2 phone has kindly recorded lots of files at very slightly different frame rates. 29.980, 29.995, 30.006 etc.. No 2 are the same. Wonderful.

Any quick way to quantize all of them so they line up frame-for-frame on a 29.970 timeline so that there will be no interpolation or blending? Just to be clear, I'm talking about stretching (or squeezing) them, not just quantizing the ends.

If it helps, I have a Vegasaur licence.

Comments

john_dennis wrote on 2/2/2016, 10:03 AM
I would try putting a file on the timeline, set no resample and render to a still image sequence. Put the stills back on a 29.97 project.

Edit:
In a more lucid moment I realized that the render to the still image sequence would contain the effects of any frames that were too early or late. It would be the same as using the variable frame rate video in a project without doing anything.
musicvid10 wrote on 2/2/2016, 11:04 AM
Another thing to try would be Yamb and mux to 29.97 Constant Frame Rate.
If the 3GP extension is bothersome, try changing to MP4.
Vegas handles VFR very poorly in my estimation.
NickHope wrote on 2/3/2016, 7:03 AM
Thanks very much guys.

So after much faffing around I ended up doing this with mp4box in a Windows batch file. This is clumsy and inelegant, and would certainly be better if it cycled through each step per file and found a nicer way to truncate the stream names, but it gets the job done, and the resulting files sit nicely on the timeline with all the frames lined up perfectly.

I do get a warning "[AVC/HEVC] Not enough bits in bitstream !!" for each file but it doesn't seem to harm the result.

I don't really get why anyone would think it's a good idea to create files with variable frame rates that are nearly but not quite constant. Just make them constant!

Also, I don't think a "quantize frames" command would be a particularly challenging addition for the SCS coders. If it was me I would design it to prompt to quantize frames when a variable frame rate file is dropped on the timeline (while displaying the range of frame rates in the said file).

By the way, the files from this phone are already named .mp4, not .3GP.

Install mp4box, paste this into notepad, save it as set-constant-fps.bat in the folder with the media in it, and double click it.

@echo off & setLocal EnableDelayedExpansion

REM Extract h264 and aac streams
for %%a in (*.mp4) do mp4box "%%a" -raw 1 -raw 2

REM Remove "_track1" from the h264 filename
for /f "tokens=* delims= " %%a in ('dir /b *.h264') do (
set F=%%~Na
set F=!F:~0,-7!
move /y "%%a" "!F!%%~Xa"
)

REM Remove "_track2" from the aac filename
for /f "tokens=* delims= " %%a in ('dir /b *.aac') do (
set F=%%~Na
set F=!F:~0,-7!
move /y "%%a" "!F!%%~Xa"
)

REM Mux with constant 29.970 frame rate
for %%a in (*.h264) do mp4box -add "%%~na.h264:fps=29.970" -add "%%~na.aac" -new "%%~na_fixed.mp4"

REM Clear up the streams
del *.h264
del *.aac

pause
Gary James wrote on 2/3/2016, 8:40 AM
Timeline Tools has a Re-Quantize to Frames option which aligns all targeted Events to place the Event Start and End locations at the closest rounded down frame position.

JohnnyRoy wrote on 2/3/2016, 8:45 AM
Nick,

Get a copy of Filmic Pro. It will record at a fixed frame rate and avoid this problem from the beginning. It will also give you lots more control than the built-in camera app. They have both iPhone and Android versions.

~jr
john_dennis wrote on 2/3/2016, 10:54 AM

"Timeline Tools has a Re-Quantize to Frames option which aligns all targeted Events to place the Event Start and End locations at the closest rounded down frame position."

But, does Timeline Tools move frames within a file?

john_dennis wrote on 2/3/2016, 10:57 AM
@ jr
If I every get around to shooting video with my iPad, I'll buy Filmic Pro to make the experience a little more controllable.
NickHope wrote on 2/3/2016, 11:55 AM
Timeline Tools doesn't move frames within a file based on my quick test. It just quantizes the ends.

JR, I like the look of Filmic Pro but it sounds from the reviews like the Android version is still new and flaky. I'll give it a whirl when I next shoot.

The other "serious" video camera app I've found on Google Play is Cinema FV-5. Cameras are one of the apps that tend to be unstable on Android, especially with the changes in SD-card write permissions between each version.
Gary James wrote on 2/3/2016, 1:29 PM
I'm not sure what you mean by "move frames". Timeline Tools can move Events, and then Quantize the moved Events. By that I mean it can move Events around, then adjust the Start and End positions to the nearest absolute frame.
john_dennis wrote on 2/3/2016, 1:49 PM
To convert a variable frame rate media file to constant frame rate every frame would have to be inspected for alignment with the fixed frame rate time.
Gary James wrote on 2/3/2016, 9:31 PM
Maybe I'm wrong, but I was always under the impression that once a piece of media was placed into an Event on the Vegas timeline, the media in the Event no longer represented a variable rate. I thought it represented the project as a fixed fps output rate by doing all the inter-frame interpolation when the media was imported.
NickHope wrote on 2/4/2016, 2:15 AM
Before I used the above script, I was setting these clips to constant-fps manually like this:

1. shorten the media to one frame
2. turn off quantize-to-frames
3. zoom right in on the end of the frame, hold down CRTL, and stretch it visually as close as possible to the first frame interval
4. stretch the file back to full length, checking that the frames more or less line up with the timeline's frame intervals
5. switch quantize-to-frames back on
6. snap the end of the file to the nearest frame interval
7. recheck

That approach was actually working, and every frame was lining up and sharp. Which makes me think Gary is right about Vegas converting the file to constant-fps. And if you dump a file on a new timeline and match framerate, Vegas makes the project frame rate match the average framerate of the file and lines the frames up properly.

BUT.... that doesn't help when you have lots of files that all have different average frame rates around 29.970, and you want them all exactly the same frame rate and properly quantized frame-for-frame on a 29.970 timeline.

Can't help thinking there's an easy answer to this somewhere within Vegas or a script. The maths is easy if a script could do the stretching.
wwaag wrote on 2/4/2016, 2:55 PM
@Nick Hope

I've observed a similar problem with MP4 files produced by Mirilli's Action. The frame rates are all over the place--some higher, some lower. Don't know if this will help, but I modified a script written by John Rofrano that I downloaded from this thread https://forums.creativecow.net/thread/24/996404 For frame rates greater than the target rate (I used 60P), an adjustment is computed and then applied to both the Undersample rate and the Playback rate. In my test footage, the individual frame boundaries seemed to match perfectly. Only the last frame had to be "Quantized" using Vegasaur. If the frame rate is less than target rate, only the change to Playback rate is applied. Again, the individual frame boundaries matched perfectly. However, since no additional frames were added, the last few frames of the event were repeats of the first frames in order to keep the event length the same. Again, the last frame required Quantizing using Vegasaur. In any case, FWIW, here is the script.

wwaag

//****************************************************************************
//* Program: Fix30pMedia.cs
//* Author: John Rofrano
//* Description: This script changes the playback rate of 30p media to 29.97
//* Created: July 25, 2010
//* Updated: Jul 28, 2010 (JR) Added Disable Resample
//* Aug 4, 2010 (JR) Changed audio to match video
//* Feb 3, 2016 (JR) Changed to only process selected events
//*
//* Copyright: (c) 2010, 2016 Sundance Media Group / VASST. All Rights Reserved
//****************************************************************************
using System;
using System.Collections;
using System.Windows.Forms;
using Sony.Vegas;

class EntryPoint
{
public void FromVegas(Vegas vegas)
{
int counter = 0;
try
{
foreach (Track track in vegas.Project.Tracks)
{
if (!track.IsVideo()) continue;

foreach (VideoEvent videoEvent in track.Events)
{
// Only process selected events
if (!videoEvent.Selected) continue;

VideoStream videoStream = videoEvent.ActiveTake.MediaStream as VideoStream;

double adjust = 1.0 /(videoStream.FrameRate / 59.94005994005994); // for 30p change to 29.97002997002997

// decimal frameRate = Math.Round((decimal)videoStream.FrameRate, 2);
// only affect 30fps media
// if (frameRate == 30.00m)
// {
// videoEvent.AdjustPlaybackRate(0.999, true);
videoEvent.AdjustPlaybackRate(adjust, true);
if (videoStream.FrameRate > 59.94005994005994) videoEvent.UnderSampleRate = adjust; // for 30p change to 29.97002997002997
videoEvent.ResampleMode = VideoResampleMode.Disable;
counter++;

// check for audio in the same file and change it too
if (videoEvent.IsGrouped)
{
foreach (TrackEvent trackEvent in videoEvent.Group)
{
if (!trackEvent.IsAudio()) continue;

// see if they are from the same file
if (trackEvent.ActiveTake != null && trackEvent.ActiveTake.MediaPath.Equals(videoEvent.ActiveTake.MediaPath))
{
AudioEvent audioEvent = trackEvent as AudioEvent;
audioEvent.AdjustPlaybackRate(adjust, true);
// }
}
}
}
}
}

// let the user know we are done
MessageBox.Show(String.Format("{0} events changed", counter), "Complete", MessageBoxButtons.OK, MessageBoxIcon.Information);
}
catch (Exception e)
{
MessageBox.Show(e.Message, "Unexpected Error", MessageBoxButtons.OK, MessageBoxIcon.Stop);
}
}
}


AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia RTX4070 graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

NickHope wrote on 2/5/2016, 1:35 PM
Thanks wwaag.

To add to my problems, I discovered that Vegas decodes my Note 2 mp4 footage with a repeat frame at the start. I think this comes from the same school of Vegas bugs as the Panasonic 24p decoding bug, and the repeat frames I get decoding FFMpeg-encoded MP4 files.

I also realised that with my mp4box method the audio will probably lose sync as the file plays, depending on how different the media frame rate is from the target frame rate.

I also think it's worth pointing out that the Vegas script method probably won't work well for video that is truly variable rate. It seems that our files are reported as variable frame rate by Properties, Mediainfo etc., but in fact exhibit video of a constant (but non-standard) frame rate when placed on the Vegas timeline. Not sure if that's because Vegas is setting it to constant or if it's really just constant in the original file.

Quick summary of the advantages/disadvantages of the 2 approaches to fixing this:

Vegas script method:
+ Audio should stay in sync
- Some formats require repeat frames to be trimmed (due to Vegas decoding issues)
- Won't work on video that is truly variable frame rate
- Events are stretched

mp4box method:
- Audio probably won't stay exactly in sync
+ No repeat frames at event start on troublesome formats
+ Will work on video that is truly variable frame rate (but such video would then vary in speed)
+ No stretching of events

I took JR's script that you had modified and hacked it around a bit so that you only have to put the target frame rate in once, and the script also sets the project frame rate to that target frame rate before starting (which is useful when you have Vegas set to match the frame rate of the first file dropped on the timeline, as I do).

I then integrated the quantize code from Randall Campbell's old script (see here and here).

It's working but the first version is not frame accurate at the event end. I've posted on the Scripting forum here to see if anyone can help. I'll come back and post a working version here when I have one. I also gave it a new name, as there are so many changes from the originals and it's for a slightly different purpose. I hope nobody minds.
NickHope wrote on 3/11/2016, 5:28 AM

Edit: For a more finished version(s) of this script, see this thread on the scripting forum. In particular Duane N.'s post of 3/13/2016 2:41:48 AM and beyond.

If you need to trim repeat or incorrect frames from the start or end of the time-stretched/quantized events after running the script, you can use the Trim Captured Clips script.

The way to avoid using a time-stretched video stream (and potentially trimming it), and to keep the audio stream in sync is to actually use the clean video stream from the mp4box method in previous posts, and the time-stretched audio stream from the script method. But then of course you'll need 2 versions of each media file while you edit.