Smartphone footage not playing smoothly in Vegas

Richard-Bril wrote on 9/29/2020, 3:32 PM

My problem is that when I import native MP4 video files into Vegas, there is a tiny bit of jitter or stutter (maybe it's a skipped frame) every now and then. It kind of looks like the camera jerked for a second. These jitters only appear once the MP4s are in Vegas, if I play the raw files in VLC (or any other media player) there is no jitter. Some of my video clips might not show any of this jitter in Vegas, while others might have several, it's sporadic…but if there is jitter in the file, it is always at the same place in the clip.

I do all my filming on a Galaxy Note 9 phone in 3840x2160, either in 30fps or 60fps. Lately I've been doing most of my filming using a Movi Freefly gimbal...and when connected to the gimbal I am not given the 60fps option, but 24/30fps (in up to 4K), or 240fps in 1080. Regardless of whether I film while connected via Bluetooth to the gimbal, or just holding the phone and filming natively, I still have this issue once importing the MP4s into Vegas.

I haven't found any solutions that address this specific problem. If I render the raw MP4 files in HandBrake and then take them over to Vegas, it fixes the problem and the jitters are gone, but I lose some of the quality in the video even when using HandBrake's highest quality settings...not to mention the fact that it can make the files sizes about 5 or 6 times larger than the native MP4 files that the phone created. I'd much rather not have to pre-render hundreds of video files for every project I work on in HandBrake before taking them to Vegas, not to mention the loss of quality. Any help would be much appreciated. Thank you.

Comments

JN- wrote on 9/29/2020, 3:38 PM

@Richard-Bril Hi Richard, strong possibility that the files are variable frame rate. If you convert them to constant frame rate your problems should go away. Handbrake can do the conversion for you. I made up some simple batch files that do that also here. But I’m getting ahead of myself. You need to post mediainfo of the files to confirm that they are vfr, maybe the problem is something else.

Later versions of VP, 17 and 18 handle vfr without conversion better than earlier versions. What VP version are you using?

rraud wrote on 9/29/2020, 3:56 PM

This has been discussed quite a few times, usually smart phones encode VFR (variable frame rate) files.. which VP does not 'like'.. Transcoding the media to a CFR (constant frame rate) allows smooth playback. Handbrake and HOS (Happy Otter Scripts) seem to be popular for transcoding amongst VP users. Search this forum for comments from the expert video folks (which I am not).

Richard-Bril wrote on 9/29/2020, 4:43 PM

Thank you for the quick answers.

I'm using VP 17 Build 452
How do I verify (or post) whether the files are vfr or not?
Thanks for the batch files, I'm reading your text file now to try and figure out how to use them. (Do your batch files *only* adjust the frame rate, leaving the video quality the same? I haven't been able to get a HandBrake render of my files to look the same quality as the original...and I've been using HandBrake for years.)

Kind Regards

j-v wrote on 9/29/2020, 4:51 PM

How do I verify (or post) whether the files are vfr or not?

https://www.vegascreativesoftware.info/us/forum/faq-how-to-post-mediainfo-and-vegas-pro-file-properties--104561/

Richard-Bril wrote on 9/29/2020, 5:01 PM

So I need to download and install MediaInfo in order to share this info?

j-v wrote on 9/29/2020, 5:07 PM

Yess if you want a possible solution for your problem.

Richard-Bril wrote on 9/29/2020, 5:18 PM

Thank you for your patience. Here is the MediaInfo for one of the native MP4 files coming from my Galaxy Note 9 phone:

General
Complete name                            : C:\Users\Trade Slave 1\Desktop\Raw Footage\Done it to Me\Movi_2020-09-14_11-43-10.mp4
Format                                   : MPEG-4
Format profile                           : Base Media / Version 2
Codec ID                                 : mp42 (isom/mp42)
File size                                : 178 MiB
Duration                                 : 12 s 450 ms
Overall bit rate                         : 120 Mb/s
Encoded date                             : UTC 2020-09-14 18:43:10
Tagged date                              : UTC 2020-09-14 18:43:10
com.android.version                      : 10

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L5.1
Format settings                          : CABAC / 1 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 1 frame
Format settings, GOP                     : M=1, N=30
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 12 s 450 ms
Bit rate                                 : 120 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 29.559 FPS
Minimum frame rate                       : 7.927 FPS
Maximum frame rate                       : 39.543 FPS
Standard                                 : NTSC
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.489
Stream size                              : 178 MiB (100%)
Title                                    : VideoHandle
Language                                 : English
Encoded date                             : UTC 2020-09-14 18:43:10
Tagged date                              : UTC 2020-09-14 18:43:10
Color range                              : Limited
colour_range_Original                    : Full
Color primaries                          : BT.709
colour_primaries_Original                : BT.601 PAL
Transfer characteristics                 : BT.709
transfer_characteristics_Original        : BT.601
Matrix coefficients                      : BT.709
matrix_coefficients_Original             : BT.470 System B/G
mdhd_Duration                            : 12450
Codec configuration box                  : avcC

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : mp4a-40-2
Duration                                 : 12 s 423 ms
Bit rate mode                            : Constant
Bit rate                                 : 128 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 44.1 kHz
Frame rate                               : 43.066 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 194 KiB (0%)
Title                                    : SoundHandle
Language                                 : English
Encoded date                             : UTC 2020-09-14 18:43:10
Tagged date                              : UTC 2020-09-14 18:43:10

Richard-Bril wrote on 9/29/2020, 5:33 PM

So obviously there is some serious "variableness" in the frame rate of my Note 9 files. Funny how VLC and other media players play these with no jitter, but Vegas (17 at least) will not.

The FR reading in this file is: 29.559 FPS, but the minimum/maximum are so far off from that (7.927 & 39.543), I'm surprised.

I checked some of the HandBrake renders I made of this clip (which I set to 29.97 in HandBrake), but MediaInfo shows them as Variable...although very close (min 29.821 & 29.831)...which I suppose is "close enough" for Vegas to play smoothly? Is it considered Constant when the min/max are that close? My stock footage from StoryBlocks is showing as Constant with no min/max.

JN- wrote on 9/29/2020, 5:43 PM

@Richard-Bril "Do your batch files *only* adjust the frame rate, leaving the video quality the same?"

No, (for say "01 VFR2CFR-CPU encoding-Output to h264-Win Drag and Drop.bat") the video is re-encoded, the default crf value I set is 17, which is going to give you a bigger file than the original but good quality, change to lower for higher quality and bigger files. Change the -r default from 29.97 to whatever fps value you require.

"Funny how VLC and other media players play these with no jitter, but Vegas (17 at least) will not."

Players are not NLE's, so thats the way it is. NLE's have stricter, more exacting file input criteria.

wwaag wrote on 9/29/2020, 7:06 PM

Anytime you "transcode" you lose some quality since the media file is re-rendered. It is far better to "rewrap" your media file into a new container and convert from variable frame rate to constant frame by adjusting the Presentation Time Stamps. Here's an example using the ImportAssist tool in HappyOtterScipts in which the iPhone11 media file is UHD HEVC at 60fps. As you can see, it's very quick compared to transcoding plus no loss in quality.

https://vimeo.com/414271488/b80168621d

Richard-Bril wrote on 9/29/2020, 7:08 PM

Thanks, JN. I tried a few times to follow the instructions for using the batch files, but couldn't get them to work. (Maybe I need a simplified step-by-step tutorial.) Do you think HandBrake does an equal (or better/worse) job than the batch files? I suppose there is no software option to only change the FR to be constant without changing anything else in the video file?

Do you think VP18 would play my MP4 files that range from 7.92fps to 39.43fps---without any jitter?

Now that I have MediaInfo installed, I did a bunch of tests with all the camera settings on my phone (and my wife's) and they pretty much all have a wide gap of FPS, but none of them went as low as the clip I posted above (min of 7.927).

Richard-Bril wrote on 9/29/2020, 7:18 PM

Thanks, wwaag, HOT looks like a great solution to this problem without suffering from any quality loss by re-renders. Does the free license version do the "rewrap" you described?

RogerS wrote on 9/29/2020, 7:25 PM

"Do you think VP18 would play my MP4 files that range from 7.92fps to 39.43fps---without any jitter?"

Others are having rendering issues with variable HEVC phone videos in VP 18. Rather than upgrading Vegas, I'd try to get them into a different format for editing.

Musicvid wrote on 9/29/2020, 7:40 PM

@Richard-Bril

To clear up a couple of things:

VLC is a player. Vegas is a decoder / encoder. It's not a curiosity that VLC plays them as intended. VFR support was introduced in Vegas 18.

Minimum frame rate : 7.927 FPS
Maximum frame rate : 39.543 FPS This is variable frame rate

min 29.821 & 29.831 This is fixed frame rate from MediaInfo being too picky over rounding jitter from Handbrake's 90khz clock frequency, which creates an irrational dividend. It's good to recognize the difference.

Hope this helps your understanding; I routinely use Handbrake, but the ffmpeg / HOS rewrap saves an encoding step, which others have pointed out is lossy. In real life, there is little for non-obsessives to worry over.

wwaag wrote on 9/30/2020, 12:58 AM

@Richard-Bril

"Does the free license version do the "rewrap" you described?". No, but there is a 30 day trial license just to see if it works plus there are quite a few other tools. Having said that, pretty much all of the tools within HOS are "open source". It's just the custom interface to Vegas that sets HOS apart.

Richard-Bril wrote on 9/30/2020, 2:07 AM

@Musicvid, thanks for the clarifications. I installed the trial version of VP18 a few hours ago...just to see if it would fix the VFR issue. I took a single MP4 file (the one I posted the MediaInfo for above) and imported it into Vegas, unfortunately the jittery action is still there at the exact same places.

I'd be interested to read up more on the ffmpeg/HOS rewrap method you described. HOS is the paid add-on, right?--but how do you use ffmpeg *and* HOS in tandem? Thanks

RogerS wrote on 9/30/2020, 2:28 AM

HOS passes data from Vegas to various tools like ffmpeg- it's integrated.

Richard-Bril wrote on 9/30/2020, 2:55 AM

Is there a free option to rewrap media files (in my case 4K MP4s from phone/camera) into a new container and convert from VFR to CFR without transcoding?

Marco. wrote on 9/30/2020, 3:09 AM

"Is there a free option to rewrap media files"

Yes, FFmpeg offers rewrapping.

Richard-Bril wrote on 9/30/2020, 3:37 AM

"Is there a free option to rewrap media files"

Yes, FFmpeg offers rewrapping.

If you would please point me to where I can read up and learn how to do this using FFmpeg (rewrap and convert VFR to CFR without transcoding), I'd appreciate it. Thank you

Marco. wrote on 9/30/2020, 4:20 AM

Sorry, I was wrong. It seems it is not possible to set VFR to CFR without a re-encoding.

JN- wrote on 9/30/2020, 6:28 AM

@Richard-Bril "I tried a few times to follow the instructions for using the batch files, but couldn't get them to work."

The readme file includes the following instructions ...

You need to download and install say a static version of ffmpeg and add it to the windows PATH.
Example ... https://www.gyan.dev/ffmpeg/builds/

Adding the ffmpeg "BIN" folder to the "PATH" ...

Search for "env" in Windows "search" ...
or Select "Settings" "System", then Search ...

Select "Edit the System enviornment variables" ... Not "Edit enviornment variables for your account"
With the "advanced" tab selected click the "Enviornment Variables button at the bottom right ...

If you need more step by step then try googling .. adding/editing items to/in the Windows PATH.

Once you have done that then you drag and drop your video clip(s) onto the batch files.

I've added a basic error trapping to the 8 batch files in the zip, since the previous download that you have. This will tell you what's up/missing when you drag and drop and if you haven't got ffmpeg.exe available. So maybe download the new zip at the previous given link.

Use this 01 ... .bat below.

01 VFR2CFR-CPU encoding-Output to h264-Win Drag and Drop.bat

If you have a nvidia gpu installed then you can use say 03 ... .bat.

If you still cannot get them to work then go for HOS, it's a very nice GUI and no quality loss, although frankly, you're probably looking at a tiny quality difference. To get some idea of the quality loss, I compared the finished re-encoded file below, to the HOS re-wrapped file, now used as "source" file. Comparing the 2 files to the original source isn't practical as it has a VFR, whereas these two files are now conformed to CFR.

RE-Encoded file below ...

SSIM All........ 0.988465 (19.379951)
PSNR Average ... 44.332059
VMAF ........... 99.870394

Legend ... [ SSIM 0.95 to 0.99 = good ] ... [ PSNR 38 = good, 45+ = Excellent no artifacts ] ... VMA F [ 80 to 100 = excellent. ]

So these are still high values, I used a -crf value of 17.

Richard-Bril wrote on 9/30/2020, 2:22 PM

Thank you, JN. I was able to work with your batch files last night and they are *great!* I found that a setting of -crf 19.5 resulted in a file size nearly the same as my Note 9 native footage...and once in Vegas, doing an A/B comparison with the original MP4, there is nearly *no* discernible difference in quality. Thank you! (I suppose I could have gotten a similar result with HandBrake had I spent more time tweaking. Do you think the HandBrake option of doing a 2-pass would yield a higher quality (at the same file size) video file?)

Thanks to all of your input, here are my take-aways so far:

My smartphone (Samsung Galaxy Note 9) records video in VFR (as do most smartphones?)
Vegas Pro (even version 18) doesn't take kindly to VFR footage
In my case I only noticed the *slight* jitter in my VP *edited* videos (of my Note 9 footage) after spending some time watching them. I initially thought maybe the camera had jerked for a fraction of a second during the filming process. I also thought maybe it was my slow computer just trying to "catch up" with the video file when playing back the rendered version of my videos (in VLC or other media players). For a minute (after uploading to youtube) I also thought it was maybe a youtube buffering issue where the video was trying to catch up every now and then. I realized it was none of the above...because when I played the raw video clips in VLC (or other media players) those "jitters" were not there.
Once I had MediaInfo installed, I ran a bunch of my native MP4 files through it to see what exactly was going on with the frame rate. I found that my (Note 9) 4K footage shot at 30fps went as low as 5fps and up to 36fps. My 4K 60fps footage ranged between 18fps and 65fps. My 1080p 240fps footage ranged between 5fps and 243fps. I also ran some of my GoPro footage through MediaInfo and found that the 1080 60fps footage was showing between 28fps and 63fps.
My semi-formed conclusions at this point are that Vegas especially doesn't like the footage that dips way down...maybe below the 10fps threshfold, or further away than maybe 50% of the set frame rate. My estimate is that about 50% of my Note 9 MP4 clips have some slight "jitter" (that only show up in Vegas) at specific places in the clip...and they are only really noticeable when the camera is moving or doing a smooth pan or sweep. Half of my clips don't seem to have these jitters in Vegas at all. For some reason I haven't noticed *any* of these jitters with my 240fps (1080p) footage (which my Movi Freefly gimbal has the option to film in even though my Note 9 doesn't natively have that option)...which I use for super slow motion once in Vegas. I haven't noticed any jitter from the GoPro footage either...though the GoPro dips down to nearly half of the set frame rate according to MediaInfo. I just get the feeling that VP will only produce these jitters when the FR is extremely low at certain parts of the video clip.
Solutions:

Research a way to record in CFR with my Note 9 (or at least get the variable gap to be less)
If that is not possible, rewrap the video files before editing in VP, preferably by *only* converting the frame rate from VFR to CFR, but (from what I understand) this is only possible with the paid HOS software. Otherwise, rewrap the files by *both* converting the frame rate *and* transcoding the files (which will degrade the video quality somewhat because they will be re-rendered).

Kind Regards,
Richard

wwaag wrote on 9/30/2020, 2:34 PM

@Richard-Bril

"Research a way to record in CFR with my Note 9 (or at least get the variable gap to be less)"

Take a look at FILMic Pro. https://www.filmicpro.com/filmicpro/

It will give you a lot more control over your recordings including frame rate options. I see that they now have an Android version. For little $, it's very worthwhile IMHO.