Smartphone footage not playing smoothly in Vegas

Richard-Bril wrote on 9/29/2020, 3:32 PM

My problem is that when I import native MP4 video files into Vegas, there is a tiny bit of jitter or stutter (maybe it's a skipped frame) every now and then. It kind of looks like the camera jerked for a second. These jitters only appear once the MP4s are in Vegas, if I play the raw files in VLC (or any other media player) there is no jitter. Some of my video clips might not show any of this jitter in Vegas, while others might have several, it's sporadic…but if there is jitter in the file, it is always at the same place in the clip.

I do all my filming on a Galaxy Note 9 phone in 3840x2160, either in 30fps or 60fps. Lately I've been doing most of my filming using a Movi Freefly gimbal...and when connected to the gimbal I am not given the 60fps option, but 24/30fps (in up to 4K), or 240fps in 1080. Regardless of whether I film while connected via Bluetooth to the gimbal, or just holding the phone and filming natively, I still have this issue once importing the MP4s into Vegas.

I haven't found any solutions that address this specific problem. If I render the raw MP4 files in HandBrake and then take them over to Vegas, it fixes the problem and the jitters are gone, but I lose some of the quality in the video even when using HandBrake's highest quality settings...not to mention the fact that it can make the files sizes about 5 or 6 times larger than the native MP4 files that the phone created. I'd much rather not have to pre-render hundreds of video files for every project I work on in HandBrake before taking them to Vegas, not to mention the loss of quality. Any help would be much appreciated. Thank you.

Back to post

Comments

Richard-Bril wrote on 10/3/2020, 6:23 PM

I want to start again by saying thank you for your time and helping me with this issue!

Before I downloaded your files, it sounded to me that it was simply the case that HB had applied a “levels” conversion to get it close to Studio Levels, recommended for finished files, and appears to be so. This is not what you need as you are going to edit them again. There’s nothing wrong with the HB clip, if you just apply a Levels VFX in VP and adjust the top two sliders to stretch the histogram from 0 to 255 it will have the increased contrast that the other two files have. But it would have been better to not have the levels conversion applied, it may be a setting you selected, or was automatically applied.

I think you're right. Although I didn't see any Levels settings in either HandBrake or Wondershare (which I used to render all 62 of my files, just as a comparison last night). Both the Wondershare and the HB renders look pretty much identical...and seem to add the Levels to it. Using the Levels VFX I was able to get the video pretty close to the original, but I don't mine re-rendering all these clips if we can figure out how to remove the added Levels filter HB and Wondershare seem to add. If you were to try and render a video using HB and get it as close to the quality that your batch files do, what settings would you use?

So the real issue you say is the jitter, which is I think most obvious when panning, otherwise i’m not sure what you mean. If its that then, and others by all means correct me, I think that its not necessarily because of vfr, rather too fast a pan. A high frame rate, say 60 fps + slower pans, can reduce this substantially in acquisition, but too late for that now.

Yes, you're right...the issue IS the jitter, random jitter in the *rendered* videos that Vegas puts out. However, I'm more convinced now than ever that it IS in fact due to the VFR the phone films in...and NOT the panning or camera movement or filming at 30fps. I've done fast pans that don't show any jitter and some completely still shots that do show jitter. I'll share with you a youtube video I put up a few months ago to showcase this better. I shot the whole thing on my Note 9 phone (apart from the stock footage...which has NO jitter at all, obviously). The jitter is minor and it may not bother you (too much) the first time around, but once you see it, you start catching it all over the place...and it's not youtube buffering. The first 2 noticeable places in this video that the jitter shows up in are not in camera pans, but in regular motion. The first one is at 0:27 when the hands pick up the tea cup. The next one that stands out to me is at 0:48 when the rear of the car drives by...it gets fuzzy and jitters a bit there. Throughout the rest of the video there is jitter all over the place which is just a bit of a bummer.

Next I'd like to share some files with you that I filmed with my Note 9, but this time on the Movi gimbal (which provided smoother camera movement).

Here is the original file straight from the phone (this should play in VLC without any jitter or jerky moves, though not the best shot in the world): http://www.riovision.org/bike_original.mp4
Next here is that same file (trimmed) and rendered in VP18 at 1080p, see if you notice the jerk (or jitter) when the bike gets closer to camera...this jerk was not showing in VLC, but was added (or caught) when VP rendered the video after editing: http://www.riovision.org/bike_original_rendered.mp4
Next, here is the same original file, but first rendered (to fix the CFR) in Wondershare. Notice how the jitter/jerk is pretty much unnoticeable: http://www.riovision.org/bike_wondershare_rendered.mp4
Next (best for last), here is the same original file, but first rendered (to fix the CFR) using your batch file. Notice how this fixes the jump/jitter even better than Wondershare did...and how the colors are more popping (maybe because of the Levels that Wondershare adds?): http://www.riovision.org/bike_batch_rendered.mp4

And here is one more example, another file that has a normal camera pan up...but as the camera pans up, VP must catch some low frame rates, because it adds jitter on the way up.

Here's the original file, which if played in VLC shouldn't have any noticeable jitter: http://www.riovision.org/bike2_original.mp4
Now here is the same file (trimmed) and rendered in VP18 at 1080p, notice the jitter as the camera pans up: http://www.riovision.org/bike2_original_rendered.mp4
Next is the Wondershare pre-rendered (to fix CFR)...notice the jitter is pretty much gone, though not as good as the batch file does: http://www.riovision.org/bike2_wondershare_rendered.mp4
And finally...the batch file version, rendered in Vegas after rendering using the batch: http://www.riovision.org/bike2_batch_rendered.mp4

If it was my footage I would simply edit the original VFR footage in vp17/18, once the vfr doesn’t cause a crash, and use proxies. Realistically, people pay far more attention to the content of a video than technical imperfections.

I don't think I've gotten a single Vegas crash in all the editing...just very slow response. I can live with choppy/slow response...I can't live with the jitter caused by the VFR...at least not going forward. If I'm going to continue filming on my phone, either I press the Movi Freefly people to add an Android feature to record in CFR, or I convert all my files after filming (which is where I'm leaning now)...or use the HOS software to rerwap the files without transcoding them (though you did say there is some quality loss when doing that).

If waag would be willing to try and rewrap one of these video files I've uploaded (the original from the phone), I'd really appreciate seeing what they look like once dropped into VP.

If you use vp18, in project properties set Pixel Format to legacy, as the Mediainfo says its Limited range even though its really Full range. You could also change the “flag” to Full, in either the event or media properties when your project properties pixel format is set to full range. That way you can then edit in the VP18 default full range mode.

Thank you, looking forward to trying that as I'm so new to Vegas!

If you’re using vp17 you need to add a levels fx to bring them to about 16-235 in the histogram. Same applies to VP18 using Legacy. See Marcos pinned post on the new 8bit levels setup for vp18. Richard, the HB file was actually corrected for final output, so nothing wrong there.

The Levels flag, Colour Range in the HB file says its limited, correct.

The Levels flag, Colour Range in the "Batch" file says its Full, correct.

The Levels flag, Colour Range in the Original file says its limited, incorrect.

Best of luck.

So do you mean there was nothing I could have done better in the HB renders? The only reason I'd rather use HB over your batch is because of all the data you said (in your next post) that the batch file adds to each file. If you can help me figure out how to do away with the added Levels in HB, I'd appreciate it...OR figure out how to make the batch file not add that much information?

Best Regards

EricLNZ wrote on 10/3/2020, 6:31 PM

Your comment about HB adding a levels adjustment puzzles me as I've never come across this. Has anyone else?

JN- wrote on 10/3/2020, 7:18 PM

There are different ways to go about setting the correct levels within your project. The aim should be to output 16 - 235 which the HB clips are already close to. Assuming you have other types of media, stills etc that has different levels to the HB clips, then just add all of the HB clips to a separate track. Then apply the levels FX to that track, and do the same to any other tracks to match and have the same levels within your project.

So if you set everything to 16 - 235 within your project then your viewing experience will be washed out, so apply temporarily a levels FX at the "Video Output FX" position, (select the levels preset of Studio RGB to Computer RGB) just above the preview window. Doing this means everything looks nice. Remember to remove it before rendering. The perceived wisdom is that players will expand the rendered out 16-235 video clip to 0 - 255 on playback. If you are using VP18 you need to study Marcos pinned post that I gave a link to earlier today.

Ok, so the jitter isn't within the Pan. TBH, in the two places you mentioned, 0:27s and 0:48s if you think its because of the VFR, then I'm not doubting you, so you feel you need to convert to CFR first. My old eyes are obviously no way to judge any of this.

Richard, you can go about this in many ways, but since your PC handles the HB footage more easily, maybe that's the best way.

JN- wrote on 10/3/2020, 7:46 PM

"or use the HOS software to rerwap the files without transcoding them (though you did say there is some quality loss when doing that)."

I don't recollect saying that, if I did it's incorrect, The HOS rewrap and PTS change is lossless. If I knew how to do that within ffmpeg I would use it instead of my re-encoding. The only problem with the HOS system that it's 1 file at a time.

wwaag wrote on 10/3/2020, 7:52 PM

"The only problem with the HOS system that it's 1 file at a time."

No. You can drag an entire folder for processing if you want.

JN- wrote on 10/3/2020, 7:58 PM

@Richard-Bril Anyway, keep trucking, as they say, nice productions, very creative indeed.

Richard-Bril wrote on 10/3/2020, 8:11 PM

If you still cannot get them to work then go for HOS, it's a very nice GUI and no quality loss, although frankly, you're probably looking at a tiny quality difference.

Probably nothing compared to what a re-render does.

Richard-Bril wrote on 10/3/2020, 8:15 PM

@wwaag, would you be willing to run a re-wrap on a single 12 second MP4 file of mine that is VFR so I can see what HOS would look like when used for CFR conversion without transcoding? (No pressure if this is not possible, but if so, here's the link to the file: http://www.riovision.org/01%20Original%20(Note9).mp4 )

Thank you,
Richard

wwaag wrote on 10/3/2020, 8:45 PM

Actually, I had already done that for both clips. Here's a link to the DropBox folder https://www.dropbox.com/sh/7wv7iwupba4gx4s/AADmuP8Jez6sC5zblWd7i-i7a?dl=0

I took a look at both of those clips and found that for 01 Original..., the clip began with 3 duplicate frames. For the bike_original clip, the first 4 frames were duplicates. You can confirm this by "playing" the files in MediaPlayerClassic-Home Cinema. If you begin in the "step" mode you will see the cursor jump frames. Moreoever, each clip displays a really dramatic luminance change around the 10th frame (it brightens). In the bike clip, the change in luminance as he approaches the tree is not gradual--it appears to be stepping.

IMHO, there is something definitely wrong with the camera that has nothing to do with the frame rate being variable. My iPhone shoots VFR and certainly doesn't show such issues. I've rewrapped other Samsung footage and have not seen that before. If it's still under warranty, I'd do some complaining. In all honesty, I don't think that rewrapping or transcoding will significantly help. If you continue using the camera, I'd suggest that you at least trim the first half sec or so, just to eliminate that luminance change. Good luck.

RogerS wrote on 10/3/2020, 9:01 PM

Another idea- maybe there are other video apps for this phone which will do a better job controlling the camera.

Musicvid wrote on 10/3/2020, 9:07 PM

@Richard-Bril

I have done a Quickstream fix on both original files; neither shows any GOP errors, drops, or resync frames.

Which leaves me 95% certain it is the Digital Image Stabilization in your camera. It tries to stabilize the scene, but misses the subject. That, plus converting to Constant Frame rate, may account for your problematic results.

Richard-Bril wrote on 10/4/2020, 4:50 AM

Actually, I had already done that for both clips. Here's a link to the DropBox folder https://www.dropbox.com/sh/7wv7iwupba4gx4s/AADmuP8Jez6sC5zblWd7i-i7a?dl=0

I took a look at both of those clips and found that for 01 Original..., the clip began with 3 duplicate frames. For the bike_original clip, the first 4 frames were duplicates. You can confirm this by "playing" the files in MediaPlayerClassic-Home Cinema. If you begin in the "step" mode you will see the cursor jump frames. Moreoever, each clip displays a really dramatic luminance change around the 10th frame (it brightens). In the bike clip, the change in luminance as he approaches the tree is not gradual--it appears to be stepping.

Thank you so much for doing this, I appreciate it. First of all, I like how HOS doesn't change the filesize, but just fixes the frame rate...and when dropping those 2 clips of yours into VP and putting the original clips above them to do an A/B comparison, the HOS conversions are spot on (with quality, color and lighting), no image degradation at all. Some of the frames were off (by 3-5 frames at times) and I'm guessing that was HOS trying to make due with the duplicate frames, or the dramatic luminance changes. The bottom line is that whether my Samsung Note 9 is the culprit in all of this...with some issue related to the way it captures, or whether it's a "smartphone" issue with VFR, had I fixed the original MP4 files before editing in VP (whether by rewrapping in HOS, or by re-rendering with the Batch file/HB/Wondershare, etc), the "jittery", jerky moves would have been gone from the final product--even though there may have been other issues with lighting and bursts of other things.

IMHO, there is something definitely wrong with the camera that has nothing to do with the frame rate being variable. My iPhone shoots VFR and certainly doesn't show such issues. I've rewrapped other Samsung footage and have not seen that before. If it's still under warranty, I'd do some complaining. In all honesty, I don't think that rewrapping or transcoding will significantly help. If you continue using the camera, I'd suggest that you at least trim the first half sec or so, just to eliminate that luminance change. Good luck.

Thank you again. The phone is not under warranty, but now this makes me want to do some filming tests with my wife's Galaxy S9 (which I believe has the same camera as the Note 9)...and see if I get any of that jittery action. My son has an older S6 and he has tons of problems with his phone footage when he imports it to Vegas. I'll have to ask him again what the issues he has to see if it was the same as mine. Otherwise you may be right...it might be just my phone's camera. Funny thing though, I've had this phone for about a year...I've always thought the camera was great, because when I send videos, they look fine, even when I play them in VLC on a computer, they look fine...but using the phone's camera to film these more "serious" videos that I edited in VP, that's when I noticed this jittery issue.

Cheers

Richard-Bril wrote on 10/4/2020, 4:52 AM

Another idea- maybe there are other video apps for this phone which will do a better job controlling the camera.

Yes, I'll be looking into them. If I want to use the Movi Freefly Gimbal (that cost $300), I kind of need to connect to it via Bluetooth...and then it takes over all the camera's settings, so I wouldn't be able to use another app. This jittery problem with my footage was taking place before using the Movi Gimbal though.

Richard-Bril wrote on 10/4/2020, 4:55 AM

@Richard-Bril

I have done a Quickstream fix on both original files; neither shows any GOP errors, drops, or resync frames.

Which leaves me 95% certain it is the Digital Image Stabilization in your camera. It tries to stabilize the scene, but misses the subject. That, plus converting to Constant Frame rate, may account for your problematic results.

I just looked up the image stabilization on my phone. When it's connected to the Movi gimbal...and running the Movi app, the app turns off both Optical image stabilization and the Electronic image stabilization.

Richard-Bril wrote on 10/4/2020, 6:34 PM

@Richard-Bril

"Research a way to record in CFR with my Note 9 (or at least get the variable gap to be less)"

Take a look at FILMic Pro. https://www.filmicpro.com/filmicpro/

It will give you a lot more control over your recordings including frame rate options. I see that they now have an Android version. For little $, it's very worthwhile IMHO.

Thank you for that suggestion. I initially blew off this idea, because I thought that using another 3rd party app for the camera would override the ability to use Movi's app. But I just looked Filmic up and now I remember their product which I saw a few months ago. They support several gimbals, the Movi being one of them...so you can run *both* Movi's app AND Filmic's app simultaneously. I just installed the Filmic Pro Evaluation app on my phone to see if I could get a confirmation as to whether they record in CFR. There is no mention of CFR, though they list the same frame rate options for my phone as the Movi app has; 24/30fps (up to 4K) and 240fps at 1080. Natively my phone doesn't offer 24fps, but it does have 30/60fps. Bummer that both Movi's app and FilmIC doesn't utilize my phone's ability to record in 60fps (which I would do most of the time since I don't usually know which shots will be edited in slow motion once in the editing studio). Have you heard that FilmIC Pro does in deed record in CFR on all phones?

wwaag wrote on 10/4/2020, 6:55 PM

@Richard-Bril

"Have you heard that FilmIC Pro does in deed record in CFR on all phones?" As I recall, I contacted FilmIC with that same question for my iPhone and the answer was "no". It was still VFR, but the frame variability was greatly reduced (at least that's what they claimed). The wide variation that you see reported in MediaInfo--E.g. 7 fps--is a result of duplicate frames at the beginning of your clips. If you simply trim the first half sec or so, it's really not that much of an issue so long as you don't have really wide luminance variations in your scene.

Former user wrote on 10/4/2020, 7:07 PM

Filmic Pro on android does VFR and wwag says it's the same with iphones. Some people state Filmic Pro gives a constant but incorrect frame rate, but it's likely their editor or some other piece of reporting software misreporting a variable range of fps's and instead giving a single value which might be an average or might not be

Richard-Bril wrote on 10/4/2020, 11:24 PM

This is the internal GOP layout of the 3 files ...
[ 01 Original (Note9).mp4 ] GOP length = [ 30 ]

IPPPPPPPPPPPPPPPPPPPPPPPPPPPPPI

[ 02 Batch (19.5).mp4 ] GOP length = [ 71 ]

IBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBPI

[ 03 HandBrake (HQ).m4v ] GOP length = [ 10 ]

IPPPPPPPPPI

So, looking at the 02 Batch ... .mp4 file, for sure, more work needs to be done all the time in VP.

Is there a tweak or modification to your batch files that can change these (GOP lengths) to be more in line with the HandBrake ones?--or would that degrade the video quality? Do you think it's those additional IBP's that makes your batch renders look richer and better than HB...and without the "Levels" type of look that HB adds?

JN- wrote on 10/5/2020, 9:09 AM

@Richard-Bril “Do you think it's those additional IBP's that makes your batch renders look richer and better than HB...and without the "Levels" type of look that HB adds?”

No. I already stated that the washed out look is because the HB levels are not full range, it only requires a small adjustment with a levels FX to line it up with the original clips Full Range values. I don’t know why HB output is doing this, it’s not that important.

Is there a tweak or modification to your batch files that can change these (GOP lengths) to be more in line with the HandBrake ones?--or would that degrade the video quality?

My speculation that the GOP layout is causing the slower playback is only that, speculation. Having say a Nvidia GPU with VP's decoding capabilities is really like night and day though. I go from about a few fps to full 29.97 fps with the decoding being set off or on, using my GTX 1070 on my laptop. If you had a modern GPU, there would be no need to look at any of this as all of the files would be fine for playback speed on the timeline.

I've added two batch files in the Richard Bril zip within the original zip. The output GOP is the same as the HB GOP. The levels is full range, unlike the HB files which are slightly less than. Obviously there may be small differences in the encoding between my batch files and HB, you can see that in the mediainfo encoder settings, but the quality in the ones I just added should be fine, with a crf of 20, to roughly match your original file size. I would still use something like 17 or smaller if I was doing this for myself

For fixing vfr and keeping full quality, surely the way to go is by using wwaags hos, given the video files can be done in batch by processing a complete directory of files at once. The only benefit of my approach is its simpler, fewer steps, and no cost, but without the original quality.

Unless I’m missing something, haven’t you already established that it’s not a VFR issue, rather the jitter may be caused by a phone issue? If so why not just use the originals in your project. Can you then do something in VP at the point of jitter, say cut a few frames out, maybe do a cross fade then?

Going forward, maybe use something like the New Sony ZV1, there are many Olympus MFT cameras also with better IBIS.

Richard-Bril wrote on 10/6/2020, 5:09 AM

I've added two batch files in the Richard Bril zip within the original zip. The output GOP is the same as the HB GOP. The levels is full range, unlike the HB files which are slightly less than. Obviously there may be small differences in the encoding between my batch files and HB, you can see that in the mediainfo encoder settings, but the quality in the ones I just added should be fine, with a crf of 20, to roughly match your original file size.

Thank you again. Would you please point me to the zip file. I tried the first zip file in your first post to this thread, but I didn't see any new batch files in it. Thanks

JN- wrote on 10/6/2020, 6:07 AM

@Richard-Bril Your right, it's not there, I was certain that I had added it yesterday. Anyway it's there now "Richard Bril.zip" beside the VFR2CFR.zip. I'm curious to know if the files you convert with it are also easier to use in your project, i.e. similar in ease of use to the HB ones, cheers.

I did a small mod to the original 03 ... .bat and 03a ... .bat, those two use Nvidia HW encoding.

Richard-Bril wrote on 10/9/2020, 5:52 AM

I'm curious to know if the files you convert with it are also easier to use in your project, i.e. similar in ease of use to the HB ones, cheers.

Hey my friend, sorry for the tardy reply (and thank you) for your custom batch files...just for me! What a treat! I've been busy with a lot of things and I knew I would need to take the time to properly compare your new batch files and the resulting renders (CFR conversions) they would create using the same "original" files I converted using your original batch files several days ago. The answer is YES, there is a BIG difference! The new batch files you modified seem to help the video clips play MUCH more smoothly in VP. I rendered about 100 files with your new batch files that I'm using in a project (because I wanted to see a larger project using many MP4s to get a proper feel) with a quality setting of 16 (a step up in quality from the default 17 setting you have the batch defaulted at). I was surprised to see the resulting files come out smaller than the files I converted using your original batch files with a setting of 17. I thought the 16 setting I used with the new batch files would result in bigger files. Anyways, I took the files over to my son's computer...since he has a gaming computer with a 6gb NVideo GPU and it's pretty fast for editing, I thought it would be better to test the difference there first. The first set of converted files I created (several days ago) with your original batch files slowed his computer down so much that we had to disable thumbnails to play the project (and when you enable the thumbnails, it takes about 1 whole minute to populate them). Even with the thumbnails/preview disabled, the project still played quite choppy. With the new CFR files, the project plays very smoothly...a stark contrast.

I then took all the files back to my studio computers and did the same tests...and had the same results. I took a little more time after that to try and figure out what was up with the smaller file sizes, I then noticed that the new batch files you created (the Richard ones) have 2 lines of code (one on top of the other) for the settings. I then created 3 different CFR conversions each with different quality settings (however using the first line in the code to edit the quality setting)...and I noticed all 3 versions came out the exact same size. That was when I saw the 2nd line of code (which was set to 20...I suppose by you). So I'm confused which line(s) these new batch files need the user to modify or edit? If the first line of code is set to a quality setting of 16, and the bottom one is set to 20, what quality setting gets used?

I'm also curious...in a nutshell what is the difference between your original batch files and these new "Richard" ones?

Thanks again,

Richard

Musicvid wrote on 10/9/2020, 6:11 AM

Sounds like "Filmic Pro" has some work that needs to be done . . .

JN- wrote on 10/9/2020, 7:18 AM

@Richard-Bril Hi Richard, The 1st. ffmpeg line is remmed out, not active. Anything after a REM is simply a remark, a comment. I just left the old line in. The 2nd. line without a REM is what does the processing, and I left it at a -crf of 20 because you said previously that you used 19.5. Sixteen is what I would use.

I'm also curious...in a nutshell what is the difference between your original batch files and these new "Richard" ones?

I removed ‘B' frames and shortened the GOP length, with a view to your cpu not having to do as much processing/translating, as you use, edit and play, preview within the project.

This is a visualization of the GOP's after converting to CFR, both using a -CFR value of 16 ...

------------------------------------------------------------------------
Filename ..... [ 01 Original (Note9)-[CFR]-NEWER.mp4 ]
GOP length ... [ 10 ]
IPPPPPPPPPI
------------------------------------------------------------------------
Filename ..... [ 01 Original (Note9)-[CFR]-OLDER.mp4 ]
GOP length ... [ 71 ]
IBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBBBPBPI
------------------------------------------------------------------------

So I'm delighted that it did work, thanks for the feedback, as I previously mentioned, on a higher specced machine, using VP's decoding, you simply wouldn’t see any difference between using the older or newer batch files.