Sony Camera & Vegas Proxy Issue

jimingo-1 wrote on 3/12/2023, 3:22 PM

I'm using both the Sony FX3 and Sony a7iv. With these cameras, it's possible to record proxies in camera.

I've been using Sony proxies for a few years and just started to notice this so this may be just be a Vegas 20 issue but when I swap the proxy out for the high quality clip, I do not get frame accurate swaps. The proxy and the high resolution clip is always one frame off. If I use the proxy clip to the very end and then swap out for the high res clip, the high res clip loops to the first frame. I thought maybe the issue was with Sony or the file types that it's recording but if I bring the same clips in to Premiere, they are frame accurate. I have no issues in Premiere.

I've tested this issues on 4 different computers (2 AMD and 2 Intel with different graphics cards) and this issue persists.

Is there a fix to this or is this a known bug? It's crazy annoying because my projects usually have hundreds of cuts so that means I have to go through and adjust hundreds of clips to make them frame accurate. Maybe there's a script or plug-in I can use for this?

Comments

Howard-Vigorita wrote on 3/12/2023, 4:25 PM

Any chance you can share sample footage? Not clear if your camera writes dual clips or adds the proxy as a 2nd video stream that needs to be extracted.

jimingo-1 wrote on 3/12/2023, 4:30 PM

No problem, I will upload some sample clips tonight. The FX3 and A7iv both write dual clips.

jimingo-1 wrote on 3/12/2023, 5:09 PM

Here's a WeTransfer link to a UHD 23.976 clip and it's proxy.

https://we.tl/t-GRLmpOluKb

Howard-Vigorita wrote on 3/12/2023, 6:05 PM

Well that's weird. The audio and video in the regular clip shows as 32 s 32 ms in MediaInfo. The video in the proxy is the same length but the audio is longer: 32 s 43 ms.

Simple work around might be to create your Vegas projects with the regular clips first, rather than with the proxies. Then Substitute the proxy in for edits and Substitute back for final renders. That should cause Vegas to size clips on the timeline based on the regular clips and truncate the proxy clips to the size they should be when you substitute them in for edit. If you've already done too much editing to want to start over, turning looping off in clip properties might make the mismatch look less noticeable... I think that would double the last video frame rather than go back to the first... although there might be an audible glitch in the audio.

Former user wrote on 3/12/2023, 6:31 PM

@jimingo-1 on a Vegas timeline you can see they are out of sync by 1 frame

But in a media player on any NLE that's not Vegas it is in sync

(left is proxy)

john_dennis wrote on 3/12/2023, 6:33 PM

@jimingo-1

They're displayed differently on my machine if you stack them on the timeline.

General
  Name: FX3_ONE_20221104_8351 (Proxy).MP4
  Folder: C:\Users\John\Downloads\23.976 FX3 Sample Footage\23.976 Sample
  Type: AVC
  Size: 33.04 MB (33,831,297 bytes)
  Created: Friday, November 4, 2022, 5:56:14 PM
  Modified: Sunday, March 12, 2023, 4:16:41 PM
  Accessed: Sunday, March 12, 2023, 4:16:58 PM
  Attributes: Archive

Streams
  Video: 00:00:32.032, 23.976 fps progressive, 1280x720x32, AVC
  Audio: 00:00:32.032, 48,000 Hz, Stereo, AAC

Summary
  [TCFM]: 34359738386

ACID information
  ACID chunk: no
  Stretch chunk: no
  Stretch list: no
  Stretch info2: no
  Beat markers: no
  Detected beats: no

Other metadata
  Regions/markers: no
  Command markers: no

Media manager
  Media tags: no

Plug-In
  Name: so4compoundplug.dll
  Folder: C:\Program Files\VEGAS\VEGAS Pro 20.0\FileIO Plug-Ins\so4compoundplug
  Format: AVC
  Version: Version 1.0 (Build 8532)
  Company: MAGIX Computer Products Intl. Co.

General
Complete name                            : C:\Users\John\Downloads\23.976 FX3 Sample Footage\23.976 Sample\FX3_ONE_20221104_8351 (Proxy).MP4
Format                                   : XAVC
Codec ID                                 : XAVC (XAVC/mp42/iso2)
File size                                : 32.3 MiB
Duration                                 : 32 s 43 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 8 446 kb/s
Encoded date                             : UTC 2022-11-04 21:55:41
Tagged date                              : UTC 2022-11-04 21:55:41

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L4
Format settings                          : CABAC / 2 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 2 frames
Format settings, GOP                     : M=1, N=6
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 32 s 32 ms
Bit rate mode                            : Variable
Bit rate                                 : 5 964 kb/s
Maximum bit rate                         : 9 999 kb/s
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.270
Stream size                              : 22.8 MiB (71%)
Encoded date                             : UTC 2022-11-04 21:55:41
Tagged date                              : UTC 2022-11-04 21:55:41
Color range                              : Full
Metas                                    : 3
Codec configuration box                  : avcC

Audio
ID                                       : 2
Format                                   : AAC LC
Format/Info                              : Advanced Audio Codec Low Complexity
Codec ID                                 : mp4a-40-2
Duration                                 : 32 s 43 ms
Bit rate mode                            : Constant
Bit rate                                 : 128 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 46.875 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 501 KiB (2%)
Encoded date                             : UTC 2022-11-04 21:55:41
Tagged date                              : UTC 2022-11-04 21:55:41

General
  Name: FX3_ONE_20221104_8351.MP4
  Folder: C:\Users\John\Downloads\23.976 FX3 Sample Footage\23.976 Sample
  Type: HEVC
  Size: 229.64 MB (235,155,315 bytes)
  Created: Friday, November 4, 2022, 5:56:14 PM
  Modified: Sunday, March 12, 2023, 4:16:42 PM
  Accessed: Sunday, March 12, 2023, 4:17:19 PM
  Attributes: Archive

Streams
  Video: 00:00:31.990, 23.976 fps progressive, 3840x2160x32, HEVC
  Audio: 00:00:32.035, 48,000 Hz, Stereo, PCM

Summary
  [TCFM]: 34359738386

ACID information
  ACID chunk: no
  Stretch chunk: no
  Stretch list: no
  Stretch info2: no
  Beat markers: no
  Detected beats: no

Other metadata
  Regions/markers: no
  Command markers: no

Media manager
  Media tags: no

Plug-In
  Name: so4compoundplug.dll
  Folder: C:\Program Files\VEGAS\VEGAS Pro 20.0\FileIO Plug-Ins\so4compoundplug
  Format: AVC
  Version: Version 1.0 (Build 8532)
  Company: MAGIX Computer Products Intl. Co.

General
Complete name                            : C:\Users\John\Downloads\23.976 FX3 Sample Footage\23.976 Sample\FX3_ONE_20221104_8351.MP4
Format                                   : XAVC
Codec ID                                 : XAVC (XAVC/mp42/iso2/nras)
File size                                : 224 MiB
Duration                                 : 32 s 35 ms
Overall bit rate                         : 58.7 Mb/s
Encoded date                             : UTC 2022-11-04 21:55:41
Tagged date                              : UTC 2022-11-04 21:55:41

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Format Range@L5@High
Codec ID                                 : hvc1
Codec ID/Info                            : High Efficiency Video Coding
Duration                                 : 32 s 32 ms
Bit rate                                 : 50.1 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (24000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:2
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.252
Stream size                              : 191 MiB (85%)
Encoded date                             : UTC 2022-11-04 21:55:41
Tagged date                              : UTC 2022-11-04 21:55:41
Color range                              : Full
Metas                                    : 3
Codec configuration box                  : hvcC

Audio
ID                                       : 2
Format                                   : PCM
Format settings                          : Big / Signed
Codec ID                                 : twos
Duration                                 : 32 s 35 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 536 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Bit depth                                : 16 bits
Stream size                              : 5.87 MiB (3%)
Encoded date                             : UTC 2022-11-04 21:55:41
Tagged date                              : UTC 2022-11-04 21:55:41

 

Former user wrote on 3/12/2023, 6:43 PM

and here's Resolve showing they are in sync. All my examples are of frame 123 because it's fast moving and will show out of sync issues well

Former user wrote on 3/12/2023, 7:14 PM

I see the problem, both files should be 768 frames, but in Vegas the proxy file is 769 frames, this is because Vegas is duplicating the first frame, throwing off the sync between the 2 files by 1 frame. This is a bug

john_dennis wrote on 3/12/2023, 7:19 PM

The Proxy and main video from my Sony RX10-IV don't sync audio in Vegas Pro 20-326 even though the files appear the same length on the timeline and are both AVC.

This stuff is driving me to drink.

2023-03-12 2107 PST Replaced video with a better example and had another drink.

Howard-Vigorita wrote on 3/12/2023, 8:42 PM

Digging deeper into the frame data, it looks like the reason the clips yield a different duration is because the start offset is set to 0 in proxy file but set to 1 in the regular file.

regular file:

channel # 0
 --- codec: HEVC
 --- content width: 3840
 --- content height: 2160
  --- scan: Progressive
 --- colorspace: YCbCr
 --- subsampling: 422
 --- bit rate: 50054649
 --- bit depth: 10
 --- frame count: 768
 --- start offset: 1
 --- duration (sec): 31.9903

proxy file:

channel # 0
 --- codec: AVC
 --- content width: 1280
 --- content height: 720
 --- frame rate: 23.976
 --- scan: Progressive
 --- colorspace: YCbCr
 --- subsampling: 420
 --- bit rate: 5964276
 --- bit depth: 8
 --- frame count: 768
 --- start offset: 0
 --- duration (sec): 32.032
Former user wrote on 3/12/2023, 8:55 PM

--- frame count: 768

@Howard-Vigorita So there's 768 frames, and I wasn't including frame 0. I"ll edit my reply,,, but with my computer not using any hardware decoder with Vegas I get 768frames from main camera file, and 769 from proxy

john_dennis wrote on 3/12/2023, 10:12 PM

@jimingo-1

I ran side-by-side tests of your sample files:

I also ran my RX10-IV:

My video syncs, audio... not so much.

Howard-Vigorita wrote on 3/12/2023, 10:18 PM

Just ran ffprobe on the regular clip to examine frame #1:

ffprobe -hide_banner -show_frames -select_streams v  -read_intervals "%+#1" -i 'FX3_ONE_20221104_8351.mp4'

It throws error messages about a missing keyframe before the 1st frame. The proxy file doesn't do that. I think that's the source of the offset. Looks like it can be cured by rewrapping into a new container:

ffmpeg -i 'FX3_ONE_20221104_8351.MP4' -c:v copy -c:a copy 'FX3_ONE_20221104_8351fix.mov'

The rewrap video and the proxy then align perfectly in Vegas. But not the audio.

Former user wrote on 3/12/2023, 10:25 PM

@Howard-Vigorita This is what the main camera file looks like, 2 B frames before the first I frame, while the proxy file starts on an I frame. It's not an error, it's normal for many Sony cameras

Howard-Vigorita wrote on 3/12/2023, 10:28 PM

@john_dennis I checked the video alignment by just stacking the clips as you did but then just turning the mute for the top clip on and off to see if anything moved. It did. Sliding the clips into end-alignment did not fix it. The rewrap did, however.

john_dennis wrote on 3/12/2023, 10:46 PM

@Howard-Vigorita

Side by Side on original media:

Side by Side after rewrapping with Shutter Encoder.

While I appreciate the power of FFMPEG, mere mortals may choose to let a front-end like Shutter Encoder do the work.

Former user wrote on 3/12/2023, 10:53 PM

@Howard-Vigorita Look at your first 2 frames now that you've rewrapped the main camera file, are they duplicate?

It may be different for you if you're using the hardware decoder. but by following your process, instead of the camera file playing correctly in Vegas, and the proxy file having 2 duplicate frames at beginning, the camera file gets 2 duplicate frames.

So you've synced the video, but for me the audio is still out of sync by 1 frame... that is if my rewrap using shutter encoder worked the same as your ffmpeg technique. This is getting complicated.

I

 

Howard-Vigorita wrote on 3/12/2023, 11:39 PM

@Former user Looks like what the rewrap did was to insert an i-frame in front of those 1st 2 b-frames. Other than that inserted i-frame, all the rest are identical. Wonder why they don't do the reg file and the proxy the same way?

@john_dennis I'm not that familiar with Shutter Encoder. Can it do a whole bunch of rewraps in one swoop? I suspect @jimingo-1 has a number of them to knock out.

RogerS wrote on 3/12/2023, 11:44 PM

Yes, Shutter Encoder can do multiple files at once.

Custom PC (2022) Intel i5-13600K with UHD 770 iGPU with latest driver, MSI z690 Tomahawk motherboard, 64GB Corsair DDR5 5200 ram, NVIDIA 2080 Super (8GB) with latest studio driver, 2TB Hynix P41 SSD and 2TB Samsung 980 Pro cache drive, Windows 11 Pro 64 bit

Dell XPS 15 laptop (2017) 32GB ram, NVIDIA 1050 (4GB) with latest studio driver, Intel i7-7700HQ with Intel 630 iGPU (latest available driver), dual internal SSD (1TB; 1TB), Windows 10 64 bit

VEGAS Pro 19.651
VEGAS Pro 20.411
VEGAS Pro 21.208
VEGAS Pro 22.93

Try the
VEGAS 4K "sample project" benchmark (works with VP 16+): https://forms.gle/ypyrrbUghEiaf2aC7
VEGAS Pro 20 "Ad" benchmark (works with VP 20+): https://forms.gle/eErJTR87K2bbJc4Q7

Former user wrote on 3/13/2023, 12:16 AM

Wonder why they don't do the reg file and the proxy the same way?

@Howard-Vigorita Also I'd like to know why they do it at all, 2 or 3 B frames before the first I frame, on AVC and HEVC. Has to be a reason behind it. Maybe it aids in faster initial playback somehow with the high resolution high data rate, high compression files. (only guessing at that, if anyone knows please share)

The proxy file as you would have seen is designed for very efficient low resource playback, so they don't use B frames

But as I mentioned the proxy and the main camera file sync in the 3 other NLE's I have and also in the player that does side by side playback compare.

Howard-Vigorita wrote on 3/13/2023, 12:45 AM

@Former user But oddly in Resolve, the rewrap video is out of alignment but the audio looks fine.

jimingo-1 wrote on 3/13/2023, 10:43 PM

Thanks for the help and @Howard-Vigorita, that's a good temporary solution for new projects (importing the regular clips first). I'll reach out to support and and hope for a fix sometime in the future.