Optical flow Resample mode works wrong (has a bug)!

Sam-Fisher wrote on 5/7/2022, 3:02 AM

First of all, thank you for adding this feature. I've been waiting it for years!!!

Unfortunately, it has some issues. When I use a 29.970 fps video in my 25 fps project the optical flow is working smoothly for several seconds, then it fails to synchronize frames and the rest of the video is jittering. I have tested it on my drone footage as well as on the artificial test footages. I'll attach the videos.

As you can see on my test, it fails after the frame that is too different.

Thanks in advance. Hope you'll fix it soon.

Tested on Vegas Pro 19 550

Original footage (29.970fps):

Resampled with optical flow to 25 fps:

 

Comments

Yelandkeil wrote on 5/7/2022, 3:28 AM

...and mostly for objects with peaceful background.

ASUS TUF Gaming B550plus Bios2803: 
*Thermaltake TP-850AH3CCG Madrid850W 
*ADATA XPG GAMMIX S11PRO 512GB4sys + 2TB4data 
*G.SKILL F4-3200C16Q-64GFX 
*AMD Ryzen9 5950x + LiquidFreezer II-240 
*XFX Speedster-MERC319雇佣兵-RX6900XT + AdrenalinEdition 22.5.1 (WHQL) 
DPort: 2xSamsung-LU28R55 10bit4k60Hz 300cd/m²HDR10 
ALC1200+Realtek 6.0.9357.1 Analog: ROCCAT Kave 5.1Headset/Mic 

ALC1200-TOSLINK: Samsung HW-Q60T 5.1 Soundbar?
USB Input: IK Multimedia iRigKeys MIDI24 
USB Device: Ultron CardReader/Writer 

Windowspro 10.0.19044.1766 
DirectX 12_2 Ultimate, Direct3D11Package installed 
OpenGL 4.6.14800 Compatibility Profile Context Version 22.5.1 30.0.15021.11005

Vegaspro19 + Handbrake/XMediaRecode + DVDArchi7 
Acidpro10 + SoundForgpro14.0.0.65 + SpectraLayerspro7 

K-LitecodecPack1662 (MPC Video Renderer for HDR10-Videoplayback) 

Sam-Fisher wrote on 5/7/2022, 3:49 AM

Not necessarily. I use it to match framerate to get rid of jittering and it works fine in After Effects.

Yelandkeil wrote on 5/7/2022, 3:54 AM

Yeh, a newborn with much to learn to improve, e.g. it uses only 2 Cores of my 16Core-CPU and no GPU-Acc...but don't change your Topic to AE pls.

ASUS TUF Gaming B550plus Bios2803: 
*Thermaltake TP-850AH3CCG Madrid850W 
*ADATA XPG GAMMIX S11PRO 512GB4sys + 2TB4data 
*G.SKILL F4-3200C16Q-64GFX 
*AMD Ryzen9 5950x + LiquidFreezer II-240 
*XFX Speedster-MERC319雇佣兵-RX6900XT + AdrenalinEdition 22.5.1 (WHQL) 
DPort: 2xSamsung-LU28R55 10bit4k60Hz 300cd/m²HDR10 
ALC1200+Realtek 6.0.9357.1 Analog: ROCCAT Kave 5.1Headset/Mic 

ALC1200-TOSLINK: Samsung HW-Q60T 5.1 Soundbar?
USB Input: IK Multimedia iRigKeys MIDI24 
USB Device: Ultron CardReader/Writer 

Windowspro 10.0.19044.1766 
DirectX 12_2 Ultimate, Direct3D11Package installed 
OpenGL 4.6.14800 Compatibility Profile Context Version 22.5.1 30.0.15021.11005

Vegaspro19 + Handbrake/XMediaRecode + DVDArchi7 
Acidpro10 + SoundForgpro14.0.0.65 + SpectraLayerspro7 

K-LitecodecPack1662 (MPC Video Renderer for HDR10-Videoplayback) 

todd-b wrote on 5/7/2022, 10:48 PM

Unfortunately, it has some issues. When I use a 29.970 fps video in my 25 fps project the optical flow is working smoothly for several seconds, then it fails to synchronize frames and the rest of the video is jittering. I have tested it on my drone footage as well as on the artificial test footages. I'll attach the videos.

As you can see on my test, it fails after the frame that is too different.

I tried your technique with Davinci Resolve as a comparison. I placed your original 29.97fps test file above optical flow versions created by Resolve and Vegas on a 25fps timeline using composite mode: SUBTRACT to better highlight the problem

Vegas is failing at 'scene change/ edit point' . A feature of many optical flow is 'detect scene change' Maybe it is detecting and behaving strange or it's not detecting and behaves strange... or unrelated

john_dennis wrote on 5/8/2022, 12:37 PM

@Sam-Fisher

I created a .png file from a Vegas screenshot of the checkerboard pattern. It is accurate to one pixel and I made it wide enough in Photoshop to scroll across the screen using Pan/Crop (linear keyframe) in Vegas Pro for the 14 seconds you posted earlier. This test scenario should not contain any "scene change" as alluded to by @todd-b.

Original 29.97 FPS Video

29.97 FPS Video rendered in a 25 FPS Project with Optical Flow

Original 59.94 FPS Video (Just Because)

Addendum

I downloaded the videos that I just uploaded to the forum and find that forum software has re-encoded the files. This kind of issue can't really be discussed rationally using the forum video.

Forum Encoded Original 29.97 FPS Video

General
Complete name                            : C:\Users\John\Desktop\4B2CDD62CB7C489192663B2EC246A1A0.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 258 KiB
Duration                                 : 14 s 125 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 150 kb/s
Writing application                      : Lavf57.25.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 14 s 115 ms
Bit rate                                 : 137 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 29.970 (30000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.002
Stream size                              : 237 KiB (92%)
Writing library                          : x264 core 148 r2748 97eaef2
Encoding settings                        : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=3 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=22.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=5000 / vbv_bufsize=2500 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00
Language                                 : English
Codec configuration box                  : avcC

Forum Encoded 29.97 FPS Video rendered in a 25 FPS Project with Optical Flow

General
Complete name                            : C:\Users\John\Desktop\A9C15E6ADAF74D33BE87B35B223DACB0.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 2.56 MiB
Duration                                 : 14 s 18 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 1 534 kb/s
Writing application                      : Lavf57.25.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 14 s 0 ms
Bit rate                                 : 1 527 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.029
Stream size                              : 2.55 MiB (99%)
Writing library                          : x264 core 148 r2748 97eaef2
Encoding settings                        : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=3 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=22.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=5000 / vbv_bufsize=2500 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00
Language                                 : English
Codec configuration box                  : avcC

Forum Encoded Original 59.94 FPS Video (Just Because)

General
Complete name                            : C:\Users\John\Desktop\84A64FCDD6144F89A568713FAAF74E95 (1).mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 1.07 MiB
Duration                                 : 14 s 134 ms
Overall bit rate mode                    : Variable
Overall bit rate                         : 635 kb/s
Writing application                      : Lavf57.25.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 14 s 134 ms
Bit rate                                 : 623 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 30.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.010
Stream size                              : 1.05 MiB (98%)
Writing library                          : x264 core 148 r2748 97eaef2
Encoding settings                        : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=3 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=22.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=5000 / vbv_bufsize=2500 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.00
Language                                 : English
Codec configuration box                  : avcC

PS

The 59.94 FPS file plays slick as butter on my monitor that runs at 59.94 FPS.

Last changed by john_dennis on 5/8/2022, 12:54 PM, changed a total of 2 times.

My main system:
Motherboard: Asus X99-AII
CPU: Intel i7-6850K
GPU: Sapphire Radeon RX480-8GB
RAM: Corsair Dominator (4 x 4 GB) DDR4 2400
Disk O/S & Programs: Intel SSD 750 (400 GB)
Disk Active Projects: 1TB & 2TB WD BLACK SN750 NVMe Internal PCI Express 3.0 x4 Solid State Drives
Disk Other: WD Ultrastar/Hitachi Hard Drives: WDBBUR0080BNC-WRSN, HGST HUH728080ALE600, 724040ALE640, HDS3020BLA642
Case: LIAN LI PC-90 Black Aluminum ATX Full Tower Case
CPU cooling: Corsair Hydro series H115i
Power supply: SeaSonic SS-750KM3 750W 80 PLUS GOLD Certified Full Modular Active PFC Power Supply
Drive Bay: Kingwin KF-256-BK 2.5" and 3.5" Trayless Hot Swap Rack with USB 3
Sound card: Crystal Sound 3 on motherboard. Recording done on another system.
Primary Monitor: Asus ProArt PA248q (24" 1920 x 1200)
O/S: Windows 10 Pro 190943
Camera: Sony RX10 Model IV

https://www.youtube.com/user/thedennischannel

john_dennis wrote on 5/8/2022, 1:03 PM

My project and output files can be downloaded from here...

https://drive.google.com/file/d/1FQcAbxYnhOcyrJxiWs1jUzmu0OA6Itct/view?usp=sharing

... for 3/8 of a non-second.

My main system:
Motherboard: Asus X99-AII
CPU: Intel i7-6850K
GPU: Sapphire Radeon RX480-8GB
RAM: Corsair Dominator (4 x 4 GB) DDR4 2400
Disk O/S & Programs: Intel SSD 750 (400 GB)
Disk Active Projects: 1TB & 2TB WD BLACK SN750 NVMe Internal PCI Express 3.0 x4 Solid State Drives
Disk Other: WD Ultrastar/Hitachi Hard Drives: WDBBUR0080BNC-WRSN, HGST HUH728080ALE600, 724040ALE640, HDS3020BLA642
Case: LIAN LI PC-90 Black Aluminum ATX Full Tower Case
CPU cooling: Corsair Hydro series H115i
Power supply: SeaSonic SS-750KM3 750W 80 PLUS GOLD Certified Full Modular Active PFC Power Supply
Drive Bay: Kingwin KF-256-BK 2.5" and 3.5" Trayless Hot Swap Rack with USB 3
Sound card: Crystal Sound 3 on motherboard. Recording done on another system.
Primary Monitor: Asus ProArt PA248q (24" 1920 x 1200)
O/S: Windows 10 Pro 190943
Camera: Sony RX10 Model IV

https://www.youtube.com/user/thedennischannel

todd-b wrote on 5/8/2022, 5:56 PM

@john_dennis I tried FlowFrames using 2 different AI models with scene detection turned off. It doesn't fail the way Vegas is doing, I can't see a difference using his video with scene detection on or off. The problem may be greater then a default setting.

@Sam-Fisher Are you using Vegas optical flow with your Drone video on a scene by scene basis, no edit points?

You didn't mention what sort of drone footage, A DJI peacefully panning across a mountain could give different results to a racing drone doing flips flying at high speed spinning about. Explain in more detail where the optical flow is failing

Sam-Fisher wrote on 5/10/2022, 1:38 PM

@john_dennis I tried FlowFrames using 2 different AI models with scene detection turned off. It doesn't fail the way Vegas is doing, I can't see a difference using his video with scene detection on or off. The problem may be greater then a default setting.

@Sam-Fisher Are you using Vegas optical flow with your Drone video on a scene by scene basis, no edit points?

You didn't mention what sort of drone footage, A DJI peacefully panning across a mountain could give different results to a racing drone doing flips flying at high speed spinning about. Explain in more detail where the optical flow is failing

I tried using Optical flow on slow flying. (It's not a racing drone.) OF works the same way as we see on the test footages. Unfortunately, it is useless until it is fixed. I always have to use videos with different framerates in my production, so It is one of the most valuable features for me.

Ok, let's try a drone footage without fast movements or scene changes at all. (It is the whole video file. No cut at all).

Original footage 29.970 fps:

Converted to 25 fps with Vegas optical flow:

The same behavior: it jitters the strange way.

todd-b wrote on 5/10/2022, 5:00 PM

That doesn't look good. Free optical flows RIFE and DAIN do 2x 4x 8x 16x so not what you need. I believe Davinci Resolve has 2 of it's 3 optical flow models available in free version, you could try it just for this task, do some color grading while you're there