Optical Flow: How does it work by U?

Yelandkeil wrote on 3/13/2022, 1:39 AM


I think my 1st crash after the 532-update was testing this function. 
I reinstalled my graphic driver to the WHQL version, it still crashed yet without shrinking the VEGAS surface. 

Digging into the Internalpreferences, I modified the "Optical flow based bllending" from 0=Blend to 
1=Nearest Frame

Now it works, with a shock! 
It's tenacious to build the RAMpreview even with Quarter Quality. 😴😴

The rendering, by my machine, takes much less than 1frame/sec. 
Here is a 4k24p scene "optically flowed" into my 4k60p project, the result also very shocking. 
👍👍👍 Thanks for the great job, VEGAS team! 

Last changed by Yelandkeil

-- Hard&Software for 5.1RealHDR10 --

ASUS TUF Gaming B550plus BIOS3202: 
*Thermaltake TOUGHPOWER GF1 850W 
*ADATA XPG GAMMIX S11PRO; 512GB/sys, 2TB/data 
*G.SKILL F4-3200C16Q-64GFX 
*AMD Ryzen9 5950x + LiquidFreezer II-240 
*XFX Speedster-MERC319-RX6900XT <-AdrenalinEdition 24.12.1
Windows11Pro: 24H2-26100.3915; Direct3D: 9.17.11.0272

Samsung 2xLU28R55 HDR10 (300CD/m², 1499Nits/peak) ->2xDPort
ROCCAT Kave 5.1Headset/Mic ->Analog (AAFOptimusPack 6.0.9403.1)
LG DSP7 Surround 5.1Soundbar ->TOSLINK

DC-GH6/H-FS12060E_HLG4k120p: WB=manual, Shutter=125, ISO=auto/manual
HERO5_ProtuneFlat2.7k60pLinear: WB=4800K, Shutter=auto, ISO=800

VEGASPro22 + XMediaRecode/Handbrake + DVDArchi7 
AcidPro10 + SoundForgePro14.0.065 + SpectraLayersPro7 
K-LitecodecPack17.8.0 (MPC Video Renderer for HDR10-Videoplayback on PC) 

Comments

3POINT wrote on 3/13/2022, 3:27 AM

I compared optical flow of VP19 with optical flow of Davinci Resolve. Quality 0:1 Speed 0:1.

Yelandkeil wrote on 3/13/2022, 3:39 AM

No!

My 13sec long test has been taken 30min for rendering.

The "Blending Frame" acts quite normal, doesn't consume the PC resource so much. But the result is also horrible:

Notice:

The uploaded video has been (by forum) converted to 30p; the original 4k60p so smooth like a natural silk.

 

 

-- Hard&Software for 5.1RealHDR10 --

ASUS TUF Gaming B550plus BIOS3202: 
*Thermaltake TOUGHPOWER GF1 850W 
*ADATA XPG GAMMIX S11PRO; 512GB/sys, 2TB/data 
*G.SKILL F4-3200C16Q-64GFX 
*AMD Ryzen9 5950x + LiquidFreezer II-240 
*XFX Speedster-MERC319-RX6900XT <-AdrenalinEdition 24.12.1
Windows11Pro: 24H2-26100.3915; Direct3D: 9.17.11.0272

Samsung 2xLU28R55 HDR10 (300CD/m², 1499Nits/peak) ->2xDPort
ROCCAT Kave 5.1Headset/Mic ->Analog (AAFOptimusPack 6.0.9403.1)
LG DSP7 Surround 5.1Soundbar ->TOSLINK

DC-GH6/H-FS12060E_HLG4k120p: WB=manual, Shutter=125, ISO=auto/manual
HERO5_ProtuneFlat2.7k60pLinear: WB=4800K, Shutter=auto, ISO=800

VEGASPro22 + XMediaRecode/Handbrake + DVDArchi7 
AcidPro10 + SoundForgePro14.0.065 + SpectraLayersPro7 
K-LitecodecPack17.8.0 (MPC Video Renderer for HDR10-Videoplayback on PC) 

Yelandkeil wrote on 3/13/2022, 3:41 AM

Do you @3POINT mean the quality of VP19 is not good?

-- Hard&Software for 5.1RealHDR10 --

ASUS TUF Gaming B550plus BIOS3202: 
*Thermaltake TOUGHPOWER GF1 850W 
*ADATA XPG GAMMIX S11PRO; 512GB/sys, 2TB/data 
*G.SKILL F4-3200C16Q-64GFX 
*AMD Ryzen9 5950x + LiquidFreezer II-240 
*XFX Speedster-MERC319-RX6900XT <-AdrenalinEdition 24.12.1
Windows11Pro: 24H2-26100.3915; Direct3D: 9.17.11.0272

Samsung 2xLU28R55 HDR10 (300CD/m², 1499Nits/peak) ->2xDPort
ROCCAT Kave 5.1Headset/Mic ->Analog (AAFOptimusPack 6.0.9403.1)
LG DSP7 Surround 5.1Soundbar ->TOSLINK

DC-GH6/H-FS12060E_HLG4k120p: WB=manual, Shutter=125, ISO=auto/manual
HERO5_ProtuneFlat2.7k60pLinear: WB=4800K, Shutter=auto, ISO=800

VEGASPro22 + XMediaRecode/Handbrake + DVDArchi7 
AcidPro10 + SoundForgePro14.0.065 + SpectraLayersPro7 
K-LitecodecPack17.8.0 (MPC Video Renderer for HDR10-Videoplayback on PC) 

3POINT wrote on 3/13/2022, 5:08 AM

Do you @3POINT mean the quality of VP19 is not good?

Indeed, compared with Davinci Resolve the quality and speed is far superior from the results I get with Vegas. Rendering was even 8 times faster. I used for the test a 1080i50 (HDV) sequence which I slowed down to 25% put optical flow to it and rendered it to 1080p50.

 DR optical flow (+ stabilization) 0:12 rendertime (12 seconds)

VP optical flow 1:39 rendertime (99 seconds)

Yelandkeil wrote on 3/13/2022, 6:48 AM

Hhm, the davinci really impressive; but I do VEGAS in focus.

The Forum allows only 30p samplevideo, so a new test under the environment: 


4k24p source: 5sec opticalflow (did some crop due to border distortion), rendertime=8:32min 


fullHD-25i source, 5sec opticalflow
1+0.25slowmotion=ca. 19.25sec, rendertime=2:26min; 


2+0.25Twixtor=ca. 19.25sec, rendertime=0:22min. 


Personal conclusion (I have a very weak graka): 
Functions of both the opticalflow and slowmotion are OK; the efficiency very dependable on source material and hardware, sure also the projectsettings. 
Quality pleasantly acceptable, especially when the output set to 60fps yields a very smooth motion. 

 
 

-- Hard&Software for 5.1RealHDR10 --

ASUS TUF Gaming B550plus BIOS3202: 
*Thermaltake TOUGHPOWER GF1 850W 
*ADATA XPG GAMMIX S11PRO; 512GB/sys, 2TB/data 
*G.SKILL F4-3200C16Q-64GFX 
*AMD Ryzen9 5950x + LiquidFreezer II-240 
*XFX Speedster-MERC319-RX6900XT <-AdrenalinEdition 24.12.1
Windows11Pro: 24H2-26100.3915; Direct3D: 9.17.11.0272

Samsung 2xLU28R55 HDR10 (300CD/m², 1499Nits/peak) ->2xDPort
ROCCAT Kave 5.1Headset/Mic ->Analog (AAFOptimusPack 6.0.9403.1)
LG DSP7 Surround 5.1Soundbar ->TOSLINK

DC-GH6/H-FS12060E_HLG4k120p: WB=manual, Shutter=125, ISO=auto/manual
HERO5_ProtuneFlat2.7k60pLinear: WB=4800K, Shutter=auto, ISO=800

VEGASPro22 + XMediaRecode/Handbrake + DVDArchi7 
AcidPro10 + SoundForgePro14.0.065 + SpectraLayersPro7 
K-LitecodecPack17.8.0 (MPC Video Renderer for HDR10-Videoplayback on PC) 

3POINT wrote on 3/13/2022, 7:03 AM

My conclusion so far is that optical flow in Vegas has to be improved a lot to become competitive.

Last changed by 3POINT on 3/13/2022, 7:07 AM, changed a total of 1 times.

3POINT, Theo Houben, Vegasuser since version 5 and co-founder and moderator of the Dutch Vegasforum https://www.vegas-videoforum.nl/index.php

Recware: DJI Osmo Pocket/Mavic Mini, GoproHero7Black, PanasonicFZ300/HCX909.

Software: Vegaspro365+Vegasaur, PowerDirector365, Davinci Resolve 20

Hardware: i910900k, 32GB, GTX2080super, 2x1920x1200 display

Playware: Samsung Qled QE65Q6FN

Former user wrote on 3/13/2022, 7:09 AM

@Yelandkeil

Could you also try FlowFrames on default settings (RIFE) sometime?

in output settings you want x4, and change normal speed to 4x slow mo, I'd be interested to see how it compares as far as distortion with your walking girl test video

https://icedrive.net/s/24emsDlBpx

With 'complex' scenes like this one, it's been my experience it also does not do very well, but I"ve only ever used it on defaults

 

Yelandkeil wrote on 3/13/2022, 7:33 AM

😁😁 @3POINT

U are an inside tester. As I just wondered where did U get that opticalflow by your post, the 4th update came.

Give Gary some time as long as he revamps and I hope the VP19's 5th(last?) update will solve the scopes freezing by HDR-editing.

Dann bin ich wunschlos und glücklich!

 

-- Hard&Software for 5.1RealHDR10 --

ASUS TUF Gaming B550plus BIOS3202: 
*Thermaltake TOUGHPOWER GF1 850W 
*ADATA XPG GAMMIX S11PRO; 512GB/sys, 2TB/data 
*G.SKILL F4-3200C16Q-64GFX 
*AMD Ryzen9 5950x + LiquidFreezer II-240 
*XFX Speedster-MERC319-RX6900XT <-AdrenalinEdition 24.12.1
Windows11Pro: 24H2-26100.3915; Direct3D: 9.17.11.0272

Samsung 2xLU28R55 HDR10 (300CD/m², 1499Nits/peak) ->2xDPort
ROCCAT Kave 5.1Headset/Mic ->Analog (AAFOptimusPack 6.0.9403.1)
LG DSP7 Surround 5.1Soundbar ->TOSLINK

DC-GH6/H-FS12060E_HLG4k120p: WB=manual, Shutter=125, ISO=auto/manual
HERO5_ProtuneFlat2.7k60pLinear: WB=4800K, Shutter=auto, ISO=800

VEGASPro22 + XMediaRecode/Handbrake + DVDArchi7 
AcidPro10 + SoundForgePro14.0.065 + SpectraLayersPro7 
K-LitecodecPack17.8.0 (MPC Video Renderer for HDR10-Videoplayback on PC) 

gary-rebholz wrote on 3/15/2022, 8:17 AM

As you guys are doing your quality tests, keep in mind that there are two settings to consider:

  • During preview from the timeline, your preview settings affect what you see. The optical flow algorithm has 3 quality levels and the preview quality dictates which is used for preview:
    • Draft: disables optical flow estimation
    • Preview: yiedls coarse estimation
    • Good: yields medium estimation
    • Best: yileds fine estimation
  • During render, the quality of Optical Flow slow motion depends upon the Render Quality setting (either from the Project Properties dialog or overridden in the Customize section of the Render dialog) with the same points as made above

This means that if you are rendering at the VEGAS Pro default setting of Good for render quality, you might be able to see better results if you change to Best during render.

Clearly this dual-consideration situation is potentially confusing. We're discussing ways for potential improvement.

Other tools could well have a speed advantage over us since we do not tie our's to any hardware (that is, we do not depend upon something like the Nvidia Optical Flow SDK). It's something we could look into, but there are issues (both of cost and dependency) that we have to consider carefully.

joelsonforte.br wrote on 3/15/2022, 7:29 PM

@gary-rebholz

Thank you very much for the significant improvements to the Vegas optical flow system. I can now work much more efficiently on projects that have files with different frame rates.

I did some testing in the trial version and wanted to leave my contribution to future improvements in optical flow system.

1.For some unknown reason, optical flow doesn't produce as smooth and crisp results in interlaced files as shown in the @3POINT video. I also tested the same interlaced files I used in Vegas in Adobe Premiere and the result was smooth and crisp.

2.For best results in Vegas you need to create a separate nested project for the interlaced file and change the project settings to progressive using the Interpolate Fields deinterlacing method and then in the main project apply optical flow in the event corresponding to the nested project. Smoothness and sharpness are greatly improved and on par with Adobe Premiere results.

The procedure I use is shown in the video below.

3.I also have an improvement suggestion to option Trim Events to Include All Frames. At the moment, this option takes into account all lenght of the media instead of just the selected event and when it is enabled, it does not allow cutting edits or adjusting the length of the selected event. You must either create a subclip of just the part that will have the frame rate changed or disable the option to be able to make crop edits or event length adjustments. The video below shows the step by step of what I'm talking about.

The Trim Events to Include All Frames option is so useful that it should run automatically by default every time the frame rate in the event is changed, but only at the selected event level and not at the media level allowing crop edits or event length adjustments and should be called Adjust Event Length to Include All Frames for a better understanding of the function. (Just a suggestion for you and your team to evaluate).

4.It is not very safe to use the Build Dynamic RAM Preview option to see the final result even if the Preview window is in Good or Best Full mode. In my tests there are glitches and artifacts in Preview window using the Build Dynamic RAM Preview option that don't show up in the final render. Currently, it is safer to use the Selectively Prerender Video option or render an intermediate file with the Render to New Track option to see the final result in the timeline.

Thanks!

Former user wrote on 3/15/2022, 7:46 PM

@gary-rebholz This is a comparision of RIFE and Vegas Pro 18.I know the VP19 interface has improved but think it's still same algorithm? This is a test of Optical flow(25%) using FINE analysis for 4 complex scenes, results are not perfect for any scene except scene 2 where RIFE looks to be perfect. Where RIFE fails it fails more elegantly and the video may still be usable

Traditionally you would not even try optical flow on such scenes. Old Twixtor tutorials recommend a camera on tripod and filming a person against the sky or a green screen. These scenes are certainly not that, but expectations are higher now, even if it's pot luck if a complex scene will work with optical flow

 

3POINT wrote on 3/16/2022, 2:17 AM

2.For best results in Vegas you need to create a separate nested project for the interlaced file and change the project settings to progressive using the Interpolate Fields deinterlacing method and then in the main project apply optical flow in the event corresponding to the nested project. Smoothness and sharpness are greatly improved and on par with Adobe Premiere results.

@Joelson_Forte Quite cumbersome. The time you needed to create the nested project, I produced in DR the finished render with optical flow.

@Former user A comparison of RIFE (Flow frames) with Vegas optical flow is not quite realistic. RIFE can only use 2x, 4x,8x frame interpolation while Vegas optical flow should be able also for in between values.

Former user wrote on 3/16/2022, 2:53 AM

@3POINT Also the test footage is a nightmare scenario, shot with Panavision anamorphic lenses which inherently contain a level of distortion, modified to alter the focus towards the edges, and transferred from digital to film intermediate and back to digital, all designed to reduce the pinpoint sharpness of digital cameras. No high speed shutter either. Everything that optical flow doesn't like. It is possible Vegas does the worst with the edges of the frame due to the fall off in sharpness, not understanding the scene as well as the AI model based RIFE