FFMetrics -- Yet another program for quality metrics calculation

Comments

Peter_P wrote on 4/21/2021, 12:33 AM

Thanks, I see it just as an indication to see what the tesclip contains.

fifonik wrote on 4/23/2021, 12:45 AM

Version 0.9.5b published

-    New:    VMAF models types supported by FFMpeg (json or pkl) detected automatically on program startup
-    New:    Frames statistics (MeanA, MeanG, Min, Max, StdDevP), frame's info and bitrate added to CSV results file
-    New:    Bad frames could be extracted as png
-    New:    Ref's thumbnail shown in UI
-    Change:    Files list area is resizeable and all 12 files could be visible at once without scrolling (handy for taking screenshots)
-    Change:    FFMpeg process priority changed to "Below Normal" for all calls

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

Peter_P wrote on 4/23/2021, 4:50 AM

Thanks again, it's working fine.

fifonik wrote on 5/3/2021, 2:00 AM

Version 0.9.6b published
-    Change:    All FFMpeg commands logged now (if appropriate option is turned ON)
-    Change:    UI: Bitrate added to short media info for distorted files
-    Change:    UI: Grid with distorted files now resizeable horizontally and "Path to file" column takes maximum available space
-    Bugfix:    VMAF parameters saved in CSV results file might be incorrect
-    Bugfix:    Total bitrate shown instead of video stream's bitrate

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

BigBadBz wrote on 5/19/2021, 7:15 PM

fifonik:

Thanks so much for creating this program! I find it very handy to be able to have a numeric way to attempt to compare the various codecs -- so I have more information in making the classic trade-off of file size vs. quality (and vs redner time).
One question - I tried to use the 'vmaf_float_v0.6.1neg' model from the Netflix/vmaf GitHub site; I copied the .json, .model and .pkl files into the FFMetrics/vmaf-models directory, but when I select the neg model, the program crashed as soon as it started processing VMAF. I'm assuming this NEG model doesn't work with your program - can you tell me when a version that does support a NEG model might become available?

Regards,

Paul

fifonik wrote on 5/19/2021, 8:10 PM

You do not need json AND pkl+pkl.model vmaf models at the same time. If your FFMpeg supports json model -- you only need json file. If it does not -- pkl+.pkl.model files. You can put all and the program will auto-detect what format is supported.

I just tried to calculate VMAF using vmaf_float_v0.6.1neg.json model with FFMetrics 0.9.6b + FFMpeg 4.4 (gyan build, release-full-shared) -- all good.

If the program crashes as soon as you hit "Start" -- this means something got wrong as soon as FFMpeg got called. So I'd suggest to try another FFMpeg build (static, put it in the program's folder).

To find out what is wrong you should turn on option "Write ffmpeg commands in log"; close the program; delete FFMetrics.log; start the program with option "-log-level=debug"; add one ref file + one distorted file, select "vmaf_float_v0.6.1neg.json" model and click Start.

Then provide FFMetrics.log.

In addition, you can try to run the last command from the log in command prompt. May be you will see some details in ffmpeg's output.

Last changed by fifonik on 5/19/2021, 8:23 PM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

Howard-Vigorita wrote on 5/20/2021, 3:09 PM

Hi, @fifonik. Just downloaded your latest build for the 1st time and it looks pretty good. Mostly want to test different render formats using footage from my camera. Got a 30-sec test shot I did in 4K zRaw (not supported by ffmpeg) but my thought is to convert it to lossless hevc with the camera maker's utility and use that as a reference... mainly because it loads into Vegas 18 build 527 which I can then use to render some of the formats I want to test. That sound like good approach?

fifonik wrote on 5/20/2021, 7:15 PM

Lossless hevc? Not sure why you'd like to do this as it is not easy to decode format.

I'd try MagicYUV or UTVideo first, but you have to check if colour info is not lost, otherwise metrics calculated by ffmpeg might be wrong. From this point of view lossless hevc (or some flavour of visually lossless Cineform) make sense.

Another option might be to use AviSynth wrapper (if you have AviSynth + some advanced source filters that can read your source installed).

Could be as easy as:

LWLibavVideoSource("C:\Path\To\Ref.mp4")

or

FFMS2("C:\Path\To\Ref.mp4")

FFMpeg/FFMetrics can read the avs files with no issues.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

Howard-Vigorita wrote on 5/20/2021, 8:41 PM

Thanks but I want to start with raw off my camera sensor and the only supported conversion format I've been able to get Magix to drop onto the timeline is hevc lossless. Far as I know Zcam's ZRAW Video Suite is the only thing around that can read its raw format and convert it. Want to be able to test both Magix and ffmpeg renders with the same reference. It edits just fine in Vegas on my systems. Ffmpeg also seems to like it. I shoot hevc in log format all the time, just not lossless. My camera's sensor layout, raw, and hvec conversion all use the same chroma sub-sampling. So I figured I'd shoot the raw clip in log too and convert to hevc-lossless. Here are some of the numbers I'm getting doing transcodes from hevc-lossless to AV1 with ffmpeg ... do you think the log format might be confusing things?

fifonik wrote on 5/20/2021, 11:42 PM

Sorry, I do not know.

It should not be an issue for PSNR/SSIM (if you not converting colour space). Number is a number. That's it. Not sure if VMAF likes the log or not.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

Howard-Vigorita wrote on 5/21/2021, 6:47 AM

I did hit a little snag. One of my transcodes was defective and not only wouldn't read right in your app, it kind of messed up it's ability to remove it from the added files list, do a reset, or do anything with any other added file. Problem persisted after exit and even reboot because the app auto-loaded it and other files previously added on every startup which messed it up all over again. Ended up deleting the C:\Users\<user>\AppData\Local\FFMetrics directory to clean the slate.

fifonik wrote on 5/21/2021, 6:53 AM

I wish I have a file that cause the issue. In this case I can see and solve the issue.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

Howard-Vigorita wrote on 5/21/2021, 7:29 AM

Sent you a couple pms with log and link.

Howard-Vigorita wrote on 5/22/2021, 12:30 PM

@fifonik Thanks for your insight that an excessive ffmpeg delay on my hq render was causing the fault. Moved off of my laptop over to a faster 9900k and was able to process it aok.

fifonik wrote on 5/22/2021, 5:31 PM

I'm making changes in the program so it will better handle heavy files. Many thanks for your input.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

BigBadBz wrote on 5/22/2021, 7:57 PM

fifonik:

I changed my FFmpeg to version 4.4 gyan build, release-full-shared, and then tried to run with the vmaf_float_v0.6.1neg.json model. FFMetrics did not crash, but showed ERROR in the VAMF column for both files to be scored. There are several lines in the FFMterics.log file, but I don't know how to upload a text file to this forum...

Regards,

Paul

fifonik wrote on 5/22/2021, 8:51 PM

Version 0.9.7b published

- New: Scaling method can be changed in menu or using command line parameter -scaling-method=NEIGHBOR|GAUSS|BILINEAR|BICUBIC|LANCZOS|SINC|SPLINE.
It is used only if distorted file's resolution is not equal to reference file's resolution. Default -- bicubic.

- Change: Processes for getting ref's media info and ref's thumbnail are off-loaded to background process.
So these processes are not blocking UI on startup any longer

- Change: UI: Block with options is resizable

- Change: UI: Block with ref file is resizable

- Change: UI: Minimum program windows width decreased

- Change: UI: Progress indicator is displayed while getting ref's thumbnail

- Change: UI: Progress indicator is displayed while getting files' media info

- Change: Timeout for process of getting ref's thumbnail increased from 3 to 5 seconds

- Change: Timeout for process of getting media info increased from 5 to 10 seconds

- Change: Process of getting media info analyzes maximum 2 seconds and maximum 30 frames (previously 60 frames were always analyzed)

- Bugfix: Old preview displayed after a new project created

 

Last changed by fifonik on 5/22/2021, 9:05 PM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

Howard-Vigorita wrote on 5/24/2021, 12:33 AM

Thanks again @fifonik for this convenient utility... here's what I've been doing with it:

http://www.rtpress.com/ffmetrics.htm

fifonik wrote on 5/24/2021, 9:55 PM

@BigBadBz As per our email conversation, your json neg-model was broken and using correct model file fixed the issue. Two json neg models are included in FFretrics archive starting from version 0.9.7b.

Thanks.

Last changed by fifonik on 5/24/2021, 9:57 PM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

BigBadBz wrote on 5/27/2021, 4:42 PM

fifonik:

I downloaded 0.9.7b, and now I can run NEG comparisons! Thanks! I'll try to learn how to correctly download files from GitHub using Windows... 😀

 

Regards,

BBB (Paul)

xiaorong wrote on 6/9/2021, 10:14 PM

hi @fifonik I I have a problem with VAMF calculation,I can't check VMAF item and it's greyish. First time I open this software ,it worked well.
 

fifonik wrote on 6/10/2021, 12:47 AM

What is your ffmpeg version? Does it support VMAF?

Do you have vmaf-metrics folder with metrics aside of ffmetrics executable?

P.S. You can also check FFMetrics.log

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

xiaorong wrote on 6/10/2021, 3:23 AM

What is your ffmpeg version? Does it support VMAF?

Do you have vmaf-metrics folder with metrics aside of ffmetrics executable?

P.S. You can also check FFMetrics.log

My ffmpeg version is 4.4 ,I think it's support VMAF while it shows this ffmpeg don't support VMAF in FFMetrics(v 0.9.7.4b)

I checked FFMetrics.log and it shows this :“WARNING: VMAF metric initialization failed”
What should I do?
 

fifonik wrote on 6/10/2021, 5:53 AM

Have you seen my other question about vmaf-models folder?

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 500GB (NVMe, OS), Samsung 870 Evo 1TB, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer