FFMetrics -- Yet another program for quality metrics calculation

Comments

fifonik wrote on 1/24/2021, 4:16 PM

Thanks for info. I was able to re-produce the issue on one my PC. Will try to investigate when I have time. However, no promises as it is very low priority on my list.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

fifonik wrote on 1/25/2021, 4:56 AM

Version 0.9.0b published.

Changes:

-    New:    UI: Program menu added
-    New:    UI: Hotkeys added (investigate program menu to find them out)
-    New:    UI: "About" with short program description and link to project homepage added
-    New:    Added ability to work with projects (save/load)
-    New:    Distorted files automatically resized to ref's resolution during metrics calculation
-    New:    Processing duration is remembered over program restarts (aligned with other options)
-    New:    Harmonic Mean added to metric value Tooltip
-    New:    The Best/The Worst Min/Max/Avg/StdDevP metrics' values in Tooltip highlighted (only ticked files are taken into consideration)
-    New:    Critical values in media info Tooltip highlighted
-    New:    Added ability to copy image to clipboard (Ctrl+C)
-    New:    Command line parameter "-duration=seconds" added (value must exactly match one of the value available in program dropdown)
-    New:    Command line parameter "-project=/path/to/project.ffmproj" added. For now values from project file have higher priority.
-    Change: Command line bool parameter "-vmaf-harmonic-mean" replaced with "-vmaf-pool=harmonic_mean|mean" (to align with libvmaf options)
-    Change:    VMAF json-models will be used instead of pkl-models if they found in models' folder.
            If you use old FFMpeg that supports pkl-models, you must ensure that there are no json-models in models' folder.
            If you use new FFMpeg that supports json-models, you must ensure that you've put json-models in models' folder (included in separate archive). pkl models will not be used in this case.
-    Change:    Execution is disabled until media info for all ticked files gathered
-    Change:    FFMetrics.conf format is changed (now model keys must not have extensions).
            You will need to re-implement your changes if you using the custom metrics configuration file.
-    Bugfix:    Command line parameters not parsed correctly if sole distorted file specified

 

P.S. With the new version I do not see the black line on the right after maximizing the program windows on my system. However, I have not done anything special for this. It is possible that the issue was fixed when I re-designed program's form. It would be nice if someone who had the issue check and report back.

Last changed by fifonik on 1/25/2021, 5:41 PM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

dr0 wrote on 1/30/2021, 8:59 AM

For some reason, I can't make v0.9.0 and/or v0.9.2 to run on my main desktop machine. When I double-click the executable a few instances of FFmetrics.exe, with very low RAM usage, show up in Task Manager, but the program window never appears on my screen. When I double-click the program executable again, another instance of FFmetrics.exe shows up in Task Manager, but nothing ever happens. I should also mention that It's impossible to kill these program instances in Task Manager. The only way to get rid of them is to reboot the machine.

Logfile in the program folder never gets created as the program can't even start running.

Any suggestions as to what this might be?

Thankfully, I can still run v0.8.7 on this machine.

 

I tried to run v0.9.0 on my laptop (also under Windows 7). I like the small touch ups of the UI that you did. I can confirm that at least on this laptop v0.9.0 doesn't have the vertical black line on the right hand side of the screen when I maximize its window.

 

musko wrote on 1/30/2021, 9:06 AM

For me version 0.9.0 working correctly. Without vertical black line (in full screen mode)

fifonik wrote on 1/30/2021, 4:51 PM

@dr0 This is strange as I have not done any big changes in program. You can try to do the following:

- Find "FFMetrics" entries under C:\Users\<your-login>\ and delete them

- Delete FFMetrics.log

- Run 'FFMetrics -log-level=DEBUG'

You should have FFMetrics.log created with something. Send it to me with information about your system please.

P.S. BTW, version 0.9.2b published, an issue when Ref's media info not refreshed.

P.P.S. You can wait a little as I'm going to upload version with more debug logging added

Last changed by fifonik on 1/30/2021, 5:15 PM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

dr0 wrote on 1/31/2021, 3:24 AM

- Find "FFMetrics" entries under C:\Users\<your-login>\ and delete them

I found three folders in %AppData% that contained FFMetrics in their name. I tried to delete them but that didn't solve the issue.

Are there any other places that I should check?

- Delete FFMetrics.log

As I said before, with 0.9.0b and above, .log file never even gets created.

- Run 'FFMetrics -log-level=DEBUG'

Thanks for the hint. I'll try that when I'll have the chance.

You should have FFMetrics.log created with something. Send it to me with information about your system please.

The only .log file that I have was created in 0.8.7b, but, I doubt that it can tell you anything about this issue. I can upload it later if you think otherwise.

dr0 wrote on 2/1/2021, 11:15 AM

Strangely enough, I can now run FFMetrics v0.9.2 on this (problematic) machine. The only thing that I did differently this time is that I used a different folder path and name for FFMetrics (than that I used for v0.8.7 previously).

fifonik wrote on 2/1/2021, 2:58 PM

Thanks for letting me know about this.

Are there any special characters in the folder path/name? Do the folder has special permissions? What is the folder filespec length?

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

fifonik wrote on 2/1/2021, 2:59 PM

Version 0.9.2b published

- Change: Options from project that was specified in command line parameter "-project=/path/to/project.ffmproj" now have lower priority than options specified in command line parameters directly.

- Change: Hotkey for "Save Project As" changed from Ctrl+Alt+S to Ctrl+Shift+S

- Change: Hotkey for selecting Ref added (Ctrl+Alt+O)

- Change: FFMetrics and FFMpeg versions added into results csv file

- Bugfix: Ref's media info not updated

- Bugfix: "Save CSV" button disabled after saving results that prevents results to be saved more than once

 

Version 0.9.3b published

- Change: Logic for generating VMAF log filename changed for working around ffmpeg/libvmaf bug

- Change: Internal changes to improve logging

- Bugfix: Un-ticking all files not disabling "Start" button

Last changed by fifonik on 2/2/2021, 2:41 PM, changed a total of 4 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

dr0 wrote on 2/2/2021, 1:24 PM

Are there any special characters in the folder path/name? Do the folder has special permissions? What is the folder filespec length?

The folder path is D:\Software\MEDIA-Edit\FFMetrics. I use this exact path on all of my computers. AFAICT, there is nothing special about this folder, no custom permissions or anything like that. I'm not really sure what filespec length is though.

I used C:\Users\dr0\Desktop\Test to create a separate instance of FFMetrics v0.9.2 , on which I was going to run a low level debug command. But before that, I decided to just doubleclick the program's executable file in that folder. And to my surprise, the program window just appeared on my screen, as it normally would. After that, I moved all files from C:\Users\dr0\Desktop\Test to D:\Software\MEDIA-Edit\FFMetrics and it's been working flawlessly ever since.

fifonik wrote on 2/2/2021, 2:40 PM

It could be your anti-virus. Some of them do not like 'fresh' executables.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

lenard wrote on 2/24/2021, 6:05 PM

I compared 1080P YT versions, AV1,VP9 and AVC. The comparisons are made against AV1 with the belief it should be highest quality based on file size and codec. Can you see why VP9 does so poorly in the comparison, to my eyes when viewing the videos I don't see an incredibly poor looking vp9

I uploaded the files if you were interested in looking at them

https://drive.google.com/file/d/1nPVeaIiscG1aIK83EV5CGAWb02dR_7Xl/view?usp=sharing

 

JN- wrote on 2/24/2021, 8:35 PM

@lenard Looking at the files mediainfo, I think that the naming of the files is incorrect. The AV1 named file is avc x264 and the AVC named file is AV1.

Even so, the data rates and file sizes aren't similar. Also the original is needed to compare others with, of ~ similar data rate and ~ file sizes to get a proper quality comparison, i.e. 4 files.

Last changed by JN- on 2/24/2021, 8:36 PM, changed a total of 1 times.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

lenard wrote on 2/24/2021, 9:09 PM

@JN- Thanks, I corrected that. Now I can see that the AV1 at it's small size isn't necessarily better quality, although I've heard it is. File size is much better rate controlled with the AV1 and VP9, if you post a black screen you'll have small files but the AVC won't be much smaller, but in this video example there is a lot of constant activity so small file sizes likely does mean lower quality

The problem with AV1 comparisions is that I can't upload a file as YT won't give an AV1, i had thought this might be the next best thing to try, but given how small the AV1 is I"m not really sure of it's quality

fifonik wrote on 2/25/2021, 2:01 AM

Interesting results. I think something wrong with ffmpeg's VP9 decoder.

I created simple AviSynth wrappers for all videos, like this (LWLibavVideoSource is the only source filter I have that decode AV1 without installing separate codec):

LWLibavVideoSource("AV1.mp4")
ConvertToYV12(matrix="PC.709")

Then I compare:

- AV1.avs (ref) vs AV1.mp4 (distorted) -- PSNR -- inf, SSIM - 1 (as it should be: the same file decoded twice with two different decoders)

- AVC.avs (ref) vs AVC.mp4 (distorted) -- PSNR -- inf, SSIM - 1 (as it should be, see above)

- VP9.avs (ref) vs VP9.webm + AVC.mp4 (both distorted) -- VP9.avs vs VP9.webm results are similar to yours:

So VP9.webm decoded with AviSynth's LWLibavVideoSource and VP9.webm decoded with ffmpeg -- two very different beasts.

P.S. I have not tried to upgrade ffmpeg and run it from command prompt directly yet.

Last changed by fifonik on 2/25/2021, 3:29 PM, changed a total of 10 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

lenard wrote on 2/25/2021, 7:33 AM

With that information I converted the vp9 to mp4, so not lossless but the results that were expected are now seen with vp9 being higher quality than AVC, or to be more accurate having more common with the AV1 which is assumed to have the higher quality

fifonik wrote on 2/25/2021, 3:57 PM

This looks like what I'm getting when analyzing your files through the AviSynth wrappers:

The situation is almost the same if I use AV1.mp4, AVC.mp4 and VP9.avs:

So yes, the file encoded with VP9 have better metrics.

P.S. I tried to use FFMpeg 4.3.2 build 2021-02-20 by gyan and FFMpeg N-101295 build 2021-02-25 by BtbN -- the issue is the same. Using FFMpeg from command line gives the same results.

P.P.S. While "playing" with your files + AviSynth I've found a bug in FFMetrics. It crashes sometimes. Going to fix it, thanks.

Last changed by fifonik on 2/25/2021, 5:41 PM, changed a total of 4 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

fifonik wrote on 3/22/2021, 6:06 AM

Version 0.9.4b published

- New:    Progress is shown in task bar (#20)
- Change:    5, 10, 15, 20, 30 minutes added to "Analyze" duration dropdown (#14)
- Change:    FFMpeg external process timeout for getting media info increased from 2 to 5 seconds (should help with #19)
- Change:    Program version number returned to title
- Bugfix:    "Save Project As" not updating title (#21)
- Bugfix:    Metrics checkboxes not ticked on the 1st program run as they should
- Bugfix:    Crash if integer returned by FFMpeg as frame metric value ("42", expected: "42.0")
- Bugfix:    Crash when FFMpeg process finished prematurely with partial output
            On my system this was caused by combination of LAVSplitter + AviSynth + DirectShowSource(VP9 or AV1)

Last changed by fifonik on 3/22/2021, 6:07 AM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

Peter_P wrote on 4/17/2021, 6:32 AM

Thank you again for this excellent tool !!! I now got the time and need to use it.

I just tried Version 0.9.4b with the actual  ffmpeg-4.4-essentials_build.7z and also with ffmpeg-2021-04-14-git-c5ca18fd1b-full_build.7z and keep getting this error on VMAF. Any hint, what I’m doing wrong?

Here is a part of the log:

Stream mapping:
  Stream #0:0 (hevc) -> settb (graph 0)
  Stream #1:0 (hevc) -> settb (graph 0)
  libvmaf (graph 0) -> Stream #0:0 (wrapped_avframe)
  Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
libvmaf INFO `compute_vmaf()` is deprecated and will be removed in a future libvmaf version
libvmaf WARNING could not read model from path: "C:/myProgramms/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.pkl"
libvmaf WARNING pkl model files have been deprecated, use json
libvmaf ERROR problem loading model file: C:/myProgramms/FFMetrics/vmaf-models/vmaf_4k_v0.6.1.pkl
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    encoder         : Lavf58.78.100
  Stream #0:0: Video: wrapped_avframe, yuv420p(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn (default)
    Metadata:
      encoder         : Lavc58.136.101 wrapped_avframe
  Stream #0:1(eng): Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2021-04-17T10:35:20.000000Z
      handler_name    : Mainconcept MP4 Sound Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc58.136.101 pcm_s16le
[Parsed_libvmaf_4 @ 000002001bbdeec0] libvmaf encountered an error, check log for details
Error while filtering: Invalid argument
Failed to inject frame into filter network: Invalid argument
Error while processing the decoded data for stream #1:0
Conversion failed!

fifonik wrote on 4/17/2021, 7:44 AM

You need to go to vmaf-models folder and unpack archive with json models.

P.S. Supported models auto-detection will be available in the next version. Already implemented.

Last changed by fifonik on 4/17/2021, 7:46 AM, changed a total of 2 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

Peter_P wrote on 4/17/2021, 8:06 AM

Thanks, works fine now.

Peter_P wrote on 4/18/2021, 2:29 AM

@fifonik

I'm very happy with your tool but have one small suggestion. What about a preview thumb as shown here ?

fifonik wrote on 4/18/2021, 3:12 AM

It is very tricky to "extract" exact frame using ffmpeg. I'm still trying to find out how to do this correctly.

Last changed by fifonik on 4/20/2021, 9:24 PM, changed a total of 1 times.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19

fifonik wrote on 4/20/2021, 9:24 PM

I implemented frame extraction (it will be used for extracting bad frames).

However, I still do not really understand benefits from having tiny ref's thumbnail displayed there.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 5700X, RAM: G'Skill 16 GB DDR4@3200, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), Samsung 870 Evo, HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19