FFMetrics -- Yet another program for quality metrics calculation

Comments

fifonik wrote on 7/20/2020, 2:42 AM

Created project page on github & released version 0.7.1.

Changes:

- Added command line options

- Changed format for results: metric names moved to header so now it is easier to open/use it in Excel.

Last changed by fifonik on 7/28/2020, 5:39 PM, changed a total of 2 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

JN- wrote on 7/20/2020, 2:57 AM

@fifonik “Using ffmpeg only it is not possible to get video stream properties (including resolution)

Agreed. Thats why I used ffprobe in my Drag n Drop batch files to get the height, width, duration, fps values.

I used the “Height” value to either select FHD or 4K. The height width was chosen because it catches both versions of 4K, UHD and 4KDCI.

This is a typical sample output to "TargetEXIF.txt" from a file being tested using ffprobe. it's nicely targeted to the specific items only ...

width=1920
height=1080
r_frame_rate=25/1
avg_frame_rate=25/1
duration=27.000000

 

Last changed by JN- on 7/20/2020, 3:39 PM, changed a total of 1 times.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

Peter_P wrote on 7/20/2020, 7:55 AM

@fifonik

Thanks, looks very interesting. Currently I'm too busy but I will try it later.

wwaag wrote on 7/20/2020, 11:16 AM

@fifonik@fifonik

“Using ffmpeg only it is not possible to get video stream properties (including resolution)”

Sure you can. I do it all of the time in different HOS tools. All it takes is to run ffmpeg -i <filepath>. No need for ffprobe.

john_dennis wrote on 7/20/2020, 1:20 PM

Just practicing...

C:\WINDOWS\system32>ffmpeg -i E:\C0001.MP4


ffmpeg version 4.2.3 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200523
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000018d35ba8080] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1001
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000018d35ba8080] st: 0 edit list 1 Cannot find an index entry before timestamp: 1001.


Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'E:\C0001.MP4':
  Metadata:
    major_brand     : XAVC
    minor_version   : 16785407
    compatible_brands: XAVCmp42iso2
    creation_time   : 2018-06-19T03:27:45.000000Z
  Duration: 00:00:32.04, start: 0.000000, bitrate: 98461 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709/bt709/iec61966-2-4), 3840x2160 [SAR 1:1 DAR 16:9], 96231 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      creation_time   : 2018-06-19T03:27:45.000000Z
      handler_name    : Video Media Handler
      encoder         : AVC Coding
    Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2018-06-19T03:27:45.000000Z
      handler_name    : Sound Media Handler
    Stream #0:2(und): Data: none (rtmd / 0x646D7472), 245 kb/s (default)
    Metadata:
      creation_time   : 2018-06-19T03:27:45.000000Z
      handler_name    : Timed Metadata Media Handler
      timecode        : 06:24:20:02

fifonik wrote on 7/20/2020, 4:13 PM

Thanks @wwaag.

When I googled I only found that ffprobe must be used for this. Never tried this simple command. Will check it out.

UPDATE: calling ffmpeg -i file reporting some input file properties, but causing ffmpeg to finish with error "At least one output file must be specified". This does not look like proper way to use external program. However, the following works without returning an error: "ffmpeg -i file -vframes 1 -t 1 -f null -"

Last changed by fifonik on 7/20/2020, 4:34 PM, changed a total of 2 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

Musicvid wrote on 7/22/2020, 4:38 PM

Is there a Windows executable available?

fifonik wrote on 7/22/2020, 5:14 PM

Sure, it can be downloaded from the github project page (there is no source codes available at this moment).

Go to releases, open assets and here is the archive.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

fifonik wrote on 7/22/2020, 7:39 PM

I have had some "fun" parsing media info returned by ffmpeg. So, in the next version simple media info will be available withing the program. Unfortunately, it is limited what I can get using ffmpeg. For example, I cannot get interlaced/progressive info in general. This can be done using ffprobe, but still not very reliable and not easy so I will leave it for now.

As the frame size will be available, it would be possible to limit available VMAF models. However, I read about these models and found out that it is perfectly fine to use 4K model on 1080p and non-4K model on 4K. This is in their FAQ and in the separate article. So I will not implement any limitations on models availability. I will only make changes so if reference file is 4K then 4K model will be selected automatically in the dropdown. Can be changed by user after that.

UPDATE: @wwaag found a reliable way to detect interlaced/progressive using ffmpeg. It works. Thanks a lot!

Last changed by fifonik on 7/23/2020, 5:46 AM, changed a total of 2 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

fifonik wrote on 7/24/2020, 2:47 AM

Released version 0.7.5b. It shows mediainfo for reference file. VMAF model selected automatically based on frame size. Also, now it is possible to select folder for storing frames' metrics csv files.

Last changed by fifonik on 7/24/2020, 2:50 AM, changed a total of 1 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

JN- wrote on 7/24/2020, 6:14 AM

@fifonik Glad to see your back on track, otherwise your results would have been regarded as simply relative, not absolute results. “VMAF model selected automatically based on frame size.

I'm not sure whether you are implementing the following, i.e. disabling tuning. The authors of the article below think that its critical? http://x265.org/compare-video-encoder/

The following is from the above link ...

“What is Tuning?

• Switches in encoding string that enables tuning (and disables these features)

ffmpeg –input.mp4 –c:v libx264 –tune psnr output.mp4

• With x264, this disables adaptive quantization and psychovisual optimizations

 

Why So Important

• Major point of contention:

• “If you’re running a test with x264 or x265, and you wish to publish PSNR or SSIM scores, you MUST use –tune PSNR or –tune SSIM, or your results will be completely invalid.”

http://x265.org/compare-video-encoder/

• Absolutely critical when comparing codecs because some may or

may not enable these adjustments

You don’t have to tune in your tests; but you should address the issue and explain why you either did or didn’t”

 

Last changed by JN- on 7/24/2020, 6:17 AM, changed a total of 1 times.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

fifonik wrote on 7/24/2020, 6:38 AM

I'm not sure what are you talking about. The tuning options can be used during encoding, not when one analyzing already encoded videos. Full commands that are used for getting metrics can be found in log.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

JN- wrote on 7/24/2020, 7:43 AM

@fifonik I think I understand it now. In the article they gave ffmpeg syntax examples, as I recollect, (cannot get to link now) before and after encoding. So maybe they were advising not to use tuning with the encoded samples that were later used for testing.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

JN- wrote on 7/24/2020, 11:21 AM

@fifonik So, just add total processing duration display on screen when complete.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

fifonik wrote on 7/28/2020, 5:35 AM

Version 0.8.0b released

- New:    Overall progress indicator added (program is trying to estimate processing time, not percentage of processed files)
- New:    Icons added to buttons
- New:    VMAF option "Harmonic Mean" + command line parameter "-vmaf-harmonic-mean" added
- New:    VMAF option "Phone" + command line parameter "-vmaf-phone-model" added
- New:    Results file can be specified
- New:    Command line parameter "-save-results-file=C:\Path\To\Results.csv" added
- New:    Program now reads config file "FFMetric.conf". Lavfi template and some other params could be adjusted there (simple example included).
            For example, one can add calculation VMAF/PSNR, VMAF/SSIM, VMAF/MSSIM, so they will be stored in frames' metrics csv files.
- Change:    Bool command line parameters now can be specified in a few different ways: "-option", "-option=1" or "-option=true"
            Note: "-option=0", "-option=false" or "-option=any-other-value" will NOT turn ON the option any longer

 

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

JN- wrote on 7/30/2020, 2:18 PM

@fifonik "Overall progress indicator added (program is trying to estimate processing time, not percentage of processed files)"

Ok, nice addition, STOP is also nice, but is there no total time taken, from start to finish, maybe I'm missing it?

Last changed by JN- on 7/30/2020, 2:18 PM, changed a total of 1 times.

---------------------------------------------

Benchmarking thread

Codec Render Quality tables

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop ... (Acer Predator G9-793-77AC)

CPU .. i7-6700HQ Skylake-H

Memory ..32 GB DDR4, was previously 16 GB

Graphics card .. Nvidia GTX 1070

Graphics driver .. latest studio

fifonik wrote on 7/30/2020, 3:30 PM

Yes, no time for now. May be later.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

fifonik wrote on 8/4/2020, 4:21 PM

Version 0.8.1b released.

Added ability to save graph as PNG.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

jason-b wrote on 9/16/2020, 2:23 PM

@fifonik Nice work! The visualization is excellent. I need to create analysis for mxf files, but the file dialog filter prevents selecting them. I've used ffmpeg from the cli and it successfully decodes mxf and provides results, but I'd rather be able to do this in your tool.

I tried editing the path directly in the text field, but ffmetrics barfs. Something that can be added in a future release?

fifonik wrote on 9/16/2020, 3:38 PM

@jason-b If ffmpeg cli is able to calculate metrics for mxf files, you can try just drop file from explorer to ref field & files list field OR in File open dialog switch from "Video files" to "All files"

Please let me know if this works.

Thanks.

Last changed by fifonik on 9/17/2020, 1:25 AM, changed a total of 1 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

jason-b wrote on 9/17/2020, 10:57 AM

@fifonik I did try both of your suggestions, and no matter which method I use, any mxf file into the 'Reference' field, makes the app go non-responsive. When setting the "All Files" filter in the open file dialoag, I can choose any random file and FFMetrics just accepts it. The only exception that I've found is with mxf.

Oddly, I can select mxf as the output file to compare - just not as the input/reference file - and it works great.
 

fifonik wrote on 9/17/2020, 6:15 PM

@jason-b mxf as output file for compare? What does it mean?

It sounds like my routine that analyze the file and display short media info under the ref input field cannot handle your file.

I rendered mxf file in VP17 using Sony MXF encoder (1080-25p HQ 35 Mbps VBR, see picture below) and it opened in FFMetrics with no issues as Reference when I selected All Files.

It would be nice if you supply short example of problematic file so I can investigate.

Last changed by fifonik on 9/17/2020, 7:59 PM, changed a total of 3 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17

jason-b wrote on 9/17/2020, 7:35 PM

@fifonik I was referring to the processing files window. I'm completely able to add a processing file that's mxf.

Let me see what I can do to generate a sample. I'm using Sony XDCAM50 files.

Thanks for looking into this. I do appreciate the help and the tool.

fifonik wrote on 9/17/2020, 9:48 PM

@jason-b I've tried to render using SonyMXF HDCAM SR and also everything works as expected (well, scan type was not detected correctly for this file so I fixed it).

So I cannot fix the issue without example of problematic file.

Last changed by fifonik on 9/17/2020, 9:48 PM, changed a total of 1 times.

Camcorder: Panasonic X920

Desktop: MB: MSI B450M MORTAR TITANIUM, CPU: AMD Ryzen 3700X (not OC), RAM: G'Skill 16 GB DDR4@3200 (not OC), Graphics card: MSI RX580 8GB (factory OC), SSD: Samsung 970 Evo+ NVMe 500MB (OS), HDDs: Seagate & Toshiba 2TB, OS: Windows 10 Pro 1909

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17