FFMetrics -- Yet another program for quality metrics calculation

Kommentare

fifonik schrieb am 20.07.2020 um 09:42 Uhr

Created project page on github & released version 0.7.1.

Changes:

- Added command line options

- Changed format for results: metric names moved to header so now it is easier to open/use it in Excel.

Zuletzt geändert von fifonik am 29.07.2020, 00:39, insgesamt 2-mal geändert.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

JN- schrieb am 20.07.2020 um 09:57 Uhr

@fifonik “Using ffmpeg only it is not possible to get video stream properties (including resolution)

Agreed. Thats why I used ffprobe in my Drag n Drop batch files to get the height, width, duration, fps values.

I used the “Height” value to either select FHD or 4K. The height width was chosen because it catches both versions of 4K, UHD and 4KDCI.

This is a typical sample output to "TargetEXIF.txt" from a file being tested using ffprobe. it's nicely targeted to the specific items only ...

width=1920
height=1080
r_frame_rate=25/1
avg_frame_rate=25/1
duration=27.000000

 

Zuletzt geändert von JN- am 20.07.2020, 22:39, insgesamt 1-mal geändert.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

Peter_P schrieb am 20.07.2020 um 14:55 Uhr

@fifonik

Thanks, looks very interesting. Currently I'm too busy but I will try it later.

wwaag schrieb am 20.07.2020 um 18:16 Uhr

@fifonik@fifonik

“Using ffmpeg only it is not possible to get video stream properties (including resolution)”

Sure you can. I do it all of the time in different HOS tools. All it takes is to run ffmpeg -i <filepath>. No need for ffprobe.

AKA the HappyOtter at https://tools4vegas.com/. System 1: Intel i7-8700k with HD 630 graphics plus an Nvidia RTX4070 graphics card. System 2: Intel i7-3770k with HD 4000 graphics plus an AMD RX550 graphics card. System 3: Laptop. Dell Inspiron Plus 16. Intel i7-11800H, Intel Graphics. Current cameras include Panasonic FZ2500, GoPro Hero11 and Hero8 Black plus a myriad of smartPhone, pocket cameras, video cameras and film cameras going back to the original Nikon S.

john_dennis schrieb am 20.07.2020 um 20:20 Uhr

Just practicing...

C:\WINDOWS\system32>ffmpeg -i E:\C0001.MP4


ffmpeg version 4.2.3 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.1 (GCC) 20200523
  configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000018d35ba8080] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1001
[mov,mp4,m4a,3gp,3g2,mj2 @ 0000018d35ba8080] st: 0 edit list 1 Cannot find an index entry before timestamp: 1001.


Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'E:\C0001.MP4':
  Metadata:
    major_brand     : XAVC
    minor_version   : 16785407
    compatible_brands: XAVCmp42iso2
    creation_time   : 2018-06-19T03:27:45.000000Z
  Duration: 00:00:32.04, start: 0.000000, bitrate: 98461 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709/bt709/iec61966-2-4), 3840x2160 [SAR 1:1 DAR 16:9], 96231 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      creation_time   : 2018-06-19T03:27:45.000000Z
      handler_name    : Video Media Handler
      encoder         : AVC Coding
    Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      creation_time   : 2018-06-19T03:27:45.000000Z
      handler_name    : Sound Media Handler
    Stream #0:2(und): Data: none (rtmd / 0x646D7472), 245 kb/s (default)
    Metadata:
      creation_time   : 2018-06-19T03:27:45.000000Z
      handler_name    : Timed Metadata Media Handler
      timecode        : 06:24:20:02

fifonik schrieb am 20.07.2020 um 23:13 Uhr

Thanks @wwaag.

When I googled I only found that ffprobe must be used for this. Never tried this simple command. Will check it out.

UPDATE: calling ffmpeg -i file reporting some input file properties, but causing ffmpeg to finish with error "At least one output file must be specified". This does not look like proper way to use external program. However, the following works without returning an error: "ffmpeg -i file -vframes 1 -t 1 -f null -"

Zuletzt geändert von fifonik am 20.07.2020, 23:34, insgesamt 2-mal geändert.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

Musicvid schrieb am 22.07.2020 um 23:38 Uhr

Is there a Windows executable available?

fifonik schrieb am 23.07.2020 um 00:14 Uhr

Sure, it can be downloaded from the github project page (there is no source codes available at this moment).

Go to releases, open assets and here is the archive.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

fifonik schrieb am 23.07.2020 um 02:39 Uhr

I have had some "fun" parsing media info returned by ffmpeg. So, in the next version simple media info will be available withing the program. Unfortunately, it is limited what I can get using ffmpeg. For example, I cannot get interlaced/progressive info in general. This can be done using ffprobe, but still not very reliable and not easy so I will leave it for now.

As the frame size will be available, it would be possible to limit available VMAF models. However, I read about these models and found out that it is perfectly fine to use 4K model on 1080p and non-4K model on 4K. This is in their FAQ and in the separate article. So I will not implement any limitations on models availability. I will only make changes so if reference file is 4K then 4K model will be selected automatically in the dropdown. Can be changed by user after that.

UPDATE: @wwaag found a reliable way to detect interlaced/progressive using ffmpeg. It works. Thanks a lot!

Zuletzt geändert von fifonik am 23.07.2020, 12:46, insgesamt 2-mal geändert.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

fifonik schrieb am 24.07.2020 um 09:47 Uhr

Released version 0.7.5b. It shows mediainfo for reference file. VMAF model selected automatically based on frame size. Also, now it is possible to select folder for storing frames' metrics csv files.

Zuletzt geändert von fifonik am 24.07.2020, 09:50, insgesamt 1-mal geändert.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

JN- schrieb am 24.07.2020 um 13:14 Uhr

@fifonik Glad to see your back on track, otherwise your results would have been regarded as simply relative, not absolute results. “VMAF model selected automatically based on frame size.

I'm not sure whether you are implementing the following, i.e. disabling tuning. The authors of the article below think that its critical? http://x265.org/compare-video-encoder/

The following is from the above link ...

“What is Tuning?

• Switches in encoding string that enables tuning (and disables these features)

ffmpeg –input.mp4 –c:v libx264 –tune psnr output.mp4

• With x264, this disables adaptive quantization and psychovisual optimizations

 

Why So Important

• Major point of contention:

• “If you’re running a test with x264 or x265, and you wish to publish PSNR or SSIM scores, you MUST use –tune PSNR or –tune SSIM, or your results will be completely invalid.”

http://x265.org/compare-video-encoder/

• Absolutely critical when comparing codecs because some may or

may not enable these adjustments

You don’t have to tune in your tests; but you should address the issue and explain why you either did or didn’t”

 

Zuletzt geändert von JN- am 24.07.2020, 13:17, insgesamt 1-mal geändert.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

fifonik schrieb am 24.07.2020 um 13:38 Uhr

I'm not sure what are you talking about. The tuning options can be used during encoding, not when one analyzing already encoded videos. Full commands that are used for getting metrics can be found in log.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

JN- schrieb am 24.07.2020 um 14:43 Uhr

@fifonik I think I understand it now. In the article they gave ffmpeg syntax examples, as I recollect, (cannot get to link now) before and after encoding. So maybe they were advising not to use tuning with the encoded samples that were later used for testing.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

JN- schrieb am 24.07.2020 um 18:21 Uhr

@fifonik So, just add total processing duration display on screen when complete.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

fifonik schrieb am 28.07.2020 um 12:35 Uhr

Version 0.8.0b released

- New:    Overall progress indicator added (program is trying to estimate processing time, not percentage of processed files)
- New:    Icons added to buttons
- New:    VMAF option "Harmonic Mean" + command line parameter "-vmaf-harmonic-mean" added
- New:    VMAF option "Phone" + command line parameter "-vmaf-phone-model" added
- New:    Results file can be specified
- New:    Command line parameter "-save-results-file=C:\Path\To\Results.csv" added
- New:    Program now reads config file "FFMetric.conf". Lavfi template and some other params could be adjusted there (simple example included).
            For example, one can add calculation VMAF/PSNR, VMAF/SSIM, VMAF/MSSIM, so they will be stored in frames' metrics csv files.
- Change:    Bool command line parameters now can be specified in a few different ways: "-option", "-option=1" or "-option=true"
            Note: "-option=0", "-option=false" or "-option=any-other-value" will NOT turn ON the option any longer

 

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

JN- schrieb am 30.07.2020 um 21:18 Uhr

@fifonik "Overall progress indicator added (program is trying to estimate processing time, not percentage of processed files)"

Ok, nice addition, STOP is also nice, but is there no total time taken, from start to finish, maybe I'm missing it?

Zuletzt geändert von JN- am 30.07.2020, 21:18, insgesamt 1-mal geändert.

---------------------------------------------

VFR2CFR, Variable frame rate to Constant frame rate link to zip here.

Copies Video Converts Audio to AAC, link to zip here.

Convert 2 Lossless, link to ZIP here.

Convert Odd 2 Even (frame size), link to ZIP here

Benchmarking Continued thread + link to zip here

Codec Render Quality tables zip

---------------------------------------------

PC ... Corsair case, own build ...

CPU .. i9 9900K, iGpu UHD 630

Memory .. 32GB DDR4

Graphics card .. MSI RTX 2080 ti

Graphics driver .. latest studio

PSU .. Corsair 850i

Mboard .. Asus Z390 Code

 

Laptop… XMG

i9-11900k, iGpu n/a

Memory 64GB DDR4

Graphics card … Laptop RTX 3080

fifonik schrieb am 30.07.2020 um 22:30 Uhr

Yes, no time for now. May be later.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

fifonik schrieb am 04.08.2020 um 23:21 Uhr

Version 0.8.1b released.

Added ability to save graph as PNG.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

jason-b schrieb am 16.09.2020 um 21:23 Uhr

@fifonik Nice work! The visualization is excellent. I need to create analysis for mxf files, but the file dialog filter prevents selecting them. I've used ffmpeg from the cli and it successfully decodes mxf and provides results, but I'd rather be able to do this in your tool.

I tried editing the path directly in the text field, but ffmetrics barfs. Something that can be added in a future release?

fifonik schrieb am 16.09.2020 um 22:38 Uhr

@jason-b If ffmpeg cli is able to calculate metrics for mxf files, you can try just drop file from explorer to ref field & files list field OR in File open dialog switch from "Video files" to "All files"

Please let me know if this works.

Thanks.

Zuletzt geändert von fifonik am 17.09.2020, 08:25, insgesamt 1-mal geändert.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

jason-b schrieb am 17.09.2020 um 17:57 Uhr

@fifonik I did try both of your suggestions, and no matter which method I use, any mxf file into the 'Reference' field, makes the app go non-responsive. When setting the "All Files" filter in the open file dialoag, I can choose any random file and FFMetrics just accepts it. The only exception that I've found is with mxf.

Oddly, I can select mxf as the output file to compare - just not as the input/reference file - and it works great.
 

fifonik schrieb am 18.09.2020 um 01:15 Uhr

@jason-b mxf as output file for compare? What does it mean?

It sounds like my routine that analyze the file and display short media info under the ref input field cannot handle your file.

I rendered mxf file in VP17 using Sony MXF encoder (1080-25p HQ 35 Mbps VBR, see picture below) and it opened in FFMetrics with no issues as Reference when I selected All Files.

It would be nice if you supply short example of problematic file so I can investigate.

Zuletzt geändert von fifonik am 18.09.2020, 02:59, insgesamt 3-mal geändert.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer

jason-b schrieb am 18.09.2020 um 02:35 Uhr

@fifonik I was referring to the processing files window. I'm completely able to add a processing file that's mxf.

Let me see what I can do to generate a sample. I'm using Sony XDCAM50 files.

Thanks for looking into this. I do appreciate the help and the tool.

fifonik schrieb am 18.09.2020 um 04:48 Uhr

@jason-b I've tried to render using SonyMXF HDCAM SR and also everything works as expected (well, scan type was not detected correctly for this file so I fixed it).

So I cannot fix the issue without example of problematic file.

Zuletzt geändert von fifonik am 18.09.2020, 04:48, insgesamt 1-mal geändert.

Camcorder: Panasonic X1500 + Panasonic X920 + GoPro Hero 11 Black

Desktop: MB: MSI B650P, CPU: AMD Ryzen 9700X, RAM: G'Skill 32 GB DDR5@6000, Graphics card: MSI RX6600 8GB, SSD: Samsung 970 Evo+ 1TB (NVMe, OS), HDD WD 4TB, HDD Toshiba 4TB, OS: Windows 10 Pro 22H2

NLE: Vegas Pro [Edit] 11, 12, 13, 15, 17, 18, 19, 22

Author of FFMetrics and FFBitrateViewer