@Musicvid you can believe me, that I am aware of this :-)

The fact that ffmpeg still does not really use SMPTE Timecodes for frame alignment, trims and offsets really bothers me. But since I'm in Europe with straight 24 or 25 fps in most assets the conversion to miliseconds isn't too hard ;-)

The way I would trim pre-roll frames from the beginning of of a clip is with ffmpeg using the -ss option ahead of the -i input clip specification. Only complication is the command takes a time specification, not a frame count, so you need to compute it out in advance. Once the 2 clips at least start on the same frame, you can compare them for identity with either ffmetrics or ffmpeg directly. But personally, I wouldn't do the identity comparison that way. I'd use a file checksum/hash utility which would take into account everything including frame count and metadata... the one that comes with Windows is called certutil.

While trimming before comparison with ffmpeg you should not forgot:

1. In most cases you will need to use "accurate trim" that require video stream decoding and so it is slow, especially if you trimming quite a lot;

2. In most cases re-encoding will occur so you should save as uncompressed/lossless (even slower, require a lot of disk space, you should be careful with colours);

3. In my experiments sometimes -ss before -i does not work as expected. This is the reason ffmetrics uses 'trim' instead that I've found more reliable.

I'm not sure how hash/checksum can help as here we are not trying to find out if two files are exactly equal.

Version 1.3.2b published
- Bugfix:    High CPU/GPU utilisation caused by hidden Progress Bars with IsIndeterminate="True" (bugfix back-ported from 1.4.7b)

Version 1.4.0b published
- New:    Added a new command line parameter "-exit" that works in conjunction with "-run"
- Change:    Updated components HarfBuzzSharp, SkiaSharp, SkiaSharp.HarfBuzz, SkiaSharp.Views.Desktop.Common and System.Memory
- Bugfix:    Fixed crash that could happen while processing short command line arguments

Version 1.4.1b published
- Change:    Default LAVFI template changed from:
            [0:v]{{main}}[main];[1:v]{{ref}}[ref];[main][ref]{{filter}}
            to:
            [0:v]{{main-trim}},{{main-settb}},{{main-setpts}},{{main-scale}},{{main-setrange}},{{main-format}}[main];[1:v]{{ref-trim}},{{ref-settb}},{{ref-setpts}},{{ref-setrange}},{{ref-format}}[ref];[main][ref]{{filter}}
            All values in double curly brackets are substituted as required.

Version 1.4.2b published
- Change:    Updated components: System.Drawing.Common (6.0.0 => 7.0.0), Newtonsoft.Json (13.0.1 => 13.0.2), OxyPlot.Core, OxyPlot.SkiaSharp, OxyPlot.SkiaSharp.Wpf, OxyPlot.Wpf, OxyPlot.Wpf.Shared (all Oxy*: 2.1.0 => 2.1.2)
- Change:    Changes to make sure that external program (ffmpeg.exe) is closed (possible fix for crash while extracting bad frames)
- Bugix:    For ffmpeg with in-build VMAF models option 'model=path=file.json' is used for external json models instead of old option 'model_path=file.json'. The old option claimed as deprecated, but in fact does not work at all (#107)

Version 1.4.3b published
- New:    A number of bad frames that should be extracted can be configured in FFMetrics.conf (see FFMetrics.conf.example)
- New:    FFMpeg arguments for getting FFMpeg version info, checking VMAF models, getting media info, extracting bad frames and getting thumbnails can be defined in FFMetrics.conf (see FFMetrics.conf.example)
- Change:    Timeout for extracting bad frames increased to 30 seconds.
- Change:    FFMetrics.conf format is changed (see FFMetrics.conf.example)

Version 1.4.3b2 published
- Bugfix:    Fixed bug with escaping FFMpeg command line parameters (introduced in 1.4.3b)

Version 1.4.5b published
- New:    FFMpeg arguments for getting metrics can be defined in FFMetrics.conf (see FFMetrics.conf.example)
- Change:    -r <framerate> added before -i <filespec> to FFMpeg arguments while getting metrics (as per VMAF recommendation)
- Change:    TBR is used instead of FPS if available and not differ too much from FPS (should help with some VFR footage)
- Bugfix:    Skip was ignored when extracting bad frames
- Bugfix:    Project options were not always restored correctly on program startup

Version 1.4.6b published
- New:    Click on metric column header order files by metric value (#98)

Version 1.4.7b published
- Bugfix:    High CPU/GPU utilisation caused by hidden Progress Bars with IsIndeterminate="True"

Version 1.4.8b published
- Change:    "Action | Show plots window" in menu should not be tickable option
- Change:    Updated component: Newtonsoft.Json (13.0.2 => 13.0.3)
 

Great Tool, but I had bad results with 1.3.1 (marked latest on Github). Frame 2,5,8,11,... showed very low values in all 3 Tabs, VMAF below 10 on a libx265 recode with CRF 28. The extracted images did not show this and even recode with CRF 1 had this issue. Finally I now use 1.4.8b and everything is fine.

Seeing a big drop in metrics across the board from beta v1.4.5b onward. Here is a comparison with same everything except the beta versions:

All the earlier versions agree with the v1.4.3b2 results.

Another data point... much less of a difference if the render is done with Voukoder v13.1 (uses ffmpeg6 libs):

The 1st render with the big difference was done with Magix Hevc default template. Both renders qsv.

How big are these mov/mp4 files? Can you share them so I investigate?

The reference file (also in my signature) is pretty big... link here. I just did a smaller Magix qsv default hevc10 render, link here. Difference between versions is similar:

Also just did a quick spot-check of the PSNR & SSIM with a simplified ffmpeg v4 script whose averages are similar to your earlier versions:

ffmpeg.exe-i 'D:\temp\ffm_v20.mp4' -i 'D:\test ffmetrics\hevc-lossless-an.mov' -lavfi psnr -f null -
ffmpeg.exe-i 'D:\temp\ffm_v20.mp4' -i 'D:\test ffmetrics\hevc-lossless-an.mov' -lavfi ssim -f null -

Output:
PSNR y:42.307164 u:54.153887 v:54.924041 average:43.939596 min:37.049516 max:52.728227
SSIM Y:0.968056 (14.956086) U:0.997327 (25.729503) V:0.997748 (26.473767) All:0.977883 (16.552730)

UPDATE: downloaded, investigating.

I checked your files and for me it looks something wrong with them.

FFProbe shows that their framerate, duration and timebase are different:

FFMetrics before 1.4.5 do not use '-r framerate' and use ffmpeg commands similar to commands you provided.

When I run such command, I see warning and error:

For me these errors means that all frames of one stream were shifted to one frame and results are incorrect.

VMAF developers recommend to use the '-r framerate' option (this is what was implemented in FFMetrics 1.4.5)

If I run command with this option, I do not see any warning and/or error any longer:

When I tried to calculate PSNR metric using VQMT 12 (free), I saw errors in log related to different framerates:

Finally, I calculated graphs using VQMT 12, FFMetrics 1.3.1 & 1.4.8b:

1. I would not expect so big spikes as I see on 1.3.1 graph. Their presense is red flag that something is not right;

2. VQMT 12 & FFMetrics 1.4.8 graphs looks similar (VQMT 12 free does not allow to work with large frames so I cropped it, as a result, frames metrics cannot be compared directly).

To be honest, I do not know how correctly calculate metrics using ffmpeg with files you provided :(

However, you can return old behaviour by creating FFMetrics.conf with this content:

{
    "Metric": {
        "Template": "-hide_banner -nostdin -i {{main-src}} -i {{ref-src}} -frames:v {{duration}} -lavfi {{lavfi}} -f null -"
    }
}

Thanks. That worked. It'll let me use your latest beta which I prefer for the way it does not remember previously loaded clips if I save my user-config with just the reference clip. Your original pre-release worked like that.

Btw, the frame rates are identical for those 2 clips. The reference is zraw straight from the camera which happens to use hevc-lossless as it's internal compression format. Which any app that reads hevc can load if moved into an hevc container. The render was made by Vegas with the Magix qsv default hevc preset set to 10-bit.

Apparently ffmpeg is squawking about some difference between the hevc clips but it's not frame-rate:

Reference clip:
 --- codec: HEVC
 --- content width: 3840
 --- content height: 2160
 --- frame rate: 29.97
 --- bit depth: 10
 --- frame count: 921
 --- start offset: 2
 --- duration (sec): 30.664
 --- GOP histogram : 1 @ 0 : 1 @ 246 : 2 @ 250 [ maxGOPlen=250 ]
 --- max GOP length: 250
 --- profile and level: 0x2ba

Vegas qsv render:
 --- codec: HEVC
 --- content width: 3840
 --- content height: 2160
 --- frame rate: 29.97
 --- bit depth: 10
 --- frame count: 921
 --- start offset: 3
 --- duration (sec): 30.6306
 --- GOP histogram : 1 @ 0 : 65 @ 14 [ maxGOPlen=14 ]
 --- max GOP length: 14
 --- profile and level: 0x296

Update: did some more testing with ffmpeg and every render format I tried with Vegas when compared to my reference clip generates that warning message about a timebase mismatch and possibly inaccurate results. Including Voukoder. Comparing my reference clip to an ffmpeg or resolve render does not generate that warning. I think the zcam-created reference clip was made with ffmpeg libs.

Version v1.4.9b 1 published on Feb 2024

• Change: File ordering re-implemented
• Change: Extension m4v added to accepted files list
• Change: Accepted files list can be configured (see FFMetrics.conf.example)
• Change: Updated components: System Drawing Common (7.0.0 => 8.0.2), HarfBuzzSharp (2.8.2.3 => 7.3.0.1), SkiaSharp (2.88.3 => 2.88.7)
• Bugfix: Searching line with file metric returned by FFMpeg is more reliable

Version v1.5.0b 1 published on May 2024

• New: VMAF subsample option added
• Change: File ordering re-implemented. More work as original issue was not fixed in 1.4.9b1
• Change: Updated components: System Drawing Common (8.0.2 => 8.0.4), HarfBuzzSharp (7.3.0.1 => 7.3.0.2), SkiaSharp (2.88.7 => 2.88.8)
• Bugfix: Temp dir availability and write access is checked on program starup
• Bugfix: Message displayed if log file cannot be created
• Bugfix: Fixed crash that might happening while drag & dropping distorted items