nasty jagged lines when keying in V19, but clean in Resolve

Mindmatter wrote on 4/1/2022, 11:20 AM

Hi all

this has been happening a couple of times when I key 4k FS7 footage. ( the pic is zoomed in to HD from a 4k frame in Vegas 19) No matter what plugin I use ( Ignite chroma key, New Blue chroma key pro, Boris Primatte studio), I get those really nasty jagged lines. I've seen all kinds of issues with green screens, but these never before. I tried with the internal MXF as well as prores 422 from a Shogun recorded via HDMI. Yes I could / will probably end up having to use some softening etc, but that's not the solution.
I sent a clip to the good folks at Boris to ask why I didn't get the great almost one click results in Vegas that they show in their demo clip. This is their answer, which I admittedly do not quite understand ( how could I possibly use less comrpession in cam, when I'm on the max bitrate and resolution MXF? - or RGB vs YUV):

"I suspect, but do not know for sure, that compression is the culprit here in regard to why I had to do so much work to generate a workable solution to this shot. And I suspect color management or internal color interpretation in the host which may account for the differences between working with Adobe AE, which is RGB under the hood and the OFX host that you're using, which might be YUV under the hood.

If you can, perhaps you can request a recapture of the footage from the camera using less compression to see if that can facilitate a better time with the keyer. And you might check your color management settings in the host that you're working with."

Could someone help me understand or even solve this? Could it be a camera issue?
Thanks!

Back to post

Comments

john_dennis wrote on 4/5/2022, 12:47 PM

@Mindmatter

"hmmm wondering if there's something wrong with my system then?"

The Chroma Key fX is described as being GPU accelerated...

... it might be worth it to try it with GPU disabled as a test.

Stranger things have happened.

Mindmatter wrote on 4/5/2022, 2:00 PM

Thanks John, I just tried...no luck.

Mindmatter wrote on 4/6/2022, 2:47 AM

could someone from Magix please chime in? I'm really clueless here and I can't work on this particular project unless this is solved. Or can someone suggest another NLE that's good at keying?

lan-mLMC wrote on 4/6/2022, 3:10 AM

Upload your footage and project Settings (even your project .veg file) so as to see whether others can reproduce your question.

In addition, here are some keying tutorials :

lan-mLMC wrote on 4/6/2022, 3:14 AM

If you want to switch to another NLE, you can try Hitfilm, they claim they have the best keying in the world. https://fxhome.com/product/hitfilm-pro

Mindmatter wrote on 4/6/2022, 7:25 AM

Thanks, I already have the Ignite Chroma keyer. It has the exact same problem inside Vegas. I'll give it a shot inside Hitfilm then.
I know how to do a chroma key, have done them for years but never saw this particular issue of such severe jagged lines before. Just trying to find out if it's my camera / footage, my system or Vegas.

lan-mLMC wrote on 4/6/2022, 7:41 AM

You should upload your footage. So we can figure out whether this is all Vegas's problem, or just your Vegas' problem.

RogerS wrote on 4/6/2022, 7:44 AM

could someone from Magix please chime in? I'm really clueless here and I can't work on this particular project unless this is solved.

For that you can file a support request.

Dexcon wrote on 4/6/2022, 7:59 AM

I've read this thread a few times and so far the Project Properties of your project are unknown. If this chroma-key task is part of a bigger project with Project Properties different to those of the video that you want to chroma-key, maybe try a sub-project for just the vertical video that you want to chroma-key and use the same pixel size in Project Properties as per the vertical video. Also, align the De-interlace Method and trial the Resample Mode options, both in Project Properties.

Mindmatter wrote on 4/6/2022, 9:33 AM

My project properties are all adjusted to the footage, ( manually and also to make sure I did the "match video settings" thing ) a totally normal QHD 25p project, and I've gone through trying out every possible project settings, 32 bit, even interlacing, GPU on or OFF, resample ON or OFF, RAM settings etc etc. I don't quite understand your pixel size suggestion, changing it to different from square just deforms the image.
This project is a simple sequence of 5 min QHD MXF or prores from a Sony FS7 ( I always record both codecs , prores simultanously on an atomos ) files. Nothing out of the ordinary, nothing unusual. It just should work as intended. Why is the key clean in Resolve and not in Vegas? What is going on in Vegas that produces the issue? is it a setting, is it a bug, is it a misinterpretation of resolution, is the footage wrongly flagged, is there a conflict between the OFX keyer and the decoder..?

Former user wrote on 4/6/2022, 6:40 PM

This project is a simple sequence of 5 min QHD MXF or prores from a Sony FS7 ( I always record both codecs , prores simultanously on an atomos ) files. Nothing out of the ordinary, nothing unusual. It just should work as intended. Why is the key clean in Resolve and not in Vegas? What is going on in Vegas that produces the issue? is it a setting, is it a bug, is it a misinterpretation of resolution, is the footage wrongly flagged, is there a conflict between the OFX keyer and the decoder..?

Many people here have BCC, create the simplest test project with media and upload to a file sharing site. People can encode and either see jagged lines or not. You can cut your video files down using shutter encoder with 'cut without re-encoding' option

Dexcon wrote on 4/6/2022, 6:43 PM

With pixel size, I was meaning to make the project width and height to match the original s noted video event size which john_dennis noted earlier is 842 x 1878. I wasn't meaning to change the pixel 'aspect ratio'. Sorry for any confusion. The gist of my suggestion was to make a sub-project with the project size (width, height) set at the same size of the video event to be chroma-keyed - if indeed the size of that video event is different in any way to the project's size of 3840x2160. My thinking is also one of your suggestions: a misinterpretation of resolution.

If this makes no difference to the result, then that's one possibility out of the way. If it does solve the jaggy problem, then how to get it back in to the main project and placed over the intended 'background' image? One way in the sub-project is to place a black solid color media generator event on the track and lower its opacity to zero so that it is transparent. Render it using Video for Windows using an apporpriate HD template but if necessary in Customize Template change the Frame Size field to Custom Frame Size so that the height will be 2160 and the width increased proportionally. Change the Video Format field to 'Uncompressed' and check the Render Alpha Channel item at the bottom of the window. Hit OK and render. An .avi render will result, but .avi files are huge. I just did a 5 seconds shot this way rendered to 1920x1080 25fps and it is 4.34 GB in size - that's around 260 GB for 5 minutes. Eek!

Import the render into the main project and R click the render and click on Properties. In the Media tab, change the Alpha Channel field to Straight or Premultiplied so that the video event's transparency is activated.

To be quite honest, if nothing readily solves the problem in the main Vegas Pro project, it would likely be quicker to do the chroma-keying in Resolve, render it in Resolve and import the render into Vegas Pro.

Mindmatter wrote on 4/7/2022, 9:56 AM

Thanks for your time and explanations Dexcon, I'll give that a try.

Mindmatter wrote on 4/7/2022, 12:54 PM

OK so I have a new element.

I dug up this footage from an older project. At the time I was on V18 I think, and I had an FS5 next to an FS7.
The FS5 records in XAVC-L and the FS7 in XAVC-I. Both in 4k, the FS5 only in 8 bit in this case as it can only do HD in 10 bit.
Now look at both pics:

FS5 FS7

As you can see, the FS7 footage has weird artefacts on the vest as soon as it gets keyed. Exact same settings on the FS5 keyer, slightly different gain.
So my only rational explanation would be that Vegas or the keyers seem to have an issue with XAVC-I, or that my FS7 is somewhat broken.

And it gets weirder: Same footage in V18. jagged but otherwise clean. Note no artefacts the vest, which is weirdly pixellated in V19.

Does V19 decode XAVC I differently that V18? Any other ideas?

john_dennis wrote on 4/7/2022, 2:03 PM

@Mindmatter

"Does V19 decode XAVC I differently that V18?"

Using some media that I collected from a previous enquiry:

Vegas 18 FS7 Mediainfo

General
Complete name                            : \\P8z77\e\Cameras\Sony FS7\150_0036.MXF
Format                                   : MXF
Format version                           : 1.3
Format profile                           : OP-1a
Format settings                          : Closed / Complete
File size                                : 67.7 MiB
Duration                                 : 2 s 333 ms
Overall bit rate                         : 243 Mb/s
Encoded date                             : 2018-03-07 15:11:20.000
Writing application                      : Sony Mem 2.00

Video
ID                                       : 2
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High 4:2:2 Intra@L5.1
Format settings, CABAC                   : No
Format settings, GOP                     : N=1
Format settings, wrapping mode           : Frame
Codec ID                                 : 0D01030102106001-0401020201323001
Duration                                 : 2 s 333 ms
Bit rate                                 : 234 Mb/s
Maximum bit rate                         : 240 Mb/s
Width                                    : 4 096 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 1.896
Frame rate                               : 24.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:2
Bit depth                                : 10 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 1.103
Stream size                              : 65.1 MiB (96%)
Color range                              : Limited
Transfer characteristics                 : BT.709

Audio #1
ID                                       : 3
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 2 s 333 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 24.000 FPS (2000 SPF)
Bit depth                                : 24 bits
Stream size                              : 328 KiB (0%)
Locked                                   : Yes

Audio #2
ID                                       : 4
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 2 s 333 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 24.000 FPS (2000 SPF)
Bit depth                                : 24 bits
Stream size                              : 328 KiB (0%)
Locked                                   : Yes

Audio #3
ID                                       : 5
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 2 s 333 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 24.000 FPS (2000 SPF)
Bit depth                                : 24 bits
Stream size                              : 328 KiB (0%)
Locked                                   : Yes

Audio #4
ID                                       : 6
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 2 s 333 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 24.000 FPS (2000 SPF)
Bit depth                                : 24 bits
Stream size                              : 328 KiB (0%)
Locked                                   : Yes

Audio #5
ID                                       : 7
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 2 s 333 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 24.000 FPS (2000 SPF)
Bit depth                                : 24 bits
Stream size                              : 328 KiB (0%)
Locked                                   : Yes

Audio #6
ID                                       : 8
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 2 s 333 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 24.000 FPS (2000 SPF)
Bit depth                                : 24 bits
Stream size                              : 328 KiB (0%)
Locked                                   : Yes

Audio #7
ID                                       : 9
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 2 s 333 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 24.000 FPS (2000 SPF)
Bit depth                                : 24 bits
Stream size                              : 328 KiB (0%)
Locked                                   : Yes

Audio #8
ID                                       : 10
Format                                   : PCM
Format settings                          : Little
Format settings, wrapping mode           : Frame (AES)
Codec ID                                 : 0D01030102060300-0402020101000000
Duration                                 : 2 s 333 ms
Bit rate mode                            : Constant
Bit rate                                 : 1 152 kb/s
Channel(s)                               : 1 channel
Sampling rate                            : 48.0 kHz
Frame rate                               : 24.000 FPS (2000 SPF)
Bit depth                                : 24 bits
Stream size                              : 328 KiB (0%)
Locked                                   : Yes

Other #1
ID                                       : 1-Material
Type                                     : Time code
Format                                   : MXF TC
Frame rate                               : 24.000 FPS
Time code of first frame                 : 04:54:34:16
Time code settings                       : Material Package
Time code, striped                       : Yes

Other #2
ID                                       : 1-Source
Type                                     : Time code
Format                                   : MXF TC
Frame rate                               : 24.000 FPS
Time code of first frame                 : 04:54:34:16
Time code settings                       : Source Package
Time code, striped                       : Yes

Other #3
ID                                       : 11
Format                                   : Acquisition Metadata
Muxing mode                              : Ancillary data / RDD 18
Duration                                 : 2 s 333 ms
Frame rate                               : 24.000 FPS
CaptureGammaEquation_FirstFrame          : HG8009G33
IrisFNumber_FirstFrame                   : 4.362400
FocusPositionFromImagePlane_FirstFrame   : 1.460 m
LensZoom35mmStillCameraEquivalent_FirstF : 38.000 mm
LensZoomActualFocalLength_FirstFrame     : 24.000 mm
FocusRingPosition_FirstFrame             : 72.9187%
ZoomRingPosition_FirstFrame              : 0.0000%
AutoExposureMode_FirstFrame              : Iris Priority Auto
AutoFocusSensingAreaSetting_FirstFrame   : Manual
NeutralDensityFilterWheelSetting_FirstFr : 1/64
ImageSensorDimensionEffectiveWidth_First : 24.003 mm
ImageSensorDimensionEffectiveHeight_Firs : 12.658 mm
CaptureFrameRate_FirstFrame              : 24.000 fps
ImageSensorReadoutMode_FirstFrame        : Progressive frame
ShutterSpeed_Angle_FirstFrame            : 180.0°
ShutterSpeed_Time_FirstFrame             : 1/48 s
CameraMasterGainAdjustment_FirstFrame    : 0.00 dB
ISOSensitivity_FirstFrame                : 1600
ElectricalExtenderMagnification_FirstFra : 100%
AutoWhiteBalanceMode_FirstFrame          : Automatic
WhiteBalance_FirstFrame                  : 4871 K
CameraMasterBlackLevel_FirstFrame        : 3.0%
ExposureIndexofPhotoMeter_FirstFrame     : 1600
GammaForCDL_FirstFrame                   : Undefined
ASC_CDL_V12_FirstFrame                   : sR=1.0 sG=1.0 sB=1.0 oR=0.0 oG=0.0 oB=0.0 pR=1.0 pG=1.0 pB=1.0 sat=1.0

Other #4
Type                                     : Time code
Format                                   : SMPTE TC
Muxing mode                              : SDTI
Frame rate                               : 24.000 FPS
Time code of first frame                 : 04:54:34:16

Vegas 18 FS7 Decode

General
  Name: 150_0036.MXF
  Folder: \\P8z77\e\Cameras\Sony FS7
  Type: Sony XAVC MXF
  Size: 69.33 MB (70,993,456 bytes)
  Created: Saturday, March 10, 2018, 8:43:24 AM
  Modified: Saturday, March 10, 2018, 8:43:33 AM
  Accessed: Thursday, April 07, 2022, 11:55:32 AM
  Attributes: Archive

Streams
  Video: 00:00:02.333, 24.000 fps progressive, 4096x2160x32, AVC_4096_2160_H422IP@L51
  Audio: 00:00:02.333, 48,000 Hz, 24 Bit, 8 Channels, Linear PCM

Other metadata
  Regions/markers: yes
  Command markers: no
  UMID: 060a2b340101010501010d43-13-000000-b82704a2848105cc08004602029979ae
  Source UMID: 060a2b340101010501010d43-13-000000-b92704a2848105cc08004602029979ae

Media manager
  Media tags: no

Plug-In
  Name: mxfxavc.dll
  Folder: C:\Program Files\VEGAS\VEGAS Pro 18.0\FileIO Plug-Ins\mxfxavc
  Format: Sony XAVC / XAVC S
  Version: Version 3.0 (Build 14)
  Company: MAGIX Computer Products Intl. Co.

Vegas 19 FS7 Decode

General
  Name: 150_0036.MXF
  Folder: \\P8z77\e\Cameras\Sony FS7
  Type: Sony XAVC MXF
  Size: 69.33 MB (70,993,456 bytes)
  Created: Saturday, March 10, 2018, 8:43:24 AM
  Modified: Saturday, March 10, 2018, 8:43:33 AM
  Accessed: Thursday, April 07, 2022, 11:51:46 AM
  Attributes: Archive

Streams
  Video: 00:00:02.333, 24.000 fps progressive, 4096x2160x32, AVC_4096_2160_H422IP@L51
  Audio: 00:00:02.333, 48,000 Hz, 24 Bit, 8 Channels, Linear PCM

Other metadata
  Regions/markers: yes
  Command markers: no
  UMID: 060a2b340101010501010d43-13-000000-b82704a2848105cc08004602029979ae
  Source UMID: 060a2b340101010501010d43-13-000000-b92704a2848105cc08004602029979ae

Media manager
  Media tags: no

Plug-In
  Name: mxfxavc.dll
  Folder: C:\Program Files\VEGAS\VEGAS Pro 19.0\FileIO Plug-Ins\mxfxavc
  Format: Sony XAVC / XAVC S
  Version: Version 3.0 (Build 18)
  Company: MAGIX Computer Products Intl. Co.

The build numbers of the plug-ins appear to be different. The literal answer is "yes" since the decoder has changed in some way. The functional answer appears to be "no" based on the following experiment.

john_dennis wrote on 4/7/2022, 2:36 PM

@Mindmatter

I did an experiment with your original media in Vegas 18-527 and Vegas 19-550.

I added a Default Pure Green Screen Chroma Key fX at the Media level and used frame 1:06 in both tests.

Vegas 18-527

Vegas 19-550

Using Save Snapshot to File from each Vegas instance, I put the two .PNG files on to a Vegas 14 timeline. Using Composite Difference, I could see no difference in the videoscopes between the Vegas 18-527 snapshot and the Vegas 19-550 snapshot.

Mindmatter wrote on 4/7/2022, 3:44 PM

thanks so much for your trouble John!
The XAVCI decoder question between V18 and V19 being cleared up, the question remains why Vegas ( or the chroma keyers inside it ) seems to have an issue with XAVCI and not XAVCL from the FS5. The FS5 footage keys so much better and cleaner that there must be a problem somewhere.

john_dennis wrote on 4/8/2022, 12:57 AM

@Mindmatter

I found an anomaly with text media rendered from Vegas Pro 19-550 to XAVC-I with a black background. When I set the Chroma Key in a new project to black to expose a graphic underneath, I get this:

This is not the case when the same project is rendered to XAVC-I with a green background then keyed green.

MagicYUV RGB and Pro Res XQ are much the best, even when rendered with a black background and chroma keyed black.

MagicYUV RGB black background / keyed black

Pro Res XQ black background / keyed black

Mindmatter wrote on 4/8/2022, 3:47 AM

That's quite a finding John! Thanks for looking into this with such persistence.
The first pic looks bad indeed.

I did check with my prores 422 footage tho , it has the same jagged edge issues in my green screen key. ( the pink costume firther up the thread) The only footage that keys really cleanly is the XAVCL from my FS5, even though it's 8 bit.

Which again leads me to the conclusioun that something in my FS7 encoding or sensor readout might be broken ( The Atomos was fed by the Fs7).