this has been happening a couple of times when I key 4k FS7 footage. ( the pic is zoomed in to HD from a 4k frame in Vegas 19) No matter what plugin I use ( Ignite chroma key, New Blue chroma key pro, Boris Primatte studio), I get those really nasty jagged lines. I've seen all kinds of issues with green screens, but these never before. I tried with the internal MXF as well as prores 422 from a Shogun recorded via HDMI. Yes I could / will probably end up having to use some softening etc, but that's not the solution.
I sent a clip to the good folks at Boris to ask why I didn't get the great almost one click results in Vegas that they show in their demo clip. This is their answer, which I admittedly do not quite understand ( how could I possibly use less comrpession in cam, when I'm on the max bitrate and resolution MXF? - or RGB vs YUV):
"I suspect, but do not know for sure, that compression is the culprit here in regard to why I had to do so much work to generate a workable solution to this shot. And I suspect color management or internal color interpretation in the host which may account for the differences between working with Adobe AE, which is RGB under the hood and the OFX host that you're using, which might be YUV under the hood.
If you can, perhaps you can request a recapture of the footage from the camera using less compression to see if that can facilitate a better time with the keyer. And you might check your color management settings in the host that you're working with."
Could someone help me understand or even solve this? Could it be a camera issue?