I've been fascinated with the difference mask ever since I saw Ron Scott's You Tube Vegas video on the technique. If you can pull off this technique, basically it would allow you to key out a subject from a background with no green screen and no need for rotoscoping or masking in post. In short, you could swap out the entire background and replace it with something else, which would be awesome if you could do it.

I've tried it about 20 times and the problem I'm finding is, as the name of the compositing mode implies, there has to be a difference between the subject and background, a significant difference in lighting, perhaps of a stop or more. And actually the more the better.

When you are shooting a green screen, there is not supposed to be any shadows on the green screen itself, including shadows cast by the actor. Also, there is supposed to be a least a stop of light difference between the actor and screen.

For the difference mask to work, the same thing applies but in reverse. There can be no shadows on the actor. Any shadows, especially if they match with the background, will create unresolvable transparencies. For the longest time, I was trying to figure out how to light someone so that they would have no shadows on them, but I'm not only finding that such a logistical problem that I might as well just shoot with a green screen/ also no shadows isn't usually flattering either. So, what I'm looking at is doing the reverse, and that is keeping the primary subject in the shadows or flagged with a diffusion screen, and making sure the background is fully lit and brighter.

The bottom line is that a difference mask can theoretically work, but there has to be a significant difference in lighting between the subject and background. In the picture below, the subject was in the shadows and the background was not. The parts that are white are shadows. The problem in this case was that the tree was casting shadows on the ground and some light was getting on the actor through the branches. The difference mask might have worked in this case if I had placed a flag overhead and adjusted my angle so I wasn't seeing any shadows on the ground.