Well, there could be a number of reasons for your problem.

The difference between film, which is 24 frames per second and NTSC video, which is 29.97 frames per second, but video interlaced and actually gives you 59.94 fields per second. A field is like a half-resolution frame. Temporal Resolution is the rate at which the motion in the original scene was originally sampled. So video at 59.94 samples per second has 2.5 times the temporal resolution of film at 24 samples per second. I think what you may be seeing is a combination of "Temporal Sampling Judder" and a problem with your MPEG2 encoder's "Motion Search Precision." Or, perhaps, a problem with the film to video transfer if they didn't do the 3:2 pulldown correctly.

"Temporal Sampling Judder" is due to the sample rate being too slow to describe the motion in the scene. The effect is like shining a stroboscope light on the scene. Moving objects are seen to jump from one position to another, rather than being seen to move smoothly.

This type of judder is commonly seen in film-originated material because a 24Hz sampling rate is far too slow for much of the motion in many scenes. It is often referred to as "film judder." It is often seen in backgrounds when the film camera pans horizontally. It is also responsible for the amusing artifact in Westerns, where wagon wheels are seen to go backwards. Since the film was originally sampled at 24 fps, there really isn't much you can do about "film judder."

Part of the way MPEG encoding works is to "predict" which direction objects in the scene are moving, how how hard the MPEG encoder works to analyze the difference between this frame and the next is called "Motion search precision." On some encoders, like TMPGENC, this paramater can be set to produce higher quality results at the expense of encoding time. Film Judder is particularly hard for an MPEG encoder to deal with. Using an encoder that doesn't employ a high quality motion search algorithm will cause the "flag waving" effect you describe. The quality of the decoder can also have something to do with it as well.

It's also possible that your MPEG encoder is having some trouble with the 3:2 pulldown. 3:2 pulldown displays a film frame for 3 video fields, then the next film frame is displayed for 2 video fields, then the next film frame is shown for 3 video fields, etc. This alternating between 2 fields and 3 fields produces the necessary 2.5 multiple of 24Hz, thus producing 60Hz. However, some video frames (which consist of two fields) will have have two identical fields and some frames will have two fields which are parts of two different film frames. This can drive the MPEG encoder's motion serach algorithm crazy. Curiously, PAL video is 50 fields per second, so they just speed up the 24 fps film to 25 fps and show the same film frame for both video fields.

I'm curious, was your original film shot at 24 fps or some other rate, like 18 fps?

John