Those Ghost Images (continued) and FX connections

Richard Jones wrote on 2/21/2014, 5:42 AM

I should like to say a very warm thank you to everyone who helped me with the problem I described under "Ghosted Image on DVD and Nowhere Else" [Link=http://www.sonycreativesoftware.com/forums/ShowMessage.asp?MessageID=884896&Replies=36] but as we've narrowed down the problem area I thought it would be sensible to start a new thread.

We know now that the problem is not with DVDA or the rendering but rests on the Timeline where it shows when previewed at Best (Full). It seems to be hidden when looked at in my usual Good (Full) setting and I have zoomed the Timeline but without being able to see it there. I also know that there are no overlaid events on the track and, as I mentioned before, the problem seems to disappear when I copy and paste everything to a new project.

I have now reviewed every event on the Timeline one by one (over 500 of them - tedious and time costly!) and have now found that the ghost (which is always of another event -- but not necessarily the same one -- from earlier in the Timeline) disappears if an FX is removed from the chain applied to the problem event. It is not always the same FX (e.g. Unsharp Mask) that causes the problem as sometimes the removal of any one of the other FX in the chain deals with it.

It's almost as though the FX itself might contain within it a copy of an earlier event to which it has already been applied and carried that copy with it to be downloaded with the FX when it's applied to a later event.

I'm baffled I'm afraid and would welcome your ideas (not merely to resolve the issue so much as to ensure that I don't get it when working on other projects).

Richard

[I'm working with an up-to-date version of Vegas Pro 12 in Standard Definition in PAL 4:3. My FX are mainly the standard Vegas Plug Ins although I do sometimes use Frederic Bauman's as well. GPU is Off]

Back to post

Comments

Richard Jones wrote on 2/27/2014, 7:40 AM

Paul

I've now rendered this short piece out (as Mpeg2 PAL 4:3 25fps with Progressive Scan as someone suggested earlier even though the Properties setting was LFF), transferred this to DVDA and burnt a disc from there (again as Progressive) and found that this plays fine from a DVD Player.

Bob

I note what you say and think I understand your line of thought but the reality seems to be that selecting Progressive seems to have solved the problem within this particular project --- although heaven knows why.

The FX are the same as I've used in other projects using the same source (i.e. Standard 8mm film stock converted by the same company to AVI at 25fps via a Telecine machine) without any problems resulting. These include FBmn Exposure, ProDad Mercalli, Unsharp Mask, Sony White Balance, Convolution Kernel, Colour Curves, Secondary Colour Correction (mainly but not always for the gamma adjustment for mid tones), Sony Colour Match, Film Grain and Film Effects (these two only on stills or stock imported from another source) and, extremely rarely, HSL or Brightness and Contrast (I try to avoid these last two as much as possible). A good many events use only a couple of these although there are more usually four or five in the chain and very occasionally six.

Richard

johnmeyer wrote on 2/27/2014, 11:36 AM

I am not sure we have come any closer to understanding what caused your problem in the first place, and it sounds like you have it working.

Now that we know that the source video is stored as DV AVI PAL, I have a few suggestions. This is based on many hundreds of hours (or more) of transferring and editing frame accurate film transfers that I do using my Workprinter (8mm / Super 8) and my own frame accurate contraption (for 16mm).

1. I use AVIFrate (simple, free utility) to set the AVI header so that the video plays back at the correct speed. I usually set this the numerator to 16 and the denominator to 1 for 8mm and old 16mm film, and set it to 18 / 1 for Super 8 and some modern (silent) 16mm film. Bob (farss) has a method he uses that lets you do this within Vegas, but I find that this works better for me.

2. I set the Vegas project properties to 16 fps and progressive (or 18 fps, as noted above).

3. This is the key setting: Within Vegas I use a script to set each piece of media to progressive even though, as noted by others above, DV will always come into Vegas showing interlaced. Regardless of the setting of that DV flag, the fact of the matter is that frame-accurate film is progressive, even if it is stored in an interlaced container. It is important that Vegas treat it as progressive.

I also use this same script (Excalibur) to set each event to Disable Resample. This is important for the rendering stage.

4. I do all my color correction (and gamma correction) before I start editing. This sometimes involves cutting each reel into sections, because the color problems almost always change from scene to scene.

5. I then use an AVISynth script to detect (and optionally remove) all the flash frames. Getting rid of these makes a huge difference to the quality of the final result, but is tedious to do without some programmatic assistance.

6. When I render to DVD, I render using the standard DVD Architect template as the starting point. I only change the average bitrate, as needed, in order to fit my material onto a single-sided DVD. The DVD spec requires interlaced, and since you have disabled resample for each event, Vegas will create standard pulldown (field duplication) during the render.

Richard Jones wrote on 2/28/2014, 2:54 AM

John

Thank you. That's fascinating and a very thorough explanation as well. Just a couple of points if I may:-

1. To be clear, you can only use this method when you have a frame accurate transfer from cine at the original fps without any interpolation --- right? If so, I'm not certain where or even if I could find this service in the UK (although admittedly I haven't had time to do any real research).

3. Would you recommend Disabling Resample anyway, even when using a project with a 'Telecined' film at 25fps?

4. Yes. I have to do my editing scene by scene from the camera. It's tedious but seems to be the only way I can get some degree of acceptable consistency.

5. The new scene overexposure problem occurred with many cine cameras as the exposure struggled to reconcile itself with the film as it started to move through the reels and I always try to remove these (another reason for editing scene by scene). I also try to eliminate the splices (whether cement or tape) which were made during the original editing of the cine film itself.

More generally, a 400' reel of cine film runs for about 20 minutes and my approach is to open this in the Trimmer and make transfers scene by scene to the TImeline. I find it easier and quicker to eliminate the flash frames and splices by making my selections from the Trimmer rather than splitting or selecting in the Timeline. This, I know is a matter of choice and others prefer to work only from the Timeline --- each to his own!

Richard

johnmeyer wrote on 3/1/2014, 10:38 AM

always try to remove these [overexposure frames] (another reason for editing scene by scene).Based on this long discussion over at doom9.org:

A script that automatically detect and replace broken frames?

I made significant revisions to my simple script that finds those "flash frames" which happen each time a film camera starts (because the rotating shutter wheel takes a few frames to come up to speed). I then take the frame numbers that result from this script and copy/paste them into the Vegas Edit Details window (after first changing the Vegas ruler to "absolute frames"). I then turn on ripple edit; use Ctrl-Right Arrow to go to the next marker, and then use the arrow keys to quickly look at the frames on either side of the edit. Sometimes the flash extends to the previous frame, and sometimes the camera operator jiggled the camera while pressing or releasing the spring-loaded on/off switch. When I find the first bad frame, I make sure the event (or events) are selected and I then press and hold the shift key, and while holding the shift key, I press the right arrow until I see the first good frame of the new scene. I then press the Delete key on the keyboard. This deletes the section of the event(s) under the region I just created. I then repeat for each frame. It is tedious, but having the flash frames all identified makes it go pretty quickly. I just finished editing amateur footage of four 1959 NFL games (someone took his camera to the Philadelphia Eagles games and filmed all their games -- in color!!) and had to edit 600+ of these flash frames.

It is very useful to do this operation before doing any other editing so that you have as few events on the timeline as possible, although I do apply a "film correction package," which I store as a Vegas preset, prior to doing this edit. This makes sure that I have my film "color curves" correction applied, and that I have the Levels and Color Corrector fX applied to each event so that I only have to make adjustments, if needed, rather than have to take the extra step, for each event, of adding these fX.

Answers to your other questions:

you can only use this method when you have a frame accurate transfer from cine at the original fps without any interpolation --- right?Most of my workflow will work for telecined film (i.e., not frame-accurate), but you certainly cannot feed any of it to the film restoration scripts I've described in the past. If your film is telecined, then you do [I]not[/I] want to use AVIFrate to change the fps header.

[I]Would you recommend Disabling Resample anyway, even when using a project with a 'Telecined' film at 25fps?[/I]Yes. If you do any sort of slow motion on the Vegas timeline, or if you render to a frame rate different from the original, the combination of the pulldown fields already in your film, and the additional "resampled" frames/fields that Vegas will add when going to a new frame rate will all conspire to produce a fine mess. By turning off resample you ensure that Vegas will instead simply add duplicates fields and/or frames in order to achieve the new frame rate. This avoids the softness and fuzziness you get with resampled frames, and also avoids all sorts of weird frame/field cadences (i.e., pattern of duplicated fields) that you might get if you add a second telecine pattern on top of an existing telecine pattern.

BTW, I have developed software that can remove pretty much any telecine pattern that has been added to your film capture. This software works perfectly if the film was actually captured using a frame-accurate capture machine and was then subsequently telecined (i.e., fields added) in order to deliver on PAL DV videotape, for instance. I [I]always[/I] use this software when dealing with such captures, prior to editing. I have used it many times, and it works perfectly.

There is also software, developed by others, that can sometimes deliver what amounts to frame-accurate film from film transfers that have been done using the inferior "film the projector screen" method of film transfer. Unlike the situation I described in the last paragraph which works with 100% reliability, this method does not produce a result that is perfectly identical to a true frame accurate transfer, but in some situations that I have come across, it produced a remarkably good result -- good enough that I was able to use with my film restoration technology.

In case it is of use, here is my newly-modified "flash frame detection" AVISynth script that I used yesterday to find 600+ flash frames in almost two hours of Philadelphia Eagles NFL fooltball plays:

#This script detects flash (too bright) frames.

#It will detect up to three flash frames in a row.

#

#Script now requires RT_Stats by StainlessS

#John Meyer © March 2, 2014



global flashthreshold = 5

filename = "e:\Flashes.txt"

source=AVISource("e:\fs.avi").convertTOYV12().killaudio().Crop(64,64,-64,-64)



#Uncomment the following lines, and comment out the WriteFileIf lines in order to show the stats (for setting "flashthreshold")

#script = """Subtitle("\nprev_luma   = " + String( RT_YPlaneMedian(source) - RT_YPlaneMedian(selectevery(source, 1, -1)) ) + #         "\n" + #         "\nnext_luma   = " + String( RT_YPlaneMedian(source) - RT_YPlaneMedian(selectevery(source, 1, 1)) ) + #         "\nnext_luma2 = " + String( RT_YPlaneMedian(source) - RT_YPlaneMedian(selectevery(source, 1, 2)) ) + #         "\nnext_luma3 = " + String( RT_YPlaneMedian(source) - RT_YPlaneMedian(selectevery(source, 1, 3)) ) + #         "\n" + #         "\nRT_Median    = " + String( RT_YPlaneMedian(source) ), lsp=0)"""

#final=Scriptclip(source, script)

#return final



#This will detect up to three flash frames in a row, and output frame number for first flash

WriteFileIf(source, filename, "    (RT_YPlaneMedian(source)   - RT_YPlaneMedian(selectevery(source, 1,  1)) > flashthreshold  \ 

   && RT_YPlaneMedian(source) - RT_YPlaneMedian(selectevery(source, 1, -1)) > flashthreshold) \ 

   ||    (RT_YPlaneMedian(source)   - RT_YPlaneMedian(selectevery(source, 1,  2)) > flashthreshold  \ 

   && RT_YPlaneMedian(source) - RT_YPlaneMedian(selectevery(source, 1, -1)) > flashthreshold) \ 

   ||    (RT_YPlaneMedian(source)   - RT_YPlaneMedian(selectevery(source, 1,  3)) > flashthreshold  \ 

   && RT_YPlaneMedian(source) - RT_YPlaneMedian(selectevery(source, 1, -1)) > flashthreshold) \ 

   ", "current_frame", append = false)

(edited to reflect new code)

Richard Jones wrote on 3/2/2014, 4:50 AM

Thank you John. As always, hugely knowledgeable, informative and extremely helpful.

Richard

Richard Jones wrote on 3/5/2014, 8:37 AM

Just to complete the story, I've now made a new DVD of the project and the ghosts are gone (thank goodness).

The solution was to set it as Progressive in Properties, to render as such and to load into DVDA and burn using the same Progressive setting. It shouldn't have worked and it doesn't make much sense as the source was AVI interlaced at 25fpsbut it did the trick which is all that matters.

So my warmest thanks to all of you. You are now officially "Ghostbusters" :):):)

Richard

1 … 2 3 4 5

of 5