Cheap film dust removal

johnmeyer wrote on 3/28/2008, 3:39 PM
Here's a before/after set of clips. Source: 12 fps 16mm 1928 home movie, transferred with my do-it-yourself "Cintel" transfer. The first clip is unmodified. The second has cheap (free, actually) dust removal and flicker removal applied. I'm not too thrilled by the flicker removal because it blew out my highlights (operated on all pixels rather than applying gamma correction), but the dust removal, after six hours of twiddling, did a pretty good job. A second pass could have done more.

And, yeah, I know that a cotton cloth and cleaning solution does wonders, but have you ever tried to remove dirt that has been sitting on emulsion for eighty years? It ain't easy.

BTW, YouTube and/or Sony's web site is being intermittent with this embedded video. Here are the URLs, which you can copy/paste if clicking on the video below doesn't work:




Before


After




Comments

farss wrote on 3/28/2008, 3:51 PM
Pretty impressive for a freebie.
We used to use 90% ethanol or propanol. Had considered building an ultrasonic cleaner before the commercial reality set in and we got out of the business. Which is a pity as it was a lot of fun and rewarding work however somethings got to pay the rent.

Bob.
CorTed wrote on 3/28/2008, 3:58 PM
John,

I am in the middle of transferring 18fps 8mm home stuff, and this looks pretty nice. Couple of questions: Can you elaborate more on the do-it-yourself "Cintel' transfer method?
And what dust removal tool did you use?

Ted
fldave wrote on 3/28/2008, 4:18 PM
That is nice, John. I would also like to know the tool used. Besides the Libya and Rome footage you helped me with 18 months ago, I have some 1930's home movies that have been transfered to DVD of my mother's family that I might want to turn the cleanup tool loose on.
johnmeyer wrote on 3/28/2008, 5:27 PM
My transfer unit is a modified Eiki 16mm projector, where I have removed the shutter, installed a low-wattage bulb, and modified the condenser. I capture at full 24 fps, using 1/1000 shutter speed on my FX1. I then developed software that removes fields captured during pulldown and any duplicate fields (where I get three fields from the same frame of film). This is tricky software, but it works 100%. Using this method, I can get frame-by-frame film transfer like a Cintel or, more appropriately, a Workprinter, but I can do it at full 24 fps (the Workprinter is very slow). I've transferred a lot of film this way.

The dust removal is done using AVISynth and a free tool called "Despot." To make this work better, I use a mask created by motion estimation software from the same Russian programmer. This helps Despot better distinguish dust spots from actual motion. This is especially important, and also incredibly difficult, when the film was created at only 12 fps, because motion jumps a LOT in that amount of time, especially during a tennis match.

Here's the AVISynth script:


# Despot script for B&W 12 fps film with lots of flicker

loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools.dll")
loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\masktools.dll")
loadplugin("c:\Program Files\AviSynth 2.5\plugins\despot.dll")

Avisource("E:\Stan's Film\1928\1928 Football and Tennis.avi")

LoadVirtualdubPlugin("C:\Program Files\Video\plugins\Deflick.vdf","vd_deflicker",8)
ConvertToRGB32()vd_Deflicker(6, 0, 0, 20, 0)

prefilt = ConvertToYV12()
i=prefilt

#Create mask
ml = 100 # mask scale
thscd1 = 800 # scene change

vf = prefilt.MVAnalyse(isb=false, truemotion=true) # forward vectors
cf = i.MVFlow(vectors=vf, thscd1 = thscd1) # previous compensated forward
sadf = i.MVMask(vectors=vf, ml=100,kind=1,gamma=1, thscd1 = thscd1)
msadf=sadf.Binarize() # binary inverted forward SAD mask

vb = prefilt.MVAnalyse(isb=true, truemotion=true) # backward vectors
cb = i.MVFlow(vectors=vb, thscd1 = thscd1) # next compensated backward
sadb = i.MVMask(vectors=vb, ml=ml, gamma=1, kind=1, thscd1 = thscd1) # backward SAD mask
msadb = sadb.Binarize() # binary inverted backward SAD mask

msad = Logic(msadf,msadb,"OR") # combined inverted SAD mask
msad = msad.Expand() # expanded inverted SAD mask
msadi = Interleave(msad, msad, msad) # interleaved 3-frame inverted SAD mask

Interleave(cf,i,cb) # interleave forward compensated, source, and backward compensated

DeSpot(show=0,maxpts=1200,p2=5,mthres=11,p1=22,pwidth=100,pheight=140, mwidth=7,mheight=5,mscene=90,interlaced=false,merode=58,ranked=true, p1percent=25,dilate=2,fitluma=true,blur=2,motpn=false,seg=1, sign=1,extmask=msadi)

SelectEvery(3,1).ConvertToRGB32()


Many of the Despot parameters are optimized for this particular instance and would need to be re-tuned for higher framerate, color, or different motion characteristics.


johnmeyer wrote on 3/29/2008, 12:17 AM
Had considered building an ultrasonic cleaner before the commercial reality set in and we got out of the business.I have an ultrasonic tank. Did you ever actually do this? One of the demos for an ultrasonic cleaner is to put a dark pencil mark on a frosted microscope slide and then dip it into the tank. The pencil mark disappears in 1/4 of a second. I'd hate to see the same thing happen to the silver crystals ...
farss wrote on 3/29/2008, 12:40 AM
No I never got around to trying it. However it did seem to be one way film was cleaned commercially. You do raise a good point though, there's not much holding those silver crystals together, moreso with older film. Still, if you've got a tank you could try an old piece of black film. If it doesn't work no harm done.
We still have this odd ball gadget that clamps two felt pads onto the film. We'd soak them in cleaner and then wind the film by hand through them. Then we got a bit smarter and had the film come off the reel, through the cleaning pads and then into the projector. You need a good length before the projector to give the film time to dry.

I've more or less got all the bits to build a new film scanner, even found a set of C Mount extension tubes for $20 off eBay and I've since found a company that does C Mount Macro lenses and 720p industrial cameras. I could probably build it all for the price of a Workprinter and scan 8mm at 720p, would be slow at 1 to 3 fps though.

Or I could use my EX1, it's got an external shutter trigger, just trigger the camera off the projector's shutter. The optics would be the tricky part though.

Bob.
johnmeyer wrote on 3/29/2008, 12:51 AM
The optics would be the tricky part though.

That's always tricky. My contribution in that arena was the discovery that if you mount a really long lens on the projector, you can point the video (or still) camera directly at the projector lens without the need for an external aerial lens. I passed that tip to Roger Evans (the Workprinter guy) and I think he is using it in his newer machines. That external aerial lens introduces all sorts of nasty chromatic aberrations and is the devil to get aligned properly. By contrast, I can get set up in about one minute.

FWIW, the lens I use on my 16mm Eiki is 100mm.
farss wrote on 3/29/2008, 8:03 AM
Thanks for that tip, that could save a lot of money.
One thought, I wonder if it'd be possible ti do dust busting by scanning each frame twice, once visible and once IR like my Nikon slide scanner does. All it'd take is an IR LED as an illuminator and obviously some fancy code to process it.

Bob.
RalphM wrote on 3/29/2008, 8:33 AM
That is impressive John,

I always get a chuckle out of the discussions on how to make video look like old film - usually gets into how to simulate scratches and dirt/mold on the film. Whereas those who transfer old film are always trying to combat those very problems...
johnmeyer wrote on 3/29/2008, 2:03 PM
I wonder if it'd be possible ti do dust busting by scanning each frame twice, once visible and once IR like my Nikon slide scanner does.
That's a really interesting idea ...

Issues that would have to be solved:

1. Registration. If the process was done by actually running the film two times, once with the white light source and once with an IR source, then the film registration would be slightly different between the two captures. There would need to be a way to re-align the frames. This would require software that could do sub-pixel movement, which is a little tricky.

2. Capture infrared. The older camera sensors are actually extremely sensitive to IR, and actually have a filter added to cut down on IR intensity. I don't know about the newer CMOS cameras. The Nikon and other scanners that use the IR channel for dust removal obviously use sensors that can capture both visible and IR spectrum. So, finding a capture device that has this capability could be tricky.

3. B&W. As you probably know, the IR subtraction doesn't work with all emulsions (Kodachrome doesn't work), and doesn't work at all with B&W film.

It is still very intriguing. I'll look and see if I can find some sources for IR LED bulbs.
farss wrote on 3/29/2008, 4:00 PM
1. Not run it twice. Have an illuminator with both white and IR LEDs.
2. The EX1 is pretty sensitive to IR, the IR filters in front of the sensors can only cut so much, hence problems with magenta shifts when using heavy ND filters that don't include IR cut. One possible problem is focus.
3. Yes, oh yes. Had serious problems with some wierd old emulsions that were home brewed.

Bob.
farss wrote on 3/29/2008, 4:04 PM
Yes,
all the things that everyone wants to 'put back' are the same things that when you shoot film considerable effort is made to remove. Grain causes problems with compositing. There's a regrain function employed after things like wire remove so you don't have a grainless streak in the film. Because contrary to what many say grain is not noise this must be a fairly tricky process.

Bob.