Random thought hit me last night

p@mast3rs wrote on 1/23/2004, 4:48 AM
A friend and i were talking about how cool it would be if you could recover lost bits from a compressed file. While its true the bits that are thrown out are unrecoverable, he borught up a point about PAR recovering damaged files.

The more he explained it to me, the more I thought that this would be possible. My understanding is that PAR takes a little bit of data from multiple files and creates recovery files. When you run a recovery PAR it checks each file's bits and if the file is damaged it automatically recreates the file with the orignal data bits it stored.

Now my question is would this work for digital video since data and video is nothing more than 0's and 1's?

Forgive me for not knowing the the techical jargon to explain it but Im sure someone who knows about PAR recovery and compression can explain a little bit better than I can.

Any thoughts?

Comments

JJKizak wrote on 1/23/2004, 4:54 AM
Youve been watching too much "Threat Matrix" lately. But anything is possible with time, money, and expertise.

JJK
p@mast3rs wrote on 1/23/2004, 5:09 AM
Not really. But you have to admit it makes sense. If you can recover lost data bits from ordinary files which are 1's and 0's then why cant you recover lost bits from digital video files which are also 1's and 0's?
farss wrote on 1/23/2004, 6:35 AM
I have no idea of even what PAR stands for BUT this isn't entirely stupid. Being able to recover lost information relies upon having one importnat thing, redundancy. If all you had was one binary number and one bit of it became corrupted the you're stuffed, you've got absolutley no way of knowing what it was.
However very little in the real world ifs that simple, there are invariably connections between pieces of data. Take a simple document, the same words are repeated many times. It doesn't take much brain power to work out thet if you see me type 'dan' and i'd never mentioned "Dan' anywhere else in the document that I probably meant "and". Soaftware to do this kind of error detection and correction has been around for decades.

Many encoidng systems also include large amounts of error detection and correction. Unfortunately DV isn't one of them, the only error correction is applied at the frame level so it doesn't take much to go wrong and you've got visible errors in the frame.
However that frame doesn't exist by itself. It's very likely that the missing bit is also in adjoing frames. The trick is knowing where it is as the camera may have been panning or zooming or something was moving.

Also redundant data can also be used to enhance resolution. I've seen a doctoral paper on this, the results are not dramatic but still noticeable. The amount of mathmatics involved was way, way over my head, to say nothing of the amount of code. I'd imagine it took serious amounts of computation just to process a few frames.

So yes many things can be repaired or enhanced. Also many things cannot so don't assume you don't need backups.

p@mast3rs wrote on 1/23/2004, 6:53 AM
http://codeccorner.com/usenethelp/SPar_Primer.html is a good page to start with PAR explanation.

I dont have the programming knowledge but I bet its possible Now tell me that wouldnt be an awesome thing.
riredale wrote on 1/23/2004, 10:47 AM
If data is gone and there is no redundancy in the form of extra "error correction" data, then the data is gone. Period.

BUT:

Look at your own experience. You scan in an old still photo, and decide to "clean it up." If there are spots, you blur the area or you use a clone tool to "fill in" the area with a similar neighboring area. You aren't really recovering lost data, just making an educated guess as to what that area looked like in the original.

Lots of software can do the same thing, from "Digital Ice" to CD player circuitry that attempts to mask dropouts. But note that they are not recovering the lost data, just attempting to hide it gracefully so the loss is never noticed.

Some of this recovery could get really sophisticated. For example, imagine a CD player that wanted to mask a .5 second dropout. It could do any of a number of things: (1) just put in an average bit level, based on the last good data it got; (2) copy and paste the last .5 second of good data into the hole (we video NLE editors do this all the time, don't we?); (3) break down the music into fundamentals and harmonics (just like the MP3 encoders do) and fill in the .5 second hole based on the assumption that the music would have continued for that time. Each step involves more complexity, but each step would sound better than the one before. But none of them exactly replace the original data that was in that hole.
Chanimal wrote on 1/23/2004, 11:40 AM
Farss,

There are already applications that fix missing data, attempt to reconstruct other information, and remove corrupt data--one is used within Photoshop--although manual, another is the clip restorer within several audio apps.

In addition, a spell checker is a rather well-used one. In fact, I bet a spell checker could probably recover the meaning of some of the words you just massacred.

But then again... the misspellings were probably just to emphasize your point--and this forum ain't got no native spell checker (I don't know if anything but the resurection can fix my bad Texas speak). ;-)

***************
Ted Finch
Chanimal.com

Windows 11 Pro, i9 (10850k - 20 logical cores), Corsair water-cooled, MSI Gaming Plus motherboard, 64 GB Corsair RAM, 4 Samsung Pro SSD drives (1 GB, 2 GB, 2 GB and 4 GB), AMD video Radeo RX 580, 4 Dell HD monitors.Canon 80d DSL camera with Rhode mic, Zoom H4 mic. Vegas Pro 21 Edit (user since Vegas 2.0), Camtasia (latest), JumpBacks, etc.

RichMacDonald wrote on 1/23/2004, 11:49 AM
>Now my question is would this work for digital video since data and video is nothing more than 0's and 1's?

Yup. Works fine right now. You can take your mpeg or avi and package it up as rars and pars on several discs, archive it, lose a disc or two, and still recover everything.

Was that what you were asking :-?

BTW, rar itself can contain optional redundancy without requiring pars. For example, I can create a rar file, tell the app to add, say, 10% redundancy, the file will be stored 10% larger, then I can scratch my CD to my little toddlers content and still be able to get a 100% accurate recovery. You could think of it as an "internal par within the rar".

RAR/PAR is useful and powerful technology, for those who don't know. Truly a better zip than ZIP for when the storage medium is unreliable.
rcrawfor42 wrote on 1/23/2004, 1:07 PM
PAR recovery depends on having LOTS of extra information. That information is what's used to reconstruct the missing data. If you don't have those extra bits, you can't recover anything -- you can't even tell it's missing! With compressed video the extra information just isn't there, and including it would result in a file just as large (maybe even a bit larger!) than the original video.

Back in EE101, the method we learned for error correction like PAR's involved using (2^n)-1 bits of information, where n is the number of bits you want to be able to fix; as a side effect you'd be able to recognize when more bits had been corrupted. For example, if you wanted to be able to recover from errors in 2 bits out of eight, you'd need three extra bits of data. That increases the size of a "word" by almost half.

(I may have the formula wrong, but it's in the ballpark. That class was more than ten years ago...)

In any case, you can't recreate information that has been lost, even if that loss was done on purpose during compression.