(continued) Defrag or not??

OldSmoke wrote on 9/18/2014, 4:44 PM

[I] would you copy from file A to file B, erase file A, copy B to C, erase B, copy C to D, erase C, then store D while having completely thrown away the original copy?[/I]

Do you think this is the way defragmenting software works?

Former user wrote on 9/18/2014, 5:44 PM

Johhmeyer, it's funny in your first link, if you scroll back thru the pages, there are several pages devoted to telling you how to set up a Defrag. The previous pages do not say DON"T defrag. They give you a warning that if your disk is corrupted or has bad sectors, then defragging might cause problems. But it does not say anywhere to not defrag.

the other link warns that if your disk is in trouble already, defragging is a bad idea. That makes sense, but still does not advise against defragging on a normal healthy drive.

Former user wrote on 9/18/2014, 5:47 PM

bob,

Facts you stated

Video files are large- agree
NLE involves more than 1 file- agree
and no. 3 agree.

Now where is a quote about not defragging from the IT people? What is your source? Logic is okay, but logic can be based on bad data as well as good.

Anyway, it is obvious that a lot of this involves conjecture, opinions anecdotal evidence and possible bad drives to start with. I backed my claims, I feel safe defragging my system drives. Probably won't defrag my video drives cause they don't get the workout like the system drive.

I am done here. thanks for the polite and informative conversation to everyone involved.

videoITguy wrote on 9/18/2014, 6:06 PM

As a function within my NLE business, I run download servers that collect dowloaded subscriptions that I pay for. These run all the time and because of the activity - I frequently defrag, optimize, and secure contiguous segments of data. Just did that with purpose, and completed transfer to optical disc storage for longevity.
Defrag is purposeful, it works over the long haul, and I continue to use the process.

Former user wrote on 9/18/2014, 6:19 PM

Bob, do you use shadow copies? I don't.

GeeBax wrote on 9/18/2014, 6:26 PM

If the concern about defragging a drive is that is hastens the demise of the drive, or somehow wears it out faster, then it is important to think about what happens inside a disk drive. The disk platter is driven by a motor that will spin the disks no matter what, so it will be running whether you defrag or not. That is unless it is set up to spin down when not in use.

I am not a fan of setting up drives to spin down when not in use, as the landing and take-off process is potentially the greatest cause of head failure, as the air cushion is not present to protect the heads during these processes.

Even if the drive is brand new and has contiguous files written to it, the heads must step from one track to the next anyway, so all that happens with a heavily fragmented drive is that the stepping occurs over a longer distance, thus the increase in time taken to reach the next sector. The stepping mechanism is a variant on the 'voice coil' method used in speakers, so the most likely points of failure are the bearing or the flexible PCB 'cable' that connects the head to the electronics.

On that basis, I would imagine that the part that is most prone to failure will be the stepping mechanism, and therefore any un-necessary exercising of that part of the drive will hasten its demise.

I consider all disk drives to be consumables, so they get replaced every three years or so to head off any possible disaster. I remove the old drives, label them with detailed information about when they were replaced and roughly what was on them and store them away in case they are needed.

The same goes for SSDs as well, although I have only just replaced my first set of them after the first 3 year period.

videoITguy wrote on 9/18/2014, 6:44 PM

GeeBax, while I am not aware of all the exact mechanics of modern harddrives - my experience tells me that 95% of drive failures have to do with head thrashing on the platter. This may happen for a variety of reasons - and your point is well taken a lot has to do with the period of disk spin-up or spin-down. Disks that spin constantly seem to suffer the least fail rate of all.

farss wrote on 9/18/2014, 7:32 PM

[I]" Now where is a quote about not defragging from the IT people?"[/I]

I never claimed that IT people said one should or shouldn't defrag. I said that IT people make the point the same as mine that "video people" requirements are different to those of people running database applications and servers. I guess that given my not very illustrious career that has spanned a pretty wide gamut of all things IT from SCADA to IBM System 34s running MAPICs to two years hard slog on a SAP rollout for a multinational I too could lay some claim to being an IT person. I normally never see that as relevant to any discussion though, what I say should stand on its own as true or false.

[I]"Logic is okay, but logic can be based on bad data as well as good.'[/I]

Agree entirely. Here though you've agreed with my data and found no flaw with my logic / reasoning.

I just fired up my editing PC and for the first time ever checked what Win7's defragmenter has done. My boot drive is an SSD and thankfully it seems Win7 is smart enough not to defrag that. My other spinning disks that hold all my many media files for multiple projects have never been defragmented by Win7. The most fragmented disk is only 2% fragmented. That's despite me adding and deleting many files both small and large. As pointed in ancient threads about this topic NTFS is pretty smart, left to its own devices it will make an effort to avoid disk fragmentation.

Bob.

Former user wrote on 9/18/2014, 7:44 PM

"Time and again I read high level IT people qualify statements with "unless you're a video guy"

Sorry, Bob, with the quote you said, I thought you were saying that IT say you should defrag "unless you're a video guy". I misunderstood.

I also do not question your credentials. But again, have you thoroughly tested defragging yourself? "

to the point that you can say (xx number of harddrives were tested, x number defragged on regular basis, y number not. Based on these tests the x drives had a shorter lifespan of Z% than the y drives.")

All I am asking is that people who make claims that certain acts performed on a computer can cause harm should make clear if it is opinion, anecdotal or based on test either by them or others that can be documented.

Judging by your comments of how you use your harddrives, it is not surprising that they are only 2% fragmented, and obviously in this case, would not benefit from being defragged.

videoITguy wrote on 9/18/2014, 8:13 PM

As I have said many many times not all defrag routines are alike - If you care to examine that built-in of Win7 - you notice it is scheduled by defautl - but it is a known "lite" defrag app meaning it looks to only the most aggregious situations to work with.
It is also stages defrag - into multiple passes. It also reports on the "lite" side meaning it just chooses to include in reporting what might be the worst case if it exists - otherwise reports simply "defrag not necessary." This is very different from defrag offered from vendors like Executive Software of some years back.

Chienworks wrote on 9/18/2014, 8:15 PM

"Do you think this is the way defragmenting software works?"

OldSmoke, i know this is exactly how defragmenting software works.

Former user wrote on 9/18/2014, 8:31 PM

A good defragment program uses checksums as it transfers data, but yes that is how it works.

johnmeyer wrote on 9/18/2014, 9:02 PM

the other link warns that if your disk is in trouble already, defragging is a bad idea. That makes sense, but still does not advise against defragging on a

But how do you know if your drive is healthy?!!

Did you test it this week? Did you test it today? In the last five minutes?

When will Windows decide to do its next automatic defrag? Murphy's law says that it will happen two minutes after your disk develops its first bad sector.

Hard drives are wonderful in many ways because they "degrade gracefully," and often give you plenty of warning of impending problems. However, if those problems have just started, and you haven't yet noticed, then a defrag will ruin your day.

I don't know why people are still discussing this. For those of you who want to defrag, please go ahead and do it!! It obviously gives you some sort of good feeling, and far be it for me to ruin your day. Odds are, nothing bad will happen.

Not much good will happen either, but I guess I've already said that a few times ...

Oh yes, I've now been running my defragged computer for two days, and I haven't noticed one single improvement. Those tests I did, and posted earlier in this thread, really do not translate to anything useful when actually using the computer to do useful work.

Former user wrote on 9/18/2014, 9:06 PM

How do we know the harddrive life is shortened, since we don't know how long the drive will last anyway?
I mean heck, testing the harddrive for failure could cause it to fail.
Turning on your computer could cause it to fail.

It's funny, we all make our living based on technology that we don't trust.

I have already stated why I started this discussion and it really was not to convince someone to change their ways, it was to say that if we make statements concerning practices with our computers, they should be qualified as to their source.

Thanks Johnmeyer for your contributions to this thread and this forum.

johnmeyer wrote on 9/18/2014, 9:07 PM

But again, have you thoroughly tested defragging yourself? I am not Bob, but I did test defragging myself, and posted the results, and despite the gains in performance when copying files, I still agree with absolutely everything that Bob has said. See my post above for more details ...

musicvid10 wrote on 9/18/2014, 9:25 PM

While all this discussion is going on (87 replies!) I hope someone (maybe jerald) is just passing the time while his tests of compressed HD rendering times complete. As one might guess, I'm not running them this time.

farss wrote on 9/18/2014, 9:37 PM

[I]"But again, have you thoroughly tested defragging yourself?"[/I]

Um no and no plans to do so either. JM's tests while far from thorough are more than enough to reveal no surprises worth anyone devoting months of time and hardware to such a venture.

I do intend to devote some time to trying to work out or at least document a problem that could actually affect all of us i.e. why is external preview different in Vegas 13 and Vegas 9.

Bob.

jerald wrote on 9/18/2014, 10:00 PM

Sorry, musicvid10, I haven't been able to complete my tests, yet. In order to do so I need to find a fragging utility to create a fragmented drive to use as a test bed. Unfortunately I haven't yet found one....

Anyone know where I can find a drive fragging utility???? (just kidding :) )
j :)

johnmeyer wrote on 9/18/2014, 10:06 PM

Anyone know where I can find a drive fragging utility????As someone who was facing the draft during the Viet Nam war, I remember what "fragging" used to mean. When used in that old sense, yes, I could easily write a utility that would frag your drive.

musicvid10 wrote on 9/18/2014, 10:06 PM

jerald,
Just delete some stuff.

Warper wrote on 9/19/2014, 6:24 AM

Let me add some obvious stuff unaware people might miss otherwise...
w1) There are several file systems, even in windows environment. Some are more secure, some are faster in simple scenarios, some are required by legacy devices.
If you use FAT/exFAT, you can meet crosslinks, lost sectors etc. upon computer crash. It does not happen all that often, but FAT is prone to such problems due to its internal organization.
NTFS is server-grade file system that ensures your data is consistent. Your write operation either succeeds or fails. If you computer crashes in between, FS rolls last transaction back and you data is still consistent on file system level. You will have partially written file at worst.
So, if you have options for permanent video storage, choose NTFS in windows environment. The same states true for SSD drives, portable HDD. It's ok to use FAT/exFAT for flash dongles and cards though as they are used for temporaty file transfer and you can replace its data in a means of minutes.

w2) Shadow defragmentation is a myth. Windows 7 has defragmentation utility that can be scheduled automatically, but you have to make sure your computer is online at that time if you want defragmentation to happen at all. My work laptop was never defragmented up to yerstarday, since I always kept it offline at wed 2:00.
It's good idea to make defragmentation of system drive at least once a month (just after you make back-up copy of it).

w3) NTFS is good enough to keep your data in qutie good shape even without defragmentation if you don't shuffle really big amount of small files. Really small files can be kept inside file allocation table and they don't intermix with bigger ones thus not creating a lot of holes in disk structure after deletion.
A big bigger files could be a problem, but up to some point they don't affect big ones. FS does not simply put new data into first free sector. It avoids filling partially used clusters so deleted small files that lived in these now empty sectors don't affect placement of new files (up to some point).
Also from Win7 (or Vista) file in different chunks of 64Mb+ size is not supposed to be fragmented at all. Hard drive will eventually go through many tracks even to read one chunk, so one more track change between chunks is not supposed overhead. It's design decision that is well thought and makes defragmentation utility to avoid unneeded work.

w4) When hard drive is filled up to magic point (about 80% capacity), you face all sort of fragmentation issues file system tried to avoid before. So it's good idea to keep your hard drive filled below 75% capacity

john_dennis wrote on 9/19/2014, 8:46 AM

"[I]Anyone know where I can find a drive fragging utility????" [/I]

The most accurate results of any test would come from starting with the identical image of a fragmented drive each time.

I know more about the "other" fragmentation.

OldSmoke wrote on 9/19/2014, 9:53 AM

This is one of the few articles I could find that talks about defragmenting an audio & video file hard drive http://www.soundonsound.com/sos/jun05/articles/pcmusician.htm

As for my questions regarding the actual process of defragmenting; I was not aware that so many read&writes are required. I thought we get away with two: read A, copy to B, verify B, erase A, copy B to C (final destination), verify C and delete B.

Chienworks wrote on 9/19/2014, 10:09 AM

I was thinking this morning about my (thankfully brief!) experience with good ol' Apple Pascal. It had quite a few things against it right from the start: floppy drive based, needed about 30K--60Kbytes of working storage space while most Apple // systems of the day had 16Kbytes (yes, that's 16,384 bytes!) of RAM, the OS had no provision for fragmented file storage, and of course, it was Pascal.

I suppose it wouldn't have been so bad if the designers had just allocated 60Kbytes of the 140Kbyte diskette as a fixed working space, but since the disks were so small they made it dynamic, allocating and deallocating at need. Well, when the space needed dropped, it became free on the diskette. The system would then store the next small file in that new free spot if it fit. Then suddenly more working space was needed, so the OS had to move that new file to another spot, if it could find one big enough, because it couldn't fragment the working space. Of course, it couldn't fragment any files so it would end up defragging on the spot to try to make room for this file it had to move out of the way.

As you worked throughout the day there was a constant shifting of files around on the diskette to keep making sure there was enough room for the next thing that needed to be stored. This was exacerbated by the fact that the defragging algorithm was the simplest possible and therefore the worst: simply crush the files down toward the beginning of the drive as they will fit, no optimization at all.

If you ran out of space on the oh-so-voluminous 140Kbyte system diskette you could insert another diskette in the 2nd drive. Of course, in order to make sure the system would keep running just in case you decided to remove the system diskette to make room for a 3rd floppy, it would immediately copy the entire OS over to disk #2, if it would fit, and invoke as much defragmenting as possible to get the job done. This scheme meant that every floppy had about 40K used for the OS and up to 60K for the working space, leaving only about 35K for user data, which constantly had to be shifted around to create new space for storing new files that wouldn't fit anywhere else.

And of course, the read/write speed of these diskettes was abysmal. A full copy from one diskette to another, of a whopping 140Kbytes, could easily take 5 to 10 minutes! So you can imagine that all this defragging wasn't exactly speedy either. And being a single-threaded OS, every time a defrag operation started the entire system stopped doing anything else until that was done.

I think for the entire 3 weeks i used it i probably spent 2 hours getting useful (questionably useful) work done and about 70 hours watching it defrag.