I come from working in the fortune 500 corporate IT world, and in the days before SSD drives - we spent a lot of effort on harddrive maintenance. In corporate IT you diagnose a need an solve it with programming automation. For example to deploy a new workstation for an employee, a script is run that downloads the operating system parameters, the necessary permissions to corporate directories etc. You can bet that setting automation of local hard drive defragmentation was a part of the scheme.

Now in the video world, I continue to work with analogue video capture, as well as heavy hard drive usage on a connected 1 gig network with many workstations. Defragmentation is a huge part of operations for many necessary reasons. I am not aware of any stat that shows you should stop defragmentation because it prematurely wears out drives. On the other hand there are many efficiencies gained from using drive clean-up and defrag schedules. The only drives that I have installed that do not defrag are SSD and they are maintained by a basic wipe instead. Note that outboard drives connected by USB3 or Firewire pose their own sort issues with defrag efforts and that is covered by temp install as internal and wiped.

@musicvid10
from one of your posts:
[I]Defragging media storage drives, ever, is a bad idea in my opinion. I won't debate the reasons, but they are real.[/I]

Maybe we can debate it now? Or maybe you can let us know the "real" reasons?

BTW: I am totally with videoITguy having worked myself in IT and now maintaining my own servers.

IN REPLY TO musicvid10 (FROM OTHER THREAD):
---------------------------------------------------------------------------------------------------
jerald,
YOU JUST DON'T GET IT, DO YOU?
Here, I'll save you the trouble.
Subtopic is continued here:
www.sonycreativesoftware.com/forums/ShowMessage.asp?ForumID=4&MessageID=906029

A certain number of off-topic replies are expected with a general question topic such as this, and I contributed to that background discussion. When those replies become excessive, redundant, and even further off-topic, it quickly becomes apparent to anyone reading -- save for the self-indulgent or clueless.

I suggested YOU and ANYONE ELSE interested in ruminating on the subtopic of defragging and drive chaining start another thread; out of respect, simple awareness, and common courtesy. Given the diversion from the original topic, it was not an unreasonable suggestion, made eight posts up.

Instead, you chose to overreact by challenging my veracity with speculation. How dare you put words in my mouth.
Instead, I'll give you permission to be right. Unconditionally. The moment is yours, seize it.

If you will reread my simple request made out of consideration for the OP and the OT, you may also infer that I don't really care in debating something that has already been beaten to death on this forum for as long for as I've been around. And you would be right.

Take it to the new thread that's been APPROPRIATELY titled.
Best.
-----------------------------------------------------------------------------------

musicvid10:
I'm very sorry, sir (that's an apology, nothing more). My bad.

Please disregard my prior posts.
I apparently don't get it. And that's ok. I don't need to.
I never intended to cause unpleasant emotion.

And again, thanks so much for all of your excellent posts over the years. I've learned much and gotten lots of enjoyment from your input. (The previous sentence was not sarcasm. I really mean it.)

Peace.
jerald

appears that jerald is the kind of a peace loving guy and I can't say that I blame him. Take that position and be done - because you notice the bully already fell into his own trap. Life happens!
Peace to all!

And my apologies as well. Carry on; I've had my say on this topic.

musicvid,

Don't disappear just yet. I am -- right at this moment -- doing a long defrag test, and also doing a lot of research, and will be posting the results of both later today or tomorrow.

I really do not understand the somewhat hostile reception you are getting. That is very disappointing, and I hope people can be a little more open and generous in their comments.

Also, remember that LOTS of the "old timers" in this forum and lots of those who have experience as software developers (I fit into both camps) are equally skeptical about the value of doing defrag. I can point to at least half a dozen posts in this forum where defrag has become an issue, and cite lots of posts from familiar names.

I have never before done a defrag, and I own over a dozen computers, and have at least two dozen external drives. However, even though I think defrag is totally unnecessary, and have links to various external test results that seem to confirm this (I will include these in my next post), I am an engineer and a scientist first, and I always keep an open mind.

The proof of that last statement is that I am doing these tests to find out if I am wrong.

The one thing I can tell you for certain as of this moment: defrag takes an extraordinarily long time, and the disk drive is working at a rate that I have never before seen in all my years in the computer business. I don't know if this relentless activity is dangerously stressful, or whether it will lead to premature failure, but I do know this: I definitely had better get some remarkable results to justify both this abuse and also the significant amount of time it takes to complete this process.

This first defrag of my life is being done on the main 1 TB SATA data drive that is a permanent drive in my computer, and has not been defragged since I purchased this computer 3.5 years ago (the defrag report said my drive was VERY defragmented). This defrag is just nearing the halfway point after 3.5 hours, so it looks like it will take about seven hours (using the Windows XP defrag utility) to defrag a 1 TB drive that is about 80% full.

Yes, I know that commercial defraggers often work faster than Windows. I have read dozens of test reviews. I know most of the numbers, and I know all the issues with the way that the tests are done. I'll explain these in my next post.

Results to follow ...

videoITguy,

I am curious about the USB3 connected HDD defrag issue you mentioned.
I have a western digital 4 TB USB3 HDD. Why is it not recomended to defrag such external HDD?


VMP

Johnmeyer, also keep in mind if you have never defragged, it will take quite a bit longer than if you defrag regularly. I originally challenged the people who say not to defrag to provide a source for their information. Other than user forums, I have not seen any evidence that defragging causes excess wear or is totally unnecessary. But I have found several sites that state the benefits of defragging. And these are legitimate technical sites as well as the Microsoft site itself.

I do see a difference if the performance of my system drive when I defragged. No, it is not 200% faster, probably a lot less, but I do notice a difference. On data drives, you wouldn't notice a difference because it depends on how the data is recalled.

The problem here is context.
For sure with a back end application or database disk array worth zillions with thousands of users hanging off it, it makes a lot of sense to do everything possible to optimise disk performance. It's quite possible that the array controllers are smart enough to "understand" the structure of the data as well. Bat that's a very different scenario to where most of us are coming from.

I'd also make the point that defragging a disk and optimizing a disk aren't the same thing. How a disk is best optimized also depends on what it's being used for.

My take on this in our context is the same as "washing and waxing a car makes it go faster.", it's a useful illusion. If it makes you feel good because you've done something you believe will produce a positive outcome, then do it. To those who caution against car washing and disk defragging because it wears the thing out I say "It'll wear out anyway" and add "I've never had something break when I was feeling positive" :)

Bob.

bob,

You are saying it is an Illusion that it is faster? I disagree. I can't quote percentage faster, but I can say there is normally a performance increase. Not an illusion.

Now with my windows 7, it defrags in the background. if you are using Win 7 or 8, you have to intentionally turn this off. So I probably would not see a performance increase if I defragged because it defrags more often than I did when using XP or ME.

To VMP - your question is welcome and the answer becomes a special issue into itself. This is about EIDE and SATA drives that you convert into external drives through adapter interfaces - those interfaces being USB2, USB3, Firewire 400, Firewire 800 and Thunderbolt. What happens with a given interface as well as the services that are built into your Win OS to control drives is something along the lines of a unique handshake. When you move the given interface to a different PC it is possible for the handshake to work differently and most decidedly when you change the interface from say USB3 to Firewire 800 on a given drive - the handshake will be of a different type.

Usually this would not be noticeable to you,... until you try to defrag an external drive as an external drive. If between steps you have re-oriented the drive in interface or WIN OS install - the defrag in worst case will not take, or in mild case work partially.
Moving the drive to an internal interface can work better or as I do - just wipe and start the whole process over. So in my view I have so many external drives - I never bother defragging them but just rotate between wipe and reuse.

Farss and DaveT2 make very good points. 1) Farss, you are correct in that I am speaking of drive optimization and the defrag cycle is just one part of that. 2) DaveT2,
the optimizing practices including the defrag cycle are no illusion. There are improvements in drive performance that are real and measureable - although my goodness no where double digits in value. I venture out on a limb with about 5-7% increase in optimum circumstances, probably 1-2% on average.

@johnmeyer

If your drive is already 80% full, assuming you have emptied your Recycle Bin because if not it might be even more full, defragmenting can take much longer.A drive that has more space to "shuffle" the data around can relocate bigger chunks of a file to the free space, less space means only small parts of a file can relocated. This could also explain why you can hear your drive working overtime.

What I don't understand is, since you are an engineer, why we are not using simple logic to this whole defragmenting debate? I am sure you remember the good old days of the shiny black 33 1/3rpm records? Try to imagine the songs on it are not recorded track per track and track after track but the songs, or even worse parts of it, are in different locations and you would have to move the needle several times just to get one song played.

Granted, modern drives are much bigger and faster and it will take much longer to have it fragmented, but it does happen and it will slow the data transfer rate down.

File servers are certainly more effected then our own PCs because hundreds of users are dumping files, erasing files, dumping new files and within a week you can see already high fragmentation.

What I don't understand is, since you are an engineer, why we are not using simple logic to this whole defragmenting debate? I am sure you remember the good old days of the shiny black 33 1/3rpm records? Try to imagine the songs on it are not recorded track per track and track after track but the songs, or even worse parts of it, are in different locations and you would have to move the needle several times just to get one song played.I am an electrical engineer, and started my career in the microwave test & measurement division of Hewlett-Packard. As for not understanding the principles involved, I understand much better than most because back in the days of MFM drives (which preceded ESDI and IDE), you could download utilities that would let you change the interleave parameters on the drive. For those not familiar with the concept, disk sectors are arranged in concentric rings, not in a spiral groove like a 33 1/3 rpm record. When the disk head moves from one ring to the next, the disk is still rotating. If you arrange all the rings so that the data starts at the same point on the compass, then since the record has rotated while the read head is moving, the drive will have to wait almost an entire revolution until the head can begin to read the next block of data.

To improve performance, each ring is staggered relative to the other rings. This stagger is called "interleave." However, the early hard drives had this stagger set for the slowest computer into which they might be plugged. If you had a much faster computer, you could sometimes get rather astounding gains in performance by changing the interleave.

So yes, I do understand a little about the subject ...

[I]"Farss, you are correct in that I am speaking of drive optimization and the defrag cycle is just one part of that"[/I]

Many years ago I had some disk optimizer application that did the whole thing i.e. not only did it make every file on the disk contiguous, it moved all the files into a contiguous space from the start of the drive thus making all the free space contiguous. This ensures that as files are added to the disk they are not split over discontiguous free space.

The problems though with optimising disk files for our kind of editing don't end there in the real world. Say I start with an empty disk and load two video files from a two camera shoot, then a load two audio files from my audio recorder. This is the worst possible scenario for playing back the files as the head needs to move to each one of those files as the NLE requests data from them. It'd be better if the clusters were interleaved. In fact early audio recorders did this for that reason.

The problem gets worse. We create project files, Vegas creates various files. Sure one could optimize all this but my goodness that's a lot of work for next to nothing.

I also tend to work on projects for different clients over the same period and they maybe on the same drive. What a mess but I long ago gave up losing sleep over it just as I rarely wash and wax a car these days, I'm a bit past it for setting new land speed records. I still don't walk under ladders though :)

Bob.

[I]disk sectors are arranged in concentric rings, not in a spiral groove like a 33 1/3 rpm record[/I]
______________________________________________________________________

I knew someone would jump on it! Great catch! But the point I was trying to make, I hope you got that part, is hat if the file is spread over several sectors and those are not consecutive in any direction, the head will have to move several times more often to get that file.

MFM drives, yes I do remember those days and we did quite a lot of interleave changes on old 20MB drives...yes 20MB! We had those in old IBM computers 8086 machines with amber color monochrome tubes.

OK, my test results are finished.

To cut to the chase: I must eat crow because I did observe a few pretty remarkable performance improvements.

However ...

I am not entirely convinced that it is worth the time for most people to defrag, but for a few select people with special needs, it is probably a very good idea. Not all the tests showed appreciable improvement, and a few actually showed some significant performance reductions. I have seen this same thing in many of the defrag tests I've read online.

My Test Setup

My main computer has two boot drives (XP, Win7) both of which are 15,000 rpm SAS drives. Four years ago, these were the "hot rod" drives prior to the arrival of reasonable SSD alternatives. My main, always-installed data drive is a 1 TB SATA 7,200 rpm drive. Additional disk specs are probably not needed.

I first did a defrag analysis on my C: drive (the SAS drive) and the report said that I didn't need to defrag. Why? Because I never update anything, and I have moved all my temp files to the E: drive. Thus, there is very little fragmentation possibility on this drive because so little new data gets written to it.

The defrag analysis on the E: drive (the SATA drive) showed considerable fragmentation (45% file fragmentation, 22% total fragmentation). I can provide the complete report, if anyone wants it.

Before I did any defrag, I measured the boot time. I started the stopwatch after the lengthy SAS drive BIOS initialize sequence (I hate this wasted time). There is an easy visual indication when this is finished, and I measured the time from that point until the desktop first appeared. I then waited about twenty seconds until all boot disk activity ceases. Since I have no background tasks running (I literally only have six tasks assigned to my user name, and sure as heck don't have any anti-virus software), the time it takes for the computer to "get quiet" happens pretty quickly.

I then measured the time for various applications to open, and then re-booted, and re-did these tests.

My Tests

Next, I got to the real meat of the matter. I did four tests which involved copying a lot of files from my heavily-fragmented SATA drive to a brand new, virgin 3 TB SATA drive, both of them 7,200 rpm.

1. 1,456 files in 22 folders, 2.9 GB, all similar size (JPEG photos).

2. 14 large files from a full DVD prepare, 4.35 B.

3. 2.678 files, 105 folders, 4.95 GB, from a folder that I constantly read & write from. The files are many different sizes.

4. The final test was suggested by reading the Windows defrag report. It showed me the most fragmented files, in descending order, and a large number of them were in one single folder. I don't know why that happened, but since I had the good luck of having that situation, I figured I'd better include that folder in my tests. It had 556 files in 25 folders, totaling 9.30 GB.

I have a brand new 3 TB 7,200 rpm SATA drive that is empty, sitting in one of my removable bays, and I used it as the target for various copy operations. For each of my tests, I copied the entire folder, measured the time, and then copied the resulting folder from the virgin drive back to the fragmented E: drive, putting it into a temporary folder. I realize that the second copy operation might partially be getting information that was cached in RAM, but that should happen the same for both my "before defrag" and "after defrag" tests. If I were doing this for publication, I wouldn't do this second test that way.

When each test was done, I deleted all the files from the two temporary target folders, and did this while pressing the shift key so that the files didn't first go to the (empty) recycle bin.

Here are the results.

I measured all of these results while booted to my Windows XP Pro 32-bit SP3 drive. I did not do any tests under my Windows 7 boot drive.

First, there was a 2.2 second improvement in boot time, from 34.7 to 32.5 seconds. The start time for Word 2003 improved from 2.0 to 1.3; the start time for Firefox 28.0 improved by 0.4 seconds, from 2.3 down to 1.9 seconds. Vegas 10.0e increased slightly from 15.1 to 15.6 seconds, but that program (and all the modern Vegas programs) have a mind of their own when starting, and I never get the same results. I have no idea why it takes so long, especially since I never enable Media Manager. No other application I own takes even half the start time as Vegas.

But here is the interesting stuff.

Test 1, which was nothing but a lot of 2 - 5 MB JPEG files, actually showed a significant degradation after defrag. Before defrag, it took 61.4 seconds to go from my fragmented drive to my virgin drive, and 64.5 seconds to go back the other way. After defrag, these numbers went to 67 seconds and 84 seconds. I have no idea why this happened.

It was a good thing I did the other tests, because if this was my only test, I would have been convinced that defrag was actually a very bad thing.

(BTW, I have seen many tests posted on the Internet that have also shown similar degradation, but only for certain types of data).

Test 2, which was the VIDEO_TS folder from a full DVD prepared by DVD Architect showed some very good results. Copying from the fragmented drive to the virgin drive went from 53 seconds down to 45 seconds. Going the other way was even more impressive: 70.3 seconds went down to 43.3 seconds.

This was the first really impressive improvement. I was surprised to see that the bigger improvement was when trying to write the large files to the fragmented disk.

The next to the last test was writing the mix of large and small files from a folder to which I constantly add and update files. The original test took 145 seconds to write from the fragmented drive to the virgin drive, but this dropped to only 88.9 seconds once I had defragged. Wow!

Oddly, copying these files back to a temporary folder on the fragmented drive took longer, increasing from 92.8 seconds to 108.4 seconds. Not so wow ...

I have absolutely no explanation for this, and I would question my methods if I hadn't done it myself.

The final test, where I purposely chose the folder which contained the most highly-fragmented files did indeed produce some rather spectacular results. This folder is basically a "scratch" folder where I store all manner of files that I download (like video files people want me to fix), but which I do not expect to keep.

Before defragmentation, it took 267.8 seconds to copy from the fragmented to the virgin drive. This dropped to only 98 seconds after defragmentation.


So, while I don't pretend that this is the be-all and end-all of defragmentation testing, it has two advantages over most of the tests you will find on the Internet:

1. It was not done by a defrag vendor, many of whom have skewed their results by including a whole drive full of files like the ones in my last test.

2. It was done using real-world defragmentation. One of the problems I have with the few independent tests I've seen (i.e., not done by a defrag vendor), is that they often artificially create a fragmented disk. The risk in doing this is that it is easy to create a "pathological case" that is not representative of what we see in the real world. This is somewhat similar to what happens when you try to use a test chart to see if some rendering algorithm or noise reduction or re-sizing scheme is better than another: you end up with results and conclusions, but when you try to apply what you learned to real video, the result is often disappointing.

Conclusion

First, defrag can make a significant difference in performance. That was a surprise to me, and I didn't expect it.

Second, that difference may actually not matter for some people. We've had many discussions recently about whether drive performance is a very large part of how long it takes to complete a render, and generally the render is not waiting for the disk to respond. Since there are several places that the data gets cached between Vegas and its final arrival on the disk surface, I'm not sure that anyone is going to see any difference in render speed after doing a defrag. I would expect no difference whatsoever.

I saw only a small difference in application startup time, but my main drive was not very fragmented. However, since I seldom update applications, even if that drive were fragmented, the applications that were installed a long time ago probably are NOT fragmented.


Will I now do defragmentation? Probably not. I don't spend my day copying files, and my computer really didn't operate enough faster for me to really "feel" any difference. I've used my computer for an hour or so since the test completed, and it pretty much feels the same.

I realize that there are other things that many of you would like to know about how defrag affects performance, such as: will defrag improve my timeline playback performance? My educated guess, based on this test, and my knowledge of how video works is that it will make no difference whatsoever unless you are playing back uncompressed files or files compressed with lossless codecs (like HuffYUV) in which case it could actually make an important difference. My test #2 copy operation which involved large VOB files from the VIDEO_TS folder would probably directly translate to the same percentage performance improvement on the Vegas timeline if your video files are uncompressed, or only compressed with a lossless codec.

However, regardless of whether I defrag or not, I will no longer say that defragmentation doesn't make any measurable difference.

just do what i do when the car's ashtrays get full - sell it and buy another one....

i used to religiously defrag uptil win7. since then, and the cost / size of drives, i haven't bothered. and all my computers run ssd as system drives.

if someone can assure me that defragging a half filled 1tb is going to make a 'significant' difference working with video files then i'll happily do so....

How do the times spent rendering some good-sized HD videos (2-4 GB) to/from the system drive, seondary defragged sata drive, and secondary fragged sata drive look? No one should have to eat crow alone, John.
;?)

How does the time spent rendering some good-sized HD videos (2-4 GB) to/from the system drive, seondary defragged sata drive, and secondary fragged sata drive look?As I said in my long post, since I wasn't doing this for publication, and since I still have a day job, I did take a few shortcuts. The biggest shortcut is that I didn't burn a disk image of my 1 TB fragmented drive before I started, so there is no way now to go back and do further tests using the same data. However, as I said in the conclusion of my tome, I am pretty certain that if you are rendering using any codec in Vegas, the compression in that codec will result in a file that is so small that the time to write it to even a 5,400 rpm laptop drive that has never been de-fragmented will probably be a fraction of the bytes/second that the rendering engine in Vegas can crank out.

This is easy to figure: an hour of good-quality AVCHD video consumes something like 10 GB. My worst-case copy took a little less than five minutes to copy 10 GB. Most AVC renders in Vegas happen at less than real-time, but suppose you could do a one hour render at 2x real time. That would be thirty minutes, and the five minutes it takes to transfer the data to the drive would be easily handled by the Windows buffers, disk drive buffers, and DMA controller so that the write operation would happen in parallel, and you'd never see any difference if that data only took 98 seconds to copy instead of almost five minutes, as it did in my tests after defrag was done.

"I am sure you remember the good old days of the shiny black 33 1/3rpm records? Try to imagine the songs on it are not recorded track per track and track after track but the songs, or even worse parts of it, are in different locations and you would have to move the needle several times just to get one song played.

While that might be a good description to introduce the subject of fragmentation to people unfamiliar with it, it doesn't really hold much weight as an argument. When's the last time you played the entire contents of a platter of one of your hard drives beginning to end sequentially? Have you ever? Do you ever expect to? Would you even know whether it has happened or not?

Now, suppose you had a record player that could read the grooves faster than the music played back and buffer ahead a bit. And suppose someone took advantage of this to record all the left channel first on the outer edge of the record and the right channel closer to the center. The record player could still produce stereo by "reading" a few seconds of the left channel, moving the stylus to the inner portion and reading a few seconds of the right channel, combining them, and then playing that part out in stereo while the stylus went back to fetch the next left & right sections. Surely a ridiculous scenario for a record player, right? But not at all farfetched for hard drive access. Quite a few of the most recent posts in this very forum deal with syncing external audio to video, or combining multi-camera shoots. In these cases it doesn't help in the slightest to be able to read a file contiguously since the head still has to bounce around between the various source files.

Two curious points, as this relates to Vegas or NLE usage specifically:

1) The more tracks you have, the greater the need for disk throughput, yet the more the head has to bounce. Conversely, in situations which can take advantage of a defragmented source file set by having fewer or one source, the need for disk throughput drops precipitously. The advantage is nullified in both cases.

2) In the case of multiple sources and needing faster access to the data streams, it can be shown mathematically that having the data blocks distributed randomly across the drive actually increases the chances that the head will be closer to the next needed block and have to move less, while a defragmented drive is much closer to a worst-case scenario. And as mentioned above, in single stream cases where there is an advantage to defragmentation, the need for it diminishes.

John Meyer
Yep. You may not see any difference, or perhaps a 2% advantage after defragging, as I did when I ran my tests rendering mpeg-2 to a 7,200 rpm eide on XP several years ago. Thanks for all the legwork.