It's 2:45am, but I'll take a stab at this before I go to bed...
DV runs at 25 megabits/second and at that rate you can fit one hour of video on a MiniDV tape. A 720x480 frame has 345,600 pixels and the HDV 1440x1080 format has 1,555,200 pixels per frame. HDV has 4.5 times as many pixels per frame than DV. So, to keep the same compression quality as DV, it would require a data rate of 112.5 megabits/sec using DV compression and a MiniDV tape would only hold 13.3 minutes of video.
The "beauty" of HDV is that it uses the same data rate as DV and the same tapes. The bottom line is that MPEG2 is a much more efficient compression scheme than DV and it is the only practical way to stuff 4.5 times as much information into the same size "container." (I didn't take the audio into consideration in the numbers above, but HDV uses 384 Kbit MPEG audio instead of the 1536 Kbit of DV.)
Interestingly, the 25megabit data rate of HDV is higher than the 19.4 megabit data rate of broadcast 1920x1080 HDTV and broadcast HDTV looks pretty good.
To look at it the other way around, there is a DV format for HiDef that doesn't use so much compression, it's been available for some time too, only downside is because of the amount of data that has to be stored, and stored quickly it costs big time. Other issue of coarse is having lots of pixels on a CCD is one thing, having the optics to match also costs and not just money, a full broadcast or film lens weighs more than the heaviest DV25 camera.
No doubt Moore's law will have a big impact on all of this (but not the question of optics), HDTV encoders are big, expensive and power hungry. I know power consumption was a big issue in the design of the HDV camera. I know I and a few others have been asking why they didn't go for mpeg-4 compression but I suspect there just isn't the horsepower available in silicon as yet or if there is it uses too much power.